pbzip2: parallel bzipping

Probably this software existed for a quite long time but I didn’t know its existence ’til now: pbzip2
it’s basically a bzip2 algorithm implementation with pthreads support. This mean, in a always more SMP world, that you can greatly improve your bzipping perfomances (divide the zipping time by the number of cores you have et voilà!)

Compression syntax is totally compatible:

$ pbzip2 big.file


while to unzip you have to do

$ pbzip2 -d big.file.bz2


Use with caution (or with -l and -p switches) cause you can easily saturate your 4xSix-cores monster.

MySQL slaves and the “corrupted” relay log/binlog problem

When trying to pull up to date a new slave, maybe using a binlog server (a MySQL server with no true tablespace, just acting as master for slaves, feeding them with replication data), you may encounter an error like this in your mysql.err file, after noticing the replication has just broken

091028 11:32:49 [ERROR] Slave SQL: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave. Error_code: 1594
091028 11:32:49 [Warning] Slave: Field 'owner' doesn't have a default value Error_code: 1364
091028 11:32:49 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysqld-bin.000780' position 786513508


Well, everything on the web points to a “OMG, my file is corrupt!! My disk is freaking out!”. Probably, instead, it’s just simpler: you have forgotten a FLUSH LOGS; before backing up your binary logs from the original master. So, when you are recreating the binlog server from the backup, the final slave arrives to the last statement wich obviously is not complete/correct and booom, it stops.

The solution is to copy the correct and complete binlog file (in my case mysqld-bin.000780) again in the binlog server, then issue a
STOP SLAVE;
CHANGE MASTER TO master_log_file="mysqld-bin.000780", master_log_pos=786513508;
SLAVE START;


and… problem solved!
(remember to adjust file name and position to suite your needs)

Lesson learned (once again): always test your backups and do not hesitate to flush with MySQL :)

Using tee to redirect output to multiple programs

Via http://linux.byexamples.com/archives/144/redirect-output-to-multiple-processes/

You already know that if you want to pass the output of a program to the input of another program, you can use the pipe | character.

You now that if you want to write the ouput of a program to the disk and at the same time pass it as input to another program, you can use tee.

But maybe you don’t know that if you want to pass the ouput of a program to multiple programs as input, you can use tee again with a little of subshelling.

# source_program | tee (> program1) (> program2) (> programN)| programN+1

Ubuntu 9.04 and UXA acceleration in X.Org

With the recent release of Ubuntu 9.04 and the brand new X.org stack, finally I have got quick and usable windows effects in my KDE 4.2 install (using Kwin, not Compiz).
My workstation is a Dell Optiplex 745 with an Intel grapphic chipset

$ lspci|grep -i vga
00:02.0 VGA compatible controller: Intel Corporation 82Q963/Q965 Integrated Graphics Controller (rev 02)

with the default install I had serious problems with X, and with serious I mean textures corruptions, artifacts with KDE/Qt4 applications and all sort of graphic glitches.. really a PITA. Then the great idea: Ubuntu 9.04 now ships with a DRI2 enabled kernel, GEM for graphic memory management and this new Intel-sponsored acceleration framework called UXA, which is supposed to improve XAA and EXA. So, if everything is already screwed, why not give it a try?

So I fired up my vim editor and changed my /etc/X11/xorg.conf to look exactly like this, no more no less:

Section "Monitor"
 Identifier      "Configured Monitor"
EndSection

Section "Screen"
        Identifier      "Default Screen"
        Monitor         "Configured Monitor"
        Device          "Configured Video Device"
EndSection

Section "Device"
        Identifier      "Configured Video Device"
        Option          "AccelMethod" "UXA"
EndSection

Section "ServerFlags"
        Option  "DontZap"       "False"
EndSection

restarted X and the magic was done. I activated the composited effects in KWin and everything worked like a charm, really flying as it never did and as it is supposed to do (Vista on the same hardware works the same).
So, if you have a similar hardware, I encourage you to try the same. If anything goes wrong, removing this line

Option "AccelMethod" "UXA"

will fall back the configuration to its default state.

HTH

Disable MOTD at login with .hushlogin

If you want to disable the nice Message of the Day (MotD) your sysadmin gently shows you evry time you login in your remote shell, then all you have to do is just create a zero-lenght (as in empty) file in your $HOME called .hushlogin

> ~/.hushlogin

(you can use touch as well, or any other mean)
That’s all

Mysql and integers: INT, TINYINT and all that jazz

Today I stumbled on a discussion here where I work about what’s the best INT field in Mysql to represent boolean values and about what’s the real meaning of the *INT(number) definition.

So, I ended looking in the online Mysql manual for answer but also in the “High Performance MySQL 2nd edition” written by worldwide-fame MySQL hackers (and published by O’Really).

To resume, as Mysql Manual says: BOOLEAN is an alias for TINYINT(1), so you can use both, although BIT could be a better suited solution.

But what about the parenthesis thingie? Here there are two different opinions on the matter. According to High Performance Mysql’s authors (page 82, emphasis’ mine):

MySQL lets you specify a “width” for integer types, such as INT(11). This is meaningless for most applications: it does not restrict the legal range of values, but simply specifies the number of characters MySQL’s interactive tools (such as commandline client) will reserve for display purposes. For storage and computational purposes, INT(1) is identical to INT(20)

but now let’s look at this other page in Mysql Manual (emphasis mine):

When used in conjunction with the optional extension attribute ZEROFILL, the default padding of spaces is replaced with zeros. For example, for a column declared as INT(5) ZEROFILL, a value of 4 is retrieved as 00004. Note that if you store larger values than the display width in an integer column, you may experience problems when MySQL generates temporary tables for some complicated joins, because in these cases MySQL assumes that the data fits into the original column width.

Now…who’s right? I use to trust in the Percona & OpenQuery crew but anyway the official Mysql Manual seems pretty clear about some cases in which the INT(x) value is important.
Any idea?

Postfix as relay to a SMTP requiring authentication

Sometimes you may in need to use an external SMTP provider to send your emails, and usually ISPs give instruction on how to configure mail clients such as Outlook or Thunderbird. But what if you are already using an internal SMTP server such as Postfix?

These guidelines are for Debian (but may be helpful with other systems as well) and are related to Postfix. The SMTP provider in the example is AuthSMTP which is a well known provider for SMTP relaying.

Given you already have a working Postfix environment, first of all edit your main.cf and add these lines:

relayhost = [mail.authsmtp.com]
smtp_sasl_auth_enable=yes
smtp_sasl_password_maps=hash:/etc/postfix/sasl-passwords
smtp_sasl_mechanism_filter = digest-md5
smtp_sasl_security_options=


then, create with $EDITOR a file called /etc/postfix/sasl-passwords and fill it with something like this:

[mail.authsmtp.com] yourusername:yourpassword

then, compile the map file

# postmap hash:/etc/postfix/sasl-passwords

now we are almost done, just restart postfix and it should work.

Now, probably it won’t really work and you’ll start to see messages like these in your postfix log:

warning: SASL authentication failure: No worthy mechs found
SASL authentication failed; cannot authenticate to server mail.authsmtp.com

that’s because you are missing some SASL packages from Debian. Issue

# aptitude install libsasl2-modules

and it should install all the missing packages and make the thing work :)

TIP: Installing untrusted packages without confirmation on Debian

Hello,

Maybe you are interested on installing untrusted packages on your Debian box, but by default you are prompted with this prompt:

Do you want to continue? [Y/n/?] y
WARNING: untrusted versions of the following packages will be installed!

Untrusted packages could compromise your system’s security.
You should only proceed with the installation if you are certain that
this is what you want to do.

untrusted_package

Do you want to ignore this warning and proceed anyway?
To continue, enter “Yes”; to abort, enter “No”:

This is fine, since it warns you about it, but it breaks non-interactive scripts, because needs user confirmation.

What then? the solution is easy, you only need to tell to aptitude that you want to use those packages without user confirmation.

From aptitude manual:
Option: Aptitude::CmdLine::Ignore-Trust-Violations
Default: false
Description: In command-line mode, causes aptitude to ignore the installation of untrusted packages. This is a synonym for Apt::Get::AllowUnauthenticated.

Just go!
# aptitude -o Aptitude::Cmdline::ignore-trust-violations=true -y install your_untrusted_package

WARNING: untrusted versions of the following packages will be installed!

Untrusted packages could compromise your system’s security.
You should only proceed with the installation if you are certain that
this is what you want to do.

untrusted_package

*** WARNING *** Ignoring these trust violations because
aptitude::CmdLine::Ignore-Trust-Violations is ‘true’!

Writing extended state information… Done

It does not use an interactive prompt and of course your script will continue :)

See you!

Self-note: Keep It Simple, Stupid

When dealing with bash (or other shells) scripts, instead of starting check outputs, write to temp file, trying to pass variables out of their scope from oine subshell to another, just remember that there is that thing called return status that can do the trick in a simpler, quicker and easier to read way.

For example, if you need to wait for a MySQL server to do not have pending queries before starting to do something, just do


while ( mysqladmin|grep -vi "show processlist"|grep "Query" > /dev/null )
do
sleep 0.1 # or whatever you want, prevent system overload
# ok, it's executing something, let's do thing A
done
# ok, it's done, let's do thing B


It’s (almost) one line long, it’s simple, effective and it just works.

mii-tool is deprecated, use ethtool

Title says it all. If you want to check (or set) your ethernet NIC configuration or status in Linux, people used to use mii-tools. But there’s a lot more powerful and modern tool that obsoletes it: ethtool
You can install it with your favourite package manager, if it’s not already present in your system.
In Debian/Ubuntu, you can issue
# aptitude install ethtool

Here it is an example:


# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Current message level: 0x000000ff (255)
Link detected: yes