pbzip2: parallel bzipping

Probably this software existed for a quite long time but I didn’t know its existence ’til now: pbzip2
it’s basically a bzip2 algorithm implementation with pthreads support. This mean, in a always more SMP world, that you can greatly improve your bzipping perfomances (divide the zipping time by the number of cores you have et voilà!)

Compression syntax is totally compatible:

$ pbzip2 big.file


while to unzip you have to do

$ pbzip2 -d big.file.bz2


Use with caution (or with -l and -p switches) cause you can easily saturate your 4xSix-cores monster.

HOWTO: Ethernet bonding in Debian Lenny

In an older post I explained how to create a bond interface in Debian Etch… now, this doesn’t work anymore due to some changes in Lenny.

So, long story short, first of all, install ifenslave

# apt-get install ifenslave-2.6


edit /etc/network/interfaces and add the bond0 config:

auto bond0
iface bond0 inet static
address 192.168.1.2
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 192.168.1.1
up /sbin/ifenslave bond0 eth0 eth1
down /sbin/ifenslave -d bond0 eth0 eth1


now edit /etc/modprobe.d/arch/x86_64 (change the filename depending on your architecture) and add these lines


alias bond0 bonding
options bonding mode=1 miimon=100 downdelay=200 updelay=200


Brief explanation:

• miimon N: check if the active interface(s) is alive every N milliseconds
• downdelay N: wait N milliseconds after a detected link failure to consider the link down
• updelay N: wait N milliseconds after a detected link restoration to consider the link up
• mode N: 1 means master/slave configuration, so there’s only one active master. If this link fails, then slave is used.

For a more complete description of all the possible parameters, refer to Linux Documentation/networking/bonding.txt

After this, you can restart networking or reboot if you are working remotely and it should work without a problem. It did for me :)

Column names with trailing spaces in tables and views

Just found out a bizarre feature of MySQL: you simply cannot create a table with a trailing space in a column name. Something like this, doesn’t work:

create table a (`a ` int);
ERROR 1166 (42000): Incorrect column name 'a '


There’s a thread in MySQL’s lists explaining it

Ok, it seems there’s a technical limitation but this solution seems like a lazy workaround to me.
The problem is with CREATE VIEW. In recent MySQL builds you cannot either create a view with a trailing space in a column name. But recent means, at least, greater than 5.0.51a, cause this version DOES let you create a view with a trailing space.

# mysql --version
mysql Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (x86_64) using readline 5.2

mysql> create view b as select a `a ` from a;
Query OK, 0 rows affected (0.00 sec)


There’s a big problem here: 5.0.51a is the default and supported version in Debian Lenny! So, if you have Lenny as a master of a more recent MySQL installation, be careful cause a CREATE VIEW with a trailing space in a column name will break your replication. Grrrr. MySQL, I hate you, really.

MySQL slaves and the “corrupted” relay log/binlog problem

When trying to pull up to date a new slave, maybe using a binlog server (a MySQL server with no true tablespace, just acting as master for slaves, feeding them with replication data), you may encounter an error like this in your mysql.err file, after noticing the replication has just broken

091028 11:32:49 [ERROR] Slave SQL: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave. Error_code: 1594
091028 11:32:49 [Warning] Slave: Field 'owner' doesn't have a default value Error_code: 1364
091028 11:32:49 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysqld-bin.000780' position 786513508


Well, everything on the web points to a “OMG, my file is corrupt!! My disk is freaking out!”. Probably, instead, it’s just simpler: you have forgotten a FLUSH LOGS; before backing up your binary logs from the original master. So, when you are recreating the binlog server from the backup, the final slave arrives to the last statement wich obviously is not complete/correct and booom, it stops.

The solution is to copy the correct and complete binlog file (in my case mysqld-bin.000780) again in the binlog server, then issue a
STOP SLAVE;
CHANGE MASTER TO master_log_file="mysqld-bin.000780", master_log_pos=786513508;
SLAVE START;


and… problem solved!
(remember to adjust file name and position to suite your needs)

Lesson learned (once again): always test your backups and do not hesitate to flush with MySQL :)

HOWTO: Install Mysql 5.1 for SPARC64 under Debian Lenny

If you happen to own a SPARC64 box, you’ll probably already know that even if the kernel is 64bit the userland comes from the normal SPARC Debian port, so it’s 32bit. Mysql is no exception, with all the 32bit limitations – mainly the 4GB RAM per process limit.

This is really  a PITA because if you have a SPARC64 box probably it has got plenty of RAM and you want to use it at its full potential, without having to messing around with Solaris (yeah, I don’t like it very much, I’m sorry).

This guide covers Mysql 5.1 installation in Debian Lenny, so we have to use SID repositories.


# echo "deb http://ftp.de.debian.org/debian/ sid main" >> /etc/apt/sources.list
# echo "deb-src http://ftp.de.debian.org/debian/ sid main" >> /etc/apt/sources.list


then let’s edit our apt preferences to avoid massive update on next dist-upgrade :)

# vim /etc/apt/preferences
Package: *
Pin: release a=stable
Pin-Priority: 900
Package: *
Pin: release a=sid
Pin-Priority: 100


and then update our repo list

# aptitude update


And here we go:

# apt-get build-dep mysql-server-5.1
# mkdir /tmp/mysql-build; cd /tmp/mysql-build
# apt-get source mysql-server-5.1
# vim mysql-dfsg-5.1*/debian/rules


here we touch a little the rules for compiling cause there are a couple of things that are not going to work by default.

The MAKE_J variable doesn’t work very well, so you can modify the grep to look for “CPU” instead of “processor” or you could hardcode it to the number of processor you have. This will make compilation a lot faster.

MAKE_J = -j$(shell if [ -f /proc/cpuinfo ] ; then grep -c CPU* /proc/cpuinfo ; else echo 1 ; fi)


then edit the CFLAGS variable because it’s used to compile some library that will ignore the environment variables we are going to set later in this howto.

CFLAGS=$${MYSQL_BUILD_CFLAGS:-"-O3 -DBIG_JOINS=1 -m64 -mcpu=niagara2 ${FORCE_FPIC_CFLAGS}"} \

it should be about line 73. Please note that -m64 will make it 64bit so it’s mandatory while the mcpu flag it’s to optimize the executable for your CPU. In my case it’s a niagara2 chip but you can use another CPU as well. Check the GCC documentation for more details
Save and quit and then we can start with the compilation process:

# export CFLAGS="-m64 -mcpu=niagara2 -O2 -g"
# export CXXFLAGS="-m64 -mcpu=niagara2 -O2 -g"
# export CPPFLAGS="-m64 -mcpu=niagara2 -O2 -g"
# export LDFLAGS="-m64 -mcpu=niagara2 -O2 -g"
# export DEB_BUILD_OPTIONS="nocheck"
# debuild -us -uc --preserve-env


that’s it. After some minutes (depending on your HW), you should have in /tmp/mysql-build all your new DEBs which you can install with dpkg -i. I advice to install the stock mysql-server-5.1 with aptitude before to get all dependencies installed, then you can use dpkg with your new DEBs.

Using tee to redirect output to multiple programs

Via http://linux.byexamples.com/archives/144/redirect-output-to-multiple-processes/

You already know that if you want to pass the output of a program to the input of another program, you can use the pipe | character.

You now that if you want to write the ouput of a program to the disk and at the same time pass it as input to another program, you can use tee.

But maybe you don’t know that if you want to pass the ouput of a program to multiple programs as input, you can use tee again with a little of subshelling.

# source_program | tee (> program1) (> program2) (> programN)| programN+1

HOWTO: install Transifex with Mysql on Debian Lenny

Transifex is a not-so-well-known opensource localization platform, written in Python and running on Django (a Python based application server, if you don’t know it). So, being not so well known, there isn’t a lot of documentation about it, and how to install it under Debian 5 Lenny it’s almost undocumented. So, here we go.

First of all, you have to install some packages. Luckily Lenny has got lot of them, although not all the needed

# aptitude install django python-urlgrabber python-setuptools python-pygments python-openid python-markdown python-httplib2
# aptitude install subversion
# aptitude install python-mysqldb
# aptitude install build-essential python-dev

These should be all the packages needed by Transifex which are available as deb packages. Now let’s install the remaining ones through easy_install

# easy_install django-authopenid django-pagination
# easy_install -f http://transifex.org/files/eggs/ contact_form tagging
# easy_install django-notification
# easy_install mercurial

Now the last package, django_evolution which is, AFAIK, only available as an SVN checkout from Google Code

# svn checkout http://django-evolution.googlecode.com/svn/trunk /tmp/django-evol
# mv /tmp/django-evol/django_evolution /usr/lib/python2.5/site-packages/

Now we can download the Transifex tarball

# cd /tmp && wget http://transifex.org/files/transifex-0.6.tar.gz
# tar xzvf transifex-0.6.tar.gz
# cp -a transifex-0.6/transifex /var/www

Now we have to edit some configuration files located in /var/www/transifex/settings with particular attention to the database backend configuration stored in 20-engines.conf. Take this as example

DATABASE_ENGINE = 'mysql'
DATABASE_NAME = 'transifex'
DATABASE_USER = 'transifex'
DATABASE_PASSWORD = 'secret_password'
DATABASE_HOST = 'ADDRESS-OF-YOUR-DB'             # Set to empty string for local socket
DATABASE_PORT = '3306'             # Set to empty string for default

obviously you must create a database (called ‘transifex’ in this example) in your database server and give full permissions to a dedicated user (called ‘transifex’ with ’secret_password’ as password in this example). You can do it with these commands in your mysql console:

CREATE DATABASE transifex;
GRANT ALL ON transifex.* to 'transifex'@'%' IDENTIFIED BY 'secret_password';

Now we can run the configuration scripts, located in the transifex’s base dir

# cd /var/www/transifex
# ./manage.py syncdb
# ./manage.py txcreatedirs
# ./manage.py runserver

Now we can execute a server instance, listening on address $IPADDRESS and port 8088,  and then we can access it fro http://$IPADDRESS:8088 in our web browser. Remember to use nohup if yoiu want to detach it from the console

# ./manage.py $IPADDRESS:8088

Ubuntu 9.04 and UXA acceleration in X.Org

With the recent release of Ubuntu 9.04 and the brand new X.org stack, finally I have got quick and usable windows effects in my KDE 4.2 install (using Kwin, not Compiz).
My workstation is a Dell Optiplex 745 with an Intel grapphic chipset

$ lspci|grep -i vga
00:02.0 VGA compatible controller: Intel Corporation 82Q963/Q965 Integrated Graphics Controller (rev 02)

with the default install I had serious problems with X, and with serious I mean textures corruptions, artifacts with KDE/Qt4 applications and all sort of graphic glitches.. really a PITA. Then the great idea: Ubuntu 9.04 now ships with a DRI2 enabled kernel, GEM for graphic memory management and this new Intel-sponsored acceleration framework called UXA, which is supposed to improve XAA and EXA. So, if everything is already screwed, why not give it a try?

So I fired up my vim editor and changed my /etc/X11/xorg.conf to look exactly like this, no more no less:

Section "Monitor"
 Identifier      "Configured Monitor"
EndSection

Section "Screen"
        Identifier      "Default Screen"
        Monitor         "Configured Monitor"
        Device          "Configured Video Device"
EndSection

Section "Device"
        Identifier      "Configured Video Device"
        Option          "AccelMethod" "UXA"
EndSection

Section "ServerFlags"
        Option  "DontZap"       "False"
EndSection

restarted X and the magic was done. I activated the composited effects in KWin and everything worked like a charm, really flying as it never did and as it is supposed to do (Vista on the same hardware works the same).
So, if you have a similar hardware, I encourage you to try the same. If anything goes wrong, removing this line

Option "AccelMethod" "UXA"

will fall back the configuration to its default state.

HTH

Mysql and integers: INT, TINYINT and all that jazz

Today I stumbled on a discussion here where I work about what’s the best INT field in Mysql to represent boolean values and about what’s the real meaning of the *INT(number) definition.

So, I ended looking in the online Mysql manual for answer but also in the “High Performance MySQL 2nd edition” written by worldwide-fame MySQL hackers (and published by O’Really).

To resume, as Mysql Manual says: BOOLEAN is an alias for TINYINT(1), so you can use both, although BIT could be a better suited solution.

But what about the parenthesis thingie? Here there are two different opinions on the matter. According to High Performance Mysql’s authors (page 82, emphasis’ mine):

MySQL lets you specify a “width” for integer types, such as INT(11). This is meaningless for most applications: it does not restrict the legal range of values, but simply specifies the number of characters MySQL’s interactive tools (such as commandline client) will reserve for display purposes. For storage and computational purposes, INT(1) is identical to INT(20)

but now let’s look at this other page in Mysql Manual (emphasis mine):

When used in conjunction with the optional extension attribute ZEROFILL, the default padding of spaces is replaced with zeros. For example, for a column declared as INT(5) ZEROFILL, a value of 4 is retrieved as 00004. Note that if you store larger values than the display width in an integer column, you may experience problems when MySQL generates temporary tables for some complicated joins, because in these cases MySQL assumes that the data fits into the original column width.

Now…who’s right? I use to trust in the Percona & OpenQuery crew but anyway the official Mysql Manual seems pretty clear about some cases in which the INT(x) value is important.
Any idea?

Postfix as relay to a SMTP requiring authentication

Sometimes you may in need to use an external SMTP provider to send your emails, and usually ISPs give instruction on how to configure mail clients such as Outlook or Thunderbird. But what if you are already using an internal SMTP server such as Postfix?

These guidelines are for Debian (but may be helpful with other systems as well) and are related to Postfix. The SMTP provider in the example is AuthSMTP which is a well known provider for SMTP relaying.

Given you already have a working Postfix environment, first of all edit your main.cf and add these lines:

relayhost = [mail.authsmtp.com]
smtp_sasl_auth_enable=yes
smtp_sasl_password_maps=hash:/etc/postfix/sasl-passwords
smtp_sasl_mechanism_filter = digest-md5
smtp_sasl_security_options=


then, create with $EDITOR a file called /etc/postfix/sasl-passwords and fill it with something like this:

[mail.authsmtp.com] yourusername:yourpassword

then, compile the map file

# postmap hash:/etc/postfix/sasl-passwords

now we are almost done, just restart postfix and it should work.

Now, probably it won’t really work and you’ll start to see messages like these in your postfix log:

warning: SASL authentication failure: No worthy mechs found
SASL authentication failed; cannot authenticate to server mail.authsmtp.com

that’s because you are missing some SASL packages from Debian. Issue

# aptitude install libsasl2-modules

and it should install all the missing packages and make the thing work :)