ssh host completion in bash with a predifined user

If you use a lot ssh from your workstation to connect to many servers using the same user (for example, err… root, or a standard user present in every of your servers) as I do, this is a must-have :)

First of all, ensure you have

HashKnownHosts no

in your ssh_config.

Then, edit your .bashrc and append these lines:

complete -r ssh
_cssh ()
{
cur=${COMP_WORDS[COMP_CWORD]};
COMPREPLY=($(compgen -W "$(cat ${HOME}/.ssh/known_hosts |awk -F ',' '{print $1}')" -- ${cur##root@}))
}
complete -P "root@" -F _cssh ssh

you can change the root@ with your user@.

Enjoy!

pbzip2: parallel bzipping

Probably this software existed for a quite long time but I didn’t know its existence ’til now: pbzip2
it’s basically a bzip2 algorithm implementation with pthreads support. This mean, in a always more SMP world, that you can greatly improve your bzipping perfomances (divide the zipping time by the number of cores you have et voilà!)

Compression syntax is totally compatible:

$ pbzip2 big.file


while to unzip you have to do

$ pbzip2 -d big.file.bz2


Use with caution (or with -l and -p switches) cause you can easily saturate your 4xSix-cores monster.

HOWTO: Ethernet bonding in Debian Lenny

In an older post I explained how to create a bond interface in Debian Etch… now, this doesn’t work anymore due to some changes in Lenny.

So, long story short, first of all, install ifenslave

# apt-get install ifenslave-2.6


edit /etc/network/interfaces and add the bond0 config:

auto bond0
iface bond0 inet static
address 192.168.1.2
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 192.168.1.1
up /sbin/ifenslave bond0 eth0 eth1
down /sbin/ifenslave -d bond0 eth0 eth1


now edit /etc/modprobe.d/arch/x86_64 (change the filename depending on your architecture) and add these lines


alias bond0 bonding
options bonding mode=1 miimon=100 downdelay=200 updelay=200


Brief explanation:

• miimon N: check if the active interface(s) is alive every N milliseconds
• downdelay N: wait N milliseconds after a detected link failure to consider the link down
• updelay N: wait N milliseconds after a detected link restoration to consider the link up
• mode N: 1 means master/slave configuration, so there’s only one active master. If this link fails, then slave is used.

For a more complete description of all the possible parameters, refer to Linux Documentation/networking/bonding.txt

After this, you can restart networking or reboot if you are working remotely and it should work without a problem. It did for me :)

Column names with trailing spaces in tables and views

Just found out a bizarre feature of MySQL: you simply cannot create a table with a trailing space in a column name. Something like this, doesn’t work:

create table a (`a ` int);
ERROR 1166 (42000): Incorrect column name 'a '


There’s a thread in MySQL’s lists explaining it

Ok, it seems there’s a technical limitation but this solution seems like a lazy workaround to me.
The problem is with CREATE VIEW. In recent MySQL builds you cannot either create a view with a trailing space in a column name. But recent means, at least, greater than 5.0.51a, cause this version DOES let you create a view with a trailing space.

# mysql --version
mysql Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (x86_64) using readline 5.2

mysql> create view b as select a `a ` from a;
Query OK, 0 rows affected (0.00 sec)


There’s a big problem here: 5.0.51a is the default and supported version in Debian Lenny! So, if you have Lenny as a master of a more recent MySQL installation, be careful cause a CREATE VIEW with a trailing space in a column name will break your replication. Grrrr. MySQL, I hate you, really.

MySQL slaves and the “corrupted” relay log/binlog problem

When trying to pull up to date a new slave, maybe using a binlog server (a MySQL server with no true tablespace, just acting as master for slaves, feeding them with replication data), you may encounter an error like this in your mysql.err file, after noticing the replication has just broken

091028 11:32:49 [ERROR] Slave SQL: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave. Error_code: 1594
091028 11:32:49 [Warning] Slave: Field 'owner' doesn't have a default value Error_code: 1364
091028 11:32:49 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysqld-bin.000780' position 786513508


Well, everything on the web points to a “OMG, my file is corrupt!! My disk is freaking out!”. Probably, instead, it’s just simpler: you have forgotten a FLUSH LOGS; before backing up your binary logs from the original master. So, when you are recreating the binlog server from the backup, the final slave arrives to the last statement wich obviously is not complete/correct and booom, it stops.

The solution is to copy the correct and complete binlog file (in my case mysqld-bin.000780) again in the binlog server, then issue a
STOP SLAVE;
CHANGE MASTER TO master_log_file="mysqld-bin.000780", master_log_pos=786513508;
SLAVE START;


and… problem solved!
(remember to adjust file name and position to suite your needs)

Lesson learned (once again): always test your backups and do not hesitate to flush with MySQL :)

HOWTO: Install Mysql 5.1 for SPARC64 under Debian Lenny

If you happen to own a SPARC64 box, you’ll probably already know that even if the kernel is 64bit the userland comes from the normal SPARC Debian port, so it’s 32bit. Mysql is no exception, with all the 32bit limitations – mainly the 4GB RAM per process limit.

This is really  a PITA because if you have a SPARC64 box probably it has got plenty of RAM and you want to use it at its full potential, without having to messing around with Solaris (yeah, I don’t like it very much, I’m sorry).

This guide covers Mysql 5.1 installation in Debian Lenny, so we have to use SID repositories.


# echo "deb http://ftp.de.debian.org/debian/ sid main" >> /etc/apt/sources.list
# echo "deb-src http://ftp.de.debian.org/debian/ sid main" >> /etc/apt/sources.list


then let’s edit our apt preferences to avoid massive update on next dist-upgrade :)

# vim /etc/apt/preferences
Package: *
Pin: release a=stable
Pin-Priority: 900
Package: *
Pin: release a=sid
Pin-Priority: 100


and then update our repo list

# aptitude update


And here we go:

# apt-get build-dep mysql-server-5.1
# mkdir /tmp/mysql-build; cd /tmp/mysql-build
# apt-get source mysql-server-5.1
# vim mysql-dfsg-5.1*/debian/rules


here we touch a little the rules for compiling cause there are a couple of things that are not going to work by default.

The MAKE_J variable doesn’t work very well, so you can modify the grep to look for “CPU” instead of “processor” or you could hardcode it to the number of processor you have. This will make compilation a lot faster.

MAKE_J = -j$(shell if [ -f /proc/cpuinfo ] ; then grep -c CPU* /proc/cpuinfo ; else echo 1 ; fi)


then edit the CFLAGS variable because it’s used to compile some library that will ignore the environment variables we are going to set later in this howto.

CFLAGS=$${MYSQL_BUILD_CFLAGS:-"-O3 -DBIG_JOINS=1 -m64 -mcpu=niagara2 ${FORCE_FPIC_CFLAGS}"} \

it should be about line 73. Please note that -m64 will make it 64bit so it’s mandatory while the mcpu flag it’s to optimize the executable for your CPU. In my case it’s a niagara2 chip but you can use another CPU as well. Check the GCC documentation for more details
Save and quit and then we can start with the compilation process:

# export CFLAGS="-m64 -mcpu=niagara2 -O2 -g"
# export CXXFLAGS="-m64 -mcpu=niagara2 -O2 -g"
# export CPPFLAGS="-m64 -mcpu=niagara2 -O2 -g"
# export LDFLAGS="-m64 -mcpu=niagara2 -O2 -g"
# export DEB_BUILD_OPTIONS="nocheck"
# debuild -us -uc --preserve-env


that’s it. After some minutes (depending on your HW), you should have in /tmp/mysql-build all your new DEBs which you can install with dpkg -i. I advice to install the stock mysql-server-5.1 with aptitude before to get all dependencies installed, then you can use dpkg with your new DEBs.

Using tee to redirect output to multiple programs

Via http://linux.byexamples.com/archives/144/redirect-output-to-multiple-processes/

You already know that if you want to pass the output of a program to the input of another program, you can use the pipe | character.

You now that if you want to write the ouput of a program to the disk and at the same time pass it as input to another program, you can use tee.

But maybe you don’t know that if you want to pass the ouput of a program to multiple programs as input, you can use tee again with a little of subshelling.

# source_program | tee (> program1) (> program2) (> programN)| programN+1

HOWTO: install Transifex with Mysql on Debian Lenny

Transifex is a not-so-well-known opensource localization platform, written in Python and running on Django (a Python based application server, if you don’t know it). So, being not so well known, there isn’t a lot of documentation about it, and how to install it under Debian 5 Lenny it’s almost undocumented. So, here we go.

First of all, you have to install some packages. Luckily Lenny has got lot of them, although not all the needed

# aptitude install django python-urlgrabber python-setuptools python-pygments python-openid python-markdown python-httplib2
# aptitude install subversion
# aptitude install python-mysqldb
# aptitude install build-essential python-dev

These should be all the packages needed by Transifex which are available as deb packages. Now let’s install the remaining ones through easy_install

# easy_install django-authopenid django-pagination
# easy_install -f http://transifex.org/files/eggs/ contact_form tagging
# easy_install django-notification
# easy_install mercurial

Now the last package, django_evolution which is, AFAIK, only available as an SVN checkout from Google Code

# svn checkout http://django-evolution.googlecode.com/svn/trunk /tmp/django-evol
# mv /tmp/django-evol/django_evolution /usr/lib/python2.5/site-packages/

Now we can download the Transifex tarball

# cd /tmp && wget http://transifex.org/files/transifex-0.6.tar.gz
# tar xzvf transifex-0.6.tar.gz
# cp -a transifex-0.6/transifex /var/www

Now we have to edit some configuration files located in /var/www/transifex/settings with particular attention to the database backend configuration stored in 20-engines.conf. Take this as example

DATABASE_ENGINE = 'mysql'
DATABASE_NAME = 'transifex'
DATABASE_USER = 'transifex'
DATABASE_PASSWORD = 'secret_password'
DATABASE_HOST = 'ADDRESS-OF-YOUR-DB'             # Set to empty string for local socket
DATABASE_PORT = '3306'             # Set to empty string for default

obviously you must create a database (called ‘transifex’ in this example) in your database server and give full permissions to a dedicated user (called ‘transifex’ with ’secret_password’ as password in this example). You can do it with these commands in your mysql console:

CREATE DATABASE transifex;
GRANT ALL ON transifex.* to 'transifex'@'%' IDENTIFIED BY 'secret_password';

Now we can run the configuration scripts, located in the transifex’s base dir

# cd /var/www/transifex
# ./manage.py syncdb
# ./manage.py txcreatedirs
# ./manage.py runserver

Now we can execute a server instance, listening on address $IPADDRESS and port 8088,  and then we can access it fro http://$IPADDRESS:8088 in our web browser. Remember to use nohup if yoiu want to detach it from the console

# ./manage.py $IPADDRESS:8088

Ubuntu 9.04 and UXA acceleration in X.Org

With the recent release of Ubuntu 9.04 and the brand new X.org stack, finally I have got quick and usable windows effects in my KDE 4.2 install (using Kwin, not Compiz).
My workstation is a Dell Optiplex 745 with an Intel grapphic chipset

$ lspci|grep -i vga
00:02.0 VGA compatible controller: Intel Corporation 82Q963/Q965 Integrated Graphics Controller (rev 02)

with the default install I had serious problems with X, and with serious I mean textures corruptions, artifacts with KDE/Qt4 applications and all sort of graphic glitches.. really a PITA. Then the great idea: Ubuntu 9.04 now ships with a DRI2 enabled kernel, GEM for graphic memory management and this new Intel-sponsored acceleration framework called UXA, which is supposed to improve XAA and EXA. So, if everything is already screwed, why not give it a try?

So I fired up my vim editor and changed my /etc/X11/xorg.conf to look exactly like this, no more no less:

Section "Monitor"
 Identifier      "Configured Monitor"
EndSection

Section "Screen"
        Identifier      "Default Screen"
        Monitor         "Configured Monitor"
        Device          "Configured Video Device"
EndSection

Section "Device"
        Identifier      "Configured Video Device"
        Option          "AccelMethod" "UXA"
EndSection

Section "ServerFlags"
        Option  "DontZap"       "False"
EndSection

restarted X and the magic was done. I activated the composited effects in KWin and everything worked like a charm, really flying as it never did and as it is supposed to do (Vista on the same hardware works the same).
So, if you have a similar hardware, I encourage you to try the same. If anything goes wrong, removing this line

Option "AccelMethod" "UXA"

will fall back the configuration to its default state.

HTH

Disable MOTD at login with .hushlogin

If you want to disable the nice Message of the Day (MotD) your sysadmin gently shows you evry time you login in your remote shell, then all you have to do is just create a zero-lenght (as in empty) file in your $HOME called .hushlogin

> ~/.hushlogin

(you can use touch as well, or any other mean)
That’s all