/etc/hosts and the thousand-characters-long line

This is a self-note in the case I encounter another strange behaviour like this. We were experiencing a strange problem with MySQL and DNS. I was trying to do this:

$ mysql -h server.mysql
Unknown MySQL server host 'server.mysql' (-1)

but both dig and a normal ping (which in turns uses libc and nsswitch to do the name resolving) were working:

$ dig +short server.mysql

$ ping server.mysql
PING server.mysql ( 56(84) bytes of data.
64 bytes from server.mysql ( icmp_req=1 ttl=64 time=0.399 ms

and obviously connecting using the MySQL client and the IP address worked. So, what was happening? The smarter amongst you maybe already know the problem: a very very large line in /etc/hosts was driving the mysql client crazy (but not ping). Removing the “files” database fron the hosts entry in /etc/nsswitch.conf showed where the problem lied, and fixing the bad-ass line fixed the problem

Customize the console prompt in VMWare ESXi 4.0

The default console prompt of VMWare ESXi 4.0 really sucks, it’s black&white, it gives no info about the host you are connected to and if you have more than one host this is becomes quickly an headache.
So, how do you change it? Pretty easy:

echo 'export PS1="\[33[01;32m\]\u@\h\[33[00m\]:\[33[01;34m\]\w\[33[00m\]\$ "' > $HOME/.profile

then exit from the shell (ssh or local) and enter again and you will have a pretty nice colored console prompt :)

EDIT: ok, it seems that I cannot post “backslash zero” with WordPress. so please put before any “33” in this string “backslash zero” (the symbol and the number, not the two words). Thanks to Daniel for pointing this out. If you know a way to solve this, please share it :)

A handful of bash tips – part 1

During last weeks I’ve been doing lot of scripting (a user management system for our internal infrastructure which is formed by lots of services) and it ended being something like 3000 lines of bash code. Not too much but probably this is my greatest experience til now with bash :)

So, I’ve learnt a couple of things with this project and I’d like to share some tips and lessons learnt during the process. I’m not a bash guru, and if you find something could be improved, feel free to leave a comment.

Stand on the giant’s shoulders

You can find lot of docs on bash, like the Advanced Bash Scripting Guide, but these are better in my opinion

and the #bash IRC channel on irc.freenode.net. Just lurking the conversations will teach you lot of things

Use the latest and greatest bash version

This means right now to use Bash4. I mean, if you are using a shell to script things, use the latest version and take advantage of all its features! Really, I don’t care about portability, I script for my systems where I have full control. I’ve upgraded a Debian box from Lenny to Squeeze just to get bash4, go figure :) Obviously if you cannot update easily or you need to be portable, don’t fall in the bashisms trap, try to be as POSIX as possible and discard my tip(s).

Quote everything. I mean, really everything

If you are used to the simple $VARIABLE form, drop it right now and use always “${VARIABLE}”. It’s cleaner, it’s safer, it supports white spaces in the variable content etc.

Only the first quote matters

I’ve seen very often (and I was using this too) escaped quote chars when creating queries or strings to be passed somewhere. For example

QUERY="SELECT * FROM foo WHERE name=\"${VAR}\";"

because you fear that ${VAR} won’t be expanded if single quoted. But what really matters in this case is the first double quote. bash will interpret every other single quote before the closing double quote as a normal char, and pass it to the next hop. So this is perfectly right:

QUERY="SELECT * FROM foo WHERE name='${VAR}';"

${VAR} will be expanded as expected. This is quite useful if calling another script with parameters via ssh, and leave a cleaner syntax (I hate escaping chars)

ssh user@host "/path/to/script 'foobar goes first' 'second parameter'"

Use shift when receiving parameters

Probably you are already doing this, anyway it’s a lesson learnt in these days. If you’re are passing parameters to functions/other scripts instead of

# etc

use the power of shift. If you are going to change your mind about those parameters, you won’t need to rename everything.

shift 1
shift 1
# etc

Use input redirection instead of pipes when possible

A classic example

cat /path/to/mylst|while read foo
# do somtething with foo

should be

while read foo
# do something with foo
done < /path/to/mylist

why? because input redirection it is meant to do that!

Disable directory listing in Apache with Debian

If you find one of your servers with the ugly directory listing enabled, there’s a quick way to disable it in Debian

# echo autoindex | a2dismod
# /etc/init.d/apache2 restart

For other Apache installations in other distro, you can simple find the Autoindex option in your config file and delete it manually, then restart Apache

EDIT: a cleaner and more elegant way to achieve the same is, as the comments section says

# a2dismod autoindex

thanks :)

ssh host completion in bash with a predifined user

If you use a lot ssh from your workstation to connect to many servers using the same user (for example, err… root, or a standard user present in every of your servers) as I do, this is a must-have :)

First of all, ensure you have

HashKnownHosts no

in your ssh_config.

Then, edit your .bashrc and append these lines:

complete -r ssh
_cssh ()
COMPREPLY=($(compgen -W "$(cat ${HOME}/.ssh/known_hosts |awk -F ',' '{print $1}')" -- ${cur##root@}))
complete -P "root@" -F _cssh ssh

you can change the root@ with your user@.


pbzip2: parallel bzipping

Probably this software existed for a quite long time but I didn’t know its existence ’til now: pbzip2
it’s basically a bzip2 algorithm implementation with pthreads support. This mean, in a always more SMP world, that you can greatly improve your bzipping perfomances (divide the zipping time by the number of cores you have et voilĂ !)

Compression syntax is totally compatible:

$ pbzip2 big.file

while to unzip you have to do

$ pbzip2 -d big.file.bz2

Use with caution (or with -l and -p switches) cause you can easily saturate your 4xSix-cores monster.

Postfix as relay to a SMTP requiring authentication

Sometimes you may in need to use an external SMTP provider to send your emails, and usually ISPs give instruction on how to configure mail clients such as Outlook or Thunderbird. But what if you are already using an internal SMTP server such as Postfix?

These guidelines are for Debian (but may be helpful with other systems as well) and are related to Postfix. The SMTP provider in the example is AuthSMTP which is a well known provider for SMTP relaying.

Given you already have a working Postfix environment, first of all edit your main.cf and add these lines:

relayhost = [mail.authsmtp.com]
smtp_sasl_mechanism_filter = digest-md5

then, create with $EDITOR a file called /etc/postfix/sasl-passwords and fill it with something like this:

[mail.authsmtp.com] yourusername:yourpassword

then, compile the map file

# postmap hash:/etc/postfix/sasl-passwords

now we are almost done, just restart postfix and it should work.

Now, probably it won’t really work and you’ll start to see messages like these in your postfix log:

warning: SASL authentication failure: No worthy mechs found
SASL authentication failed; cannot authenticate to server mail.authsmtp.com

that’s because you are missing some SASL packages from Debian. Issue

# aptitude install libsasl2-modules

and it should install all the missing packages and make the thing work :)