pbzip2: parallel bzipping

Probably this software existed for a quite long time but I didn’t know its existence ’til now: pbzip2
it’s basically a bzip2 algorithm implementation with pthreads support. This mean, in a always more SMP world, that you can greatly improve your bzipping perfomances (divide the zipping time by the number of cores you have et voilà!)

Compression syntax is totally compatible:

$ pbzip2 big.file


while to unzip you have to do

$ pbzip2 -d big.file.bz2


Use with caution (or with -l and -p switches) cause you can easily saturate your 4xSix-cores monster.

Using tee to redirect output to multiple programs

Via http://linux.byexamples.com/archives/144/redirect-output-to-multiple-processes/

You already know that if you want to pass the output of a program to the input of another program, you can use the pipe | character.

You now that if you want to write the ouput of a program to the disk and at the same time pass it as input to another program, you can use tee.

But maybe you don’t know that if you want to pass the ouput of a program to multiple programs as input, you can use tee again with a little of subshelling.

# source_program | tee (> program1) (> program2) (> programN)| programN+1

Disable MOTD at login with .hushlogin

If you want to disable the nice Message of the Day (MotD) your sysadmin gently shows you evry time you login in your remote shell, then all you have to do is just create a zero-lenght (as in empty) file in your $HOME called .hushlogin

> ~/.hushlogin

(you can use touch as well, or any other mean)
That’s all

Mysql and integers: INT, TINYINT and all that jazz

Today I stumbled on a discussion here where I work about what’s the best INT field in Mysql to represent boolean values and about what’s the real meaning of the *INT(number) definition.

So, I ended looking in the online Mysql manual for answer but also in the “High Performance MySQL 2nd edition” written by worldwide-fame MySQL hackers (and published by O’Really).

To resume, as Mysql Manual says: BOOLEAN is an alias for TINYINT(1), so you can use both, although BIT could be a better suited solution.

But what about the parenthesis thingie? Here there are two different opinions on the matter. According to High Performance Mysql’s authors (page 82, emphasis’ mine):

MySQL lets you specify a “width” for integer types, such as INT(11). This is meaningless for most applications: it does not restrict the legal range of values, but simply specifies the number of characters MySQL’s interactive tools (such as commandline client) will reserve for display purposes. For storage and computational purposes, INT(1) is identical to INT(20)

but now let’s look at this other page in Mysql Manual (emphasis mine):

When used in conjunction with the optional extension attribute ZEROFILL, the default padding of spaces is replaced with zeros. For example, for a column declared as INT(5) ZEROFILL, a value of 4 is retrieved as 00004. Note that if you store larger values than the display width in an integer column, you may experience problems when MySQL generates temporary tables for some complicated joins, because in these cases MySQL assumes that the data fits into the original column width.

Now…who’s right? I use to trust in the Percona & OpenQuery crew but anyway the official Mysql Manual seems pretty clear about some cases in which the INT(x) value is important.
Any idea?

Self-note: Keep It Simple, Stupid

When dealing with bash (or other shells) scripts, instead of starting check outputs, write to temp file, trying to pass variables out of their scope from oine subshell to another, just remember that there is that thing called return status that can do the trick in a simpler, quicker and easier to read way.

For example, if you need to wait for a MySQL server to do not have pending queries before starting to do something, just do


while ( mysqladmin|grep -vi "show processlist"|grep "Query" > /dev/null )
do
sleep 0.1 # or whatever you want, prevent system overload
# ok, it's executing something, let's do thing A
done
# ok, it's done, let's do thing B


It’s (almost) one line long, it’s simple, effective and it just works.

Securely forward a TCP service with SSH

Sometimes you want to directly access a server on a remote LAN beyond a firewall and you don’t want to set up a VPN, or maybe you want to encrypt an unencrypted service in simple and easy way. If you can contact a [remote] SSH server, then you only need a ssh client, and that’s all!

Let’s see it more in deep:
ssh -fn -N -L 1080:remote_www.server.com:80 root@remote-ssh-proxy.server.com

The -N -L switches do the trick! The first parameter to the L siwtch (1080 in this example) will be the local port you will use to direct connect to the remote service, located at remote_www.server.com address on port 80. So, for example, you can point your browser to http://localhost:1080 and magically you will have established an encrypted connection to that web server (well, if you have a user/password for remote-ssh-prxy ;)
The -N switch is mandatory in this use case because it will disable the need of a program to be passed as an argument to ssh, permitting the tunnel-only connection.
The -fn is to put in background the connection, so the tunnel will stay open and your console won’t be blocked.

You can change the -L for -R which will do just the reverse. It will forward a port from the remote proxy to a local machine.

FreeBSD6 and nfsd gotchas

If you’re using FreeBSD6 as NFS server, you may find useful these quick tips related to /etc/export syntax, because otherwise you will be stuck with a generic

mountd[321]: bad exports list

in your logs.
So, what went wrong here?
One possible cause is that in your exports file you’re trying to export as shared NFS resource a symlink and not a real directory. NFS doesn’t like it at all and will simply no work.
One other, curious, glitch I found is that if you have two resource in two separate lines with the same options, the latter will fail.
Example of /etc/exports:

/path/share1
/path/share2 -network 192.168.1.0
/path/share3 -network 192.168.1.0

in this case share1 and share2 will work, while share3 won’t work and you’ll get a

mountd[321]: can't change attributes for /path/share3
mountd[321]: bad exports list line /path/share3

but if you change the network value in share3 (and only this), it will work!
Maybe there’s an explanation for this (I didn’t read all the exports(5) manpage) but anyway it’s a little bit strange.

Create/modify user passwords in batch mode

Hello again,

Sometimes, you need to put a non-interactive passwords using scripts, as you can see, you can’t use passwd tool, because it only works using interactive way.

On Debian, and of course lot of distros, you can use chpasswd instead (create, update & modify)

From chpasswd manual:

chpasswd reads a list of user name and password pairs from standard input and uses this information to update a group of existing users, so you can update passwords in batch mode.

echo “user:pass” | chpasswd

Regards,

.forward and /dev/null

Just a quick tip for you out there and a reminder for me. If you are using .forward files with your MTA (so, real users, not virtuals) and you want to discard everything received by that user, just put in his/her .forward


"|cat > /dev/null"


Remember that the quotes, they are mandatory (if you’re copy’n'pasting this, change the quotes because WordPress loves to change things)

Cron and multiple recipients

If you are using a maintainance user executing scheduled cronjobs, maybe you’ve found yourself needing to report the result of these jobs to different email addresses. Everyone knows for sure the MAILTO parameter, as explained in the crontab(5) manpages.

But maybe you don’t know that MAILTO is interpreted sequencially, when it’s found, so you can have different recipients in the same crontab, like in this example:


MAILTO="user1@domain.tld"
* * * * * echo "abc"
MAILTO=”user2@domain.tld”
* * * * * echo “dfe”

so user1 will get mailed with "abc" and user2 with "dfe".
It works in the standard "cron" program, so for example you can use this tip in Debian or FreeBSD (and in other Unices to, I guess)