Scratchpad

While Python as the extremely useful virtualenv tool, I haven't found anything similar for Perl. What I want is quite simple: being able to test random perl tools. They usually depend on a various set of libraries, some of them either too old on my system or not even present. Most of the time, I just want to test the tool, so installing random perl modules system-wide is not an option. Moreover, I really don't like having to install something system-wide which is not coming as a package for my distribution. The solution is to have a virtual environment where I can install all the libraries I want, somewhere in my user directory. This has the advantage too of allowing to use a different version of a library for different perl tools.

While I haven't found an all-in-one tool for perl, there's the CPAN module called local-lib which helps a lot. You can find its documentation here. Here is how to use it to create a virtual perl environment:

• ${PERLENV} represents the directory where you want to have your perl virtual env. E.g.:

export PERLENV="$HOME"/perl5

• Create the directory and go in it to make our life easier:

mkdir -p "${PERLENV}"
cd "${PERLENV}"

• On http://search.cpan.org/dist/local-l... obtain the link to the tarball ("download" link), download, extract the library and go in its directory:

wget -O local-lib.tar.gz "${TARBALL_URL}"
tar xvf local-lib.tar.gz
cd local-lib-*

• Bootstrap it:

  perl Makefile.PL --bootstrap="${PERLENV}"

• If that's the first time you use CPAN, you will have a message like, that you probably want to confirm:

   Would you like me to configure as much as possible automatically? [yes]

• Then install local-lib:

   make test && make install

The environment is now more or less setup. However, to use it, you need to set a few environment variables. Here are 2 helpers for that:

• A wrapper script for perl. Anytime this particular perl wrapper will be called, it will behave like a normal perl, but using your environment:

cat > $PERLENV/perl << EOF
#!/bin/sh
PERLENV="${PERLENV}"
perl -I\$PERLENV/lib/perl5 -Mlocal::lib=\$PERLENV -MCPAN "\$@"
EOF

chmod 755 $PERLENV/perl

• If you need to use other programs without an easy way to override the perl binary to use, you can create an activate script which would setup the environment in your current shell. Once created, you will just need to do source "${PERLENV}"/activate and then anything that needs perl in the current shell will use your custom environment.

cat > "${PERLENV}"/activate << EOF
PERLENV="${PERLENV}"
eval \$(perl -I\$PERLENV/lib/perl5 -Mlocal::lib=\$PERLENV -MCPAN "\$@")
EOF

Tadam, you local perl is now usable. You can start installing the CPAN modules you want and they will be installed in the environment:

"${PERLENV}"/perl -MCPAN -e 'CPAN::install(POE::Filter::IRCD)'

Or the CPAN shell:

"${PERLENV}"/perl -MCPAN -e shell

And if you want to install your favorite perl tool:

source "${PERLENV}"/activate
cd myfavoritetool
./runit

Various notes:

• I hardly know anything about perl - I might have missed some things. However, this approach works and seems somehow clean.
• This is not a fully isolated environment; system libraries will be available too. That's usually more an advantage than an issue though.
• The main issue is that it still relies on your //~/.cpan// configuration directory, so you cannot have CPAN parameters specific to your environment. AFAIK, there's no easy way around that.

For some reason, I have a smtp server running completely as user and which cannot open port 25; it listens to port 2225 instead. So, I want to redirect port 25 to port 2225, using iptables.

The standard redirection is simple and can be easily found on your favorite search engine:

iptables -t nat -A PREROUTING -p tcp --dport 25 -j REDIRECT --to-port 2225

However, due to the way iptables works, this redirection won't work if you're trying to connect to port 25 from the machine itself, through localhost or through one of the IPs of the machine.

The problem is that there's no INPUT chain in the nat table, so we cannot modifies the packets when they arrive on the host. A classic option is to use the CONNMARK jump target to mark appropriate packets when you can detect them and act on it later just by detecting the mark. In this case, it would mean marking the packet in the INPUT chain of the filter table and modifying the packet in the OUTPUT chain of the nat table. However, the INPUT chain for packets going to the machine is obviously taken after the OUTPUT chain for packets issued by the machine.

The solution I've kept is to simply do a maching based on the IPs of the machine. It means doing basically one rule per IP of the machine, which is not that great but probably ok as long as you don't often change of IPs.

In my case, this machine has 2 IPs and of course the localhost interface. In pratice, for the two IPs:

iptables -t nat -A OUTPUT -p tcp -d 1.2.3.4 --dport 25 -j REDIRECT --to-port 2225
iptables -t nat -A OUTPUT -p tcp -d 5.6.7.8 --dport 25 -j REDIRECT --to-port 2225

And for localhost:

iptables -t nat -A OUTPUT -p tcp -d 127.0.0.0/8 --dport 25 -j REDIRECT --to-port 2225

I'm often playing with tools to manipulate full system images for virtual machines and so I often need to create disk images.

The script that you can find here allows to create a disk image of an arbitrary size with partitions and a working grub bootloader. This kind of script can be a bit dangerous, so I put it there just as an example, be careful. It is using qemu-img to create the disk image (which can be easily replaced by dd), sfdisk to create the partitions, and grub to install a boot loader.

There is behind that a couple of not often documented issues.

Disk size

The PC partition table still uses the wonderful Cylinder/Head/Sector (a.k.a. CHS) scheme to address the disk. Of course, those values do not correspond anymore to any physical reality, but they are still here to annoy you. The idea with PC partition table is that partitions can start and stop only at CHS boundaries. Typically, you count 255 heads and 63 sectors, and you have a number of cylinders depending on the disk size. Usually one sector is 512 octets, so the math is easily done.

It is not a problem to have a disk size not strictly aligned with a CHS boundary. Worst thing is that you lose a few kilobytes, which is not an issue. However when it comes to creating a disk image from an host, you can have the following problem:

• You create the image:

qemu-img create -f raw "${IMGFILE}" ${IMGSIZE}

• Then you create the partition table. I'm using here option -D for sfdisk. It will move the first partition a bit forward (one head further). In practice, I do that to match the usual partition scheme of tools such as fdisk. The following command creates only one linux partition, taking the whole disk and marks it as bootable.

sfdisk -D $IMGFILE <<EOF
,,L,*
;
;
;
EOF

• And then, you want to format the partition. The partition is not starting at the beginning of the file, so you first need to do some trickery using linux loopback devices. The offset 32256 come from 63 * 512, with 63 being the number of sectors and 512 the size of one sector.

losetup -o 32256 /dev/loop0 "${IMGFILE}"

• Now, you have a block device, /dev/loop0, which corresponds to the partition, so you can format it. Be careful, there's no confirmation here :

mkfs.ext3 /dev/loop0

And now you have a problem if you look carefully. The mkfs called will determine (by default) the filesystem size based on the block device size. But if your disk size was not aligned on CHS boundaries, it means that you have more place on the disk image, hence on /dev/loop0 than you really have on the partition. Another way to say that is that the end of the filesystem will be after the end of the partition. And of course, fsck won't be too thrilled about that.

There's a couple of solutions:

• Explicitely set number of blocks on the mkfs command. E.g, mkfs.ext3 -b 4096 /dev/loop0 <number of block>. You can compute number of blocks from the disk geometry.
• Set the size of the disk to perfectly fit with the geometry, so you don't have to teach mkfs about size. That the approach I'm doing in the script, because sfdisk is a pain to parse to obtain the geometry.

Configuring grub

Grub can be easily put a disk image. In its default setup, it needs to have a few files on the partition to be able to boot and show up a menu. At the beginning, I was simply counting on the files provided by the linux distribution I was putting on the disk. However, it can be sometime incompatible with the grub version you're using from the host machine. So you need to put grub files from the host machine first. I'm doing something along those lines after having mounted the partition on $INSTDIR:

cp /boot/grub/{stage1,stage2,e2fs_stage1_5} $INSTDIR/local/grub
ln -s /boot/grub/menu.lst $INSTDIR/local/grub/menu.lst

I'm not putting the host file in the usual /boot/grub directory. The idea is that I'm going to put a full distribution image here, and I don't want to override the real grub files. Moreover, this way, if I reinstall grub from the virtual machine afterwards, it will probably do the right thing by using files from /boot/grub.

Now, I just need to setup grub using the following commands:

grub --batch <<EOF
device (hd0) ${IMGFILE}
root (hd0,0)
setup --prefix=/local/grub (hd0)
quit
EOF

There are two things here. I need to specify that hd0 is in fact my disk image using the device stanza, to avoid writing on the real disk. Then, I use --prefix in the setup command to make sure that grub will be using the files I copied from the host.

I've updated the script to launch qemu in screen that I described in this post. You can get the new version here.

It nows take care of creating a new screen session if needed, and if launched from an existing screen, it will simply had new tabs for the serial port and the monitor. Its usage changed too, you now needs to specify the qemu binary to launch, since the wrapper now just add the necessary options to the qemu line. And it now doesn't close the screen, so you can look at what qemu wrote on its output.

Basic usage : qemu-wrapper qemu -hda vm.img -nographic

I'm currently using dspam to filter my mails. However, as I'm using IMap, spam filtering is done server side. So, to identify false negative (FN) and false positive (FP), I cannot use some built-in feature of my mail clients (I have severals), I need to communicate with the server. Until recently, I was using the classic approach: when I got a FN or FP, I redirect the mail (with full headers) to a special address, which send it to dspam, telling it that it was a misclassification.

The problem with this approach in practice is that to mark a FP/FN I need to retransmit the mail, and move it to the correct folder, which is redundant. Of course, most mail clients can help doing that with some configuration, but still, that's several operations where it is not really needed. Moreover, in the case of FN, it means sending through SMTP a spam, which can sometimes be a problem.

So, I've made a script which watches the content of the spam folder and detects mails which are added and removed. This way, to mark a FN as spam, I just need to move it to the Spam folder: the script will detect that a mail has been added, and will re-train dspam with the signature of the email. For FP, it's the same thing: I just need to move the mail out of the spam folder, the script will detect that and call dspam with the signature of the moved email.

The script is a single-file python script : dspam_auto.py

It works with Maildir style mailboxes, dspam and a mysql database. However, the principle is simple and can easily be adapted. The implementation is currently really dumb and could be enhanced (especially resource-wise, for the regular scan) but it's working.

The principle of the script is to scan the directory regularly to look for missing and added mails. The script must be plugged to the delivery system too (procmail in my case) to avoid trying to re-learn a spam already classified as spam.

How to use it :

• First, your dspam must be configured to put the signature as an header, not in the body of emails.
• Download the script on the server, let's say in ~/bin/dspam_auto.py and don't forget to mark it as executable.
• Edit the beginning of the script to adapt the settings to your configuration
  • DB_USER, DB_PASS, DB_NAME : Access to the mysql database. You can reuse the DB you're using for dspam.
  • DB_TABLE is the name of the table which will be used to store the script information. It shall be a non-existing table, the default value is probably usually ok.
  • DSPAM_USER is the name of the dspam user you're using; usually your login name.
  • DSPAM_UID is the uid of the user for the script. It's probably good practice to use the same as in dspam, but it's in practice independant. You can check for user/uid in the table 'dspam_virtual_uids' of your dspam database.
  • LOG_FILE : Where to log all of script runs. It's really useful for debugging or just checking that the script isn't going rogue.
  • The dspam command to re-classifies FN/FP is in the classify function, feel free to adapt it to your installation. E.g., the script is currently using the option --client which you might not need.
• Check that DRY_RUN is True ; that's needed to initialize correctly the script database without polluting the dspam database.
• Initialize the database: ~/bin/dspam_auto.py init
• Add a regular scan, in cron (using crontab -e as example, all on one line):

*/10 * * * * $HOME/bin/dspam_auto.py update $HOME/Maildir/.Spam

This line make the scan run every 10 minutes which is probably largely enough (especially that the current version of the script is not really nice to database :). Note that the first scan will detect all existing spam as FN, so double check that DRY_RUN is True before screwing your dpsam.

• Modify the procmailrc to tell the script when each spam is detected. I have something like that in my .procmailrc :

# Spam filtering:
:0fw
| /usr/bin/dspam --stdout --deliver=spam,innocent --user pierre

# Tell the script for each detected spam
:0 ic
* ^X-DSPAM-Result: spam
| /home/pierre/dspam/dspam_auto.py push

# And deliver spam in the spam folder
:0:
* ^X-DSPAM-Result: spam
.Spam/

Note that the script is slightly racy, as calling the script and delivering the script is not atomic. However, as long as you don't run the scan every 10 seconds it shall not matter much, and recover itself from previous mistakes anyway. The way to implement that with no race condition would be to do the delivery ourselves, but I prefer not to for reliability reason: if my script is screwed up, it won't trash mails.

• Configuration is done. As long as you're in dry run mode, you can watch the effect of the script by moving mail in and out from the Spam folder. Typically, moving a spam out then in (don't forget to wait for the cron scan between operations) will produce those kind of log lines :

INFO 2008-11-09 11:40:09,710 [dryrun] Classify command: /usr/bin/dspam --signature=4916ba3b179033708835974 --class=innocent --source=error --client --user pierre
INFO 2008-11-09 11:50:09,338 [dryrun] Classify command: /usr/bin/dspam --signature=4916ba3b179033708835974 --class=spam --source=error --client --user pierre

• Once you think that your all set (a.k.a, you've configured the above and at least one scan was fully done), you can set DRY_RUN to False and enjoy a simple way to mark FP and FN in imap :)

I'm now always using a US keyboard. However, I still often need to write french text with those keyboards, hence needing accents. Of course, I don't like when what's written on the keyboard doesn't match what it does :-)

On classical linux, there is an international version of the US layout. However, it modifies "standard" behaviour; i.e., if you want to do a double quote, you need to press it twice because in this intl layout, it is, by default, a dead key. Same applies for a couple of other symbols, such as the simple quote or tilde. So, I've made a custom version of the US layout, providing easy access to accent through alt-gr (both with dead keys and common accents in french), but without modifying standard behaviour of keys.

It's partly based on us-intl, with deadkeys removed from default bindings, and the following bindings added (mainly):

• altgr-` : dead grave accent
• altgr-' : dead acute accent
• altgr-^ : dead circonflex accent
• altgr-" : dead diaresis
• altgr-a : à
• altgr-e : é
• altgr-c : ç
• altgr-u : ù

You can find the xkbmap file here : us-custom

It still needs tuning, but it works already quite well for my purposes. You just need to the file in /usr/share/X11/xkb/symbols and do a setxkbmap us-custom (if you named the file us-custom) to load it.