How to remove a port from a port-channel in a Dell PowerConnect switch

This is a “note to self” type post, and basically because Google seems unable to find a direct answer to this simple question.
So it’s simple as this


# configure
(config)# interface ethernet NN
(config-if)# no channel-group

et voilà, the ethernet port doesn’t belong anymore to the port channel. It should work with PowerConnect 5324, PowerConnect 5424, PowerConnect 5448, PowerConnect 5548 etc.

HOWTO: Poor man VPN in Debian/Ubuntu with OpenSSH

If you are managing a remote Linux network and you are tired of NATting or two ssh hops to enter a remote server, but OpenVPN poses too much overhead, you can use ssh tunneling to easily create a workstation-to-site VPN.
I’ve tested this with Ubuntu 9.10 Karmic Koala as the workstation and Debian 5.0 Lenny as the server, but it should work identically with older Ubuntu and Debian (both server or workstation).

I’ve been inspired by these two tutorials, although both didn’t work 100% for me, but joining pieces did the trick, so here I am :)

Software prerequisites:

  • Standard Debian or Ubuntu
  • openssh-server on the remote side of the VPN
  • openssh-client on the local side of the VPN (your PC)

Network configuration (as an example)

  • Workstation LAN: 192.168.0.0/24
  • Server LAN: 192.168.10.0/24 on eth1
  • VPN: 10.0.0.0/24
  • Remote server public address: 1.2.3.4 on eth0

First of all, on the workstation generate a dedicated key (it should be a dedicated one cause the server will identify you’re going to bring up a tunnel based on the key you’re using to connect) with

# ssh-keygen -f /root/.ssh/VPNkey -b 2048

Now edit /etc/network/interfaces and create a new stanza like this one (remember to change IP addresses – in bold – according to your personal network configuration)

iface tun0 inet static
# from pre-up to true on the same line
pre-up ssh -i /root/.ssh/VPN -S /var/run/ssh-vpn-tunnel-control -M -f -w 0:0 1.2.3.4 true
pre-up sleep 5
address 10.1.0.2
pointopoint 10.1.0.1
netmask 255.255.255.0
up route add -net 192.168.10.0 netmask 255.255.255.0 gw 10.1.0.1 tun0
post-down ssh -i /root/.ssh/VPN -S /var/run/ssh-vpn-tunnel-control -O exit 1.2.3.4

Just a copuple of notes: address is your VPN local endpoint address (say, your workstation) while pointopoint is the remote VPNaddress (your server), which are the two tunnel’s endpoints.

Now let’s go to the server.

Edit /etc/ssh/sshd_server, add the line
PermitTunnel point-to-point

and restart your sshd instance.
Now edit (or create) /root/.ssh/authorized_keys (remember, we are on the server now, not your workstation) and add a line like

tunnel="0",command="/sbin/ifdown tun0; /sbin/ifup tun0" ssh-rsa HERE IT GOES YOUR VPNkey.pub FROM YOUR WORKSTATION

now edit /etc/network/interfaces and add this stanza:

iface tun0 inet static
address 10.1.0.1
netmask 255.255.255.0
pointopoint 10.1.0.2
post-up /sbin/sysctl -w net.ipv4.ip_forward=1
post-up /sbin/iptables -t nat -A POSTROUTING -s 10.1.0.0/24 -o eth1 -j MASQUERADE
post-down /sbin/iptables -t nat -D POSTROUTING -s 10.1.0.0/24 -o eth1 -j MASQUERADE
post-down /sbin/sysctl -w net.ipv4.ip_forward=0

the post-up and post-down commands enable the network sharing between the VPN server endpoint and the remote LAN (it’s called masquerading), so you can access the remote LAN from your workstation and not only the remote server. Obviously you need to instruct your workstation with a dedicated static route to reach the remote LAN network, and this is the route add -net in your workstation config.

Now, bring up the tunnel on the workstation with
# ifup tun0
and you should be able to reach a remote server on your remote LAN, with traffic secured by OpenSSH encryption.

HOWTO: Ethernet bonding in Debian Lenny

In an older post I explained how to create a bond interface in Debian Etch… now, this doesn’t work anymore due to some changes in Lenny.

So, long story short, first of all, install ifenslave

# apt-get install ifenslave-2.6

edit /etc/network/interfaces and add the bond0 config:

auto bond0
iface bond0 inet static
address 192.168.1.2
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 192.168.1.1
up /sbin/ifenslave bond0 eth0 eth1
down /sbin/ifenslave -d bond0 eth0 eth1

now edit /etc/modprobe.d/arch/x86_64 (change the filename depending on your architecture) and add these lines


alias bond0 bonding
options bonding mode=1 miimon=100 downdelay=200 updelay=200

Brief explanation:

  • miimon N: check if the active interface(s) is alive every N milliseconds
  • downdelay N: wait N milliseconds after a detected link failure to consider the link down
  • updelay N: wait N milliseconds after a detected link restoration to consider the link up
  • mode N: 1 means master/slave configuration, so there’s only one active master. If this link fails, then slave is used.

For a more complete description of all the possible parameters, refer to Linux Documentation/networking/bonding.txt

After this, you can restart networking or reboot if you are working remotely and it should work without a problem. It did for me :)

mii-tool is deprecated, use ethtool

Title says it all. If you want to check (or set) your ethernet NIC configuration or status in Linux, people used to use mii-tools. But there’s a lot more powerful and modern tool that obsoletes it: ethtool
You can install it with your favourite package manager, if it’s not already present in your system.
In Debian/Ubuntu, you can issue
# aptitude install ethtool

Here it is an example:


# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Current message level: 0x000000ff (255)
Link detected: yes

HOWTO: Managing Active Directory users under Linux with adtool

Usually people manages Linux boxes using Windows clients but sometimes, someone (like me, for example) needs to manage a Windows server from a Linux host (it could be a normal client or another server which wants to talk to Windows).

IMO, Active Directory is one of the best product from Microsoft, since it’s based on a well known standard like X.500 (aka LDAP) and it has a good interoperabilty (although it could be better, see all the problems Samba people had in the past).  So, even if there are tools like PHPLdapAdmin which are pretty good, if you need to automate users and groups management, there’s nothing better than a command line tool. Enter adtool.

adtool is very simple to use, but it’s not so simple to have it up&running, because this involves, amongst other things, to activate Secure LDAP in your Active Directory installation. To do this, you can follow this guide which will lead you through all the steps you have to do to enable LDAPS in Windows Server 2003. It may look scary but it works indeed, I used it myself.

Then, install adtool. In Debian/Ubuntu

# aptitude install adtool

Probably adtool is already present in your distribution’s repositories, so use your package manager. In the case it’s not present, simply download the adtool tarball from its homepage and do the usual

$ tar xzvf adtool-1.3.tar.gz
$ cd adtool-1.3
$ ./configure
$ make
# make install

It should be quite straightforward.
Now we have everything installed, so we can configure adtool.
Create /etc/adtool.cfg or, even better $HOME/.adtool.cfg because it will contain sensitive information, so lock it up to the user you’re willing to employ to modify Active Directory.
Put this in the config file (adapt to your needs)

uri ldaps://domain-controller.domain.tld
binddn cn=Administrator,cn=Users,dc=domain,dc=tld
bindpw $ADMIN_PASSWORD
searchbase dc=domain,dc=tld

As you can see we are using LDAPS here, because otherwise some adtool features like changing users’ passwords wouldn’t be available.
You don’t necessarily have to use the Administrator account, you can use whatever account you want, it just needs to have the right permissions (create user, change passwords etc).

So you can start poking your AD from the Linux command line, like this:

# create a new user with a dn like cn=$NAME,ou=$DEP,dc=domain,dc=tld
$ adtool usercreate "$NAME" "ou=$DEP,dc=domain,dc=tld"

# set user logon password
$ adtool setpass "$NAME" $my_secret_password

# to unlock the account (locked by default)
$ adtool userunlock "$NAME"

# to disable all the "account options" in the user's account tabe. Amongst them the "Password never expires" which is again enabled by default
$ adtool attributereplace "$NAME" userAccountControl 512

# to set user's mail address
$ adtool attributeadd "$NAME" mail user@domain.tld

# add the user to a group of users
$ adtool groupadduser $my_group "$NAME"

This could be very useful for user scripting and system integration if you’re in a mixed environment, just like we are.

Running out of TCP/IP ports in your Linux box?

If you have a Linux box acting as transparent/reverse/cache proxy and you see something like this in your squid logs:

squid[24228]: commBind: Cannot bind socket FD 91 to *:0: (98) Address already in use

and moreover you’re noticing that the load value is increasing too much, you are running out of available TCP/IP ports in your Squid IP address.
This is due to the fact that by default Linux 2.6 reserves about 30.000 ports (from 32768 to 61000) as local ports. Considering that a closed connection stays in the TIME_WAIT status for 60 seconds (hardcoded value in the Linux kernel), you can have ~600 TCP connections per second in your box before starting to see this problem (at least with that particular IP).
A temporary solution it is to increase the local ports range with the following command:

echo "10000   61000" > /proc/sys/net/ipv4/ip_local_port_range

giving in this case Squid a range of ~50.000 ports.

Using Wanem to simulate a wide-area network

If you (or your company) are in the web-development business, one thing you need when testing your application, besides trying it in different browsers, is trying the user experience at differents conenction speed.

This can be achieved easily with Wanen, a live CD that enables you to create a gateway for your test computer which slows down (or intruduces errors, jitters, random disconnections) the network experience. Wanem uses the almighty iproute2 tools (more in details, tc) to accomplish its tasks.

The easiest way is to try Wanem is to boot it in a virtual environment (for example, VMWare Server, for easy management). Its use is quite straightforward, so I won’t nag you here and I’ll just invite you to read the documentation the project provides, but I let you know a couple of things/errors I found while using it (I’m going to report them to the developers as well)

  • The remote administration interface, which is basically a web page, it’s so bad written HTML that onlky works with Internet Explorer 6. Firefox will render a completely useless mess instead of the simple, plain HTML table that is supposed to be. I can’t understand how is this still possible in 2007 from people using Linux (it’s a Knoppix-based live cd!). So, be careful with the advanced settings.
  • Even if you specify a static IP address on startup (in the end it’s meant as a gateway, so DHCP is almost useless), there’ll always be a “pump” (DHCP client) process active in memory, resetting your IP from time to time, if you are in a DHCP’ed environment. To solve this, you have to do a couple of tricks because by default Wanem only gives you access to a limited control shell.
    So, acces Wanem from a remote ssh with these options:

    ssh perc@$WANEM_IP -t /bin/bash

    and enter the password you created at boot time. Now you’re in the live cd and you may try to kill the pump process. But you can’t since you don’t have enough permissions! And sudo/su ask you an inexistent root password. The solution is the “dosu” executable found in the home you’ve just entered. Resuming:

    dosu killall pump

Keepalived and TCP_CHECK problem

Today I was debugging a problem I had with keepalived not discovering that a real server behind a virtual IP it manages, had died.

The problem was really strange because the check was very, very simple

real_server 192.168.1.65 3306
{
TCP_CHECK
{
connect_port 3306
bindto 192.168.1.65
connect_timeout 2
}
}

This configuration was created after reading keepalived.conf man pages, that talk about these 3 options for the TCP_CHECK, without going in deeper details. So I assumed that bindto IPADDR has to be used to indicate to which IP address we should connect to do the check. But I was wrong, because with this configuration if the real server behind dies, keepalived doesn’t notice anything at all. This is because the “bindto” option, I guess, is used to choose to which local (to the LVS director) IP address bind to check the external IP:port.
So, changing the configuration to looks like this:


real_server 192.168.1.65 3306
{
TCP_CHECK
{
connect_port 3306
connect_timeout 2
}
}

fixed the problem. Keepalived is a great product and works quite well, but it’s documentation is a bit disappointing.

Windows 2003 DHCP/DNS server and non-Windows clients

Usually, in a 100% Microsoft environment, DHCP that automagically updates the DNS entries for every new DHCP client in the network is not a problem, since every client is part of the Active Directory and every machine as by default the rights to “talk” to the DNS server and tell it which is the client’s new IP address and hostname.

But if you have other clients in your network (for example Linux, FreeBSD, embedded devices like JetPrint etc) that need DHCP and you want automatic DNS update, this will not work because they don’t have the rights to write in the DNS records list.

To solve this problem, the update has top be done by the DHCP server itself, and to do the trick you have

  • Go to an AD user management snap-in
  • Create a new user called, for example, dhcp2dns and make it member of the DnsUpdateProxy group
  • Give a password to the dhcp2dns user
  • Go to the DHCP snap-in, and select the server you want to configure
  • Right click on the server name /address, and select properties
  • Here, select the DNS tab
  • Tick the Enable DNS dynamic updates checkbox
  • Select Always dynamically update DNS A and PTR records
  • Now, select the “Advanced” tab and click the “Credentials” button
  • Here, put the credentials of the dhcp2dns user you have created in the first steps

and you are done, it should work or, at least, it did work for me :)

Linux 2.6.x as real server in a LVS system

DISCLAIMER: this is not an howto, it’s just a reminder for myself and a tip for someone who already knows LVS (Linux Virtual Server) basics.

So, if you need to use a Linux as a real server behind a LVS and you’re using kernel 2.6.x, you will know that if you try a

ifconfig lo:0 192.168.1.131 -arp netmask 255.255.255.255 up

then arping from an external host will be answered ANYWAY by your host, and this is a VERY BAD THING in an LVS environment (cause the client will contact directly only one real server and will not pass always through the virtual server). This could seem a bug cause we are using the -arp switch in ifconfig which should tell the kernel to ignore the ARP replies for this IP.
To solve this problem, you have to change these kernel settings with sysctl:

net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.eth0.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

in sysctl.conf (you can use sysctl net.ipv4.conf.eth0.arp_ignore=1 for example if you want to give it a try before).

With these parameters set, if you try to arping the lo:0 IP address from an external host, it won’t answer, but nonetheless your real server will accept the packets sent by the director of the LVS system.

I’ve tried this in Debian.