LVS/TUN mode with FreeBSD and Solaris realserver
Author: Ma Luyao RHCE maluyao at gmail dot com
August 16, 2006
Contents
Introduction
The Linux Virtual Server Project (LVS) is a project started by Dr. Wensong Zhang. LVS implements three load balance technology.
a. Network Address Translation (VS/NAT)
b. Direct Routing (VS/DR)
c. IP Tunneling (VS/TUN)
VS/NAT is easy to setup. The load balancer may be a bottleneck of the whole system where the number of servers is more than 20, because both the request packets and response packets need to be rewritten by the load balancer.
VS/DR has the best performance. VS/DR uses MAC-spoofing technology, so it requires that one of the load balancer’s NIC and the real server’s NIC must be in the same IP network segment, and physical segment as well.
VS/TUN is the most scalable. The advantage is since servers connect each other by using IP-tunneling, the load balance and real servers can reside on different LAN , or even WAN.
I plan to setup a set of servers with load balance technique. Realserver run FreeBSD and Solaris system in different places on our business. So it is the only choice for us to use VS/TUN.
We install Fedora Core 5 Linux on our Load Balance server. Kernel-2.6.15 is default.
FreeBSD version is 5.4, we recompile the kernel in order to improve the performance. The default kernel can work too.
Solaris box runs Solaris 10 x86.
Normal Linux Solaris/FreeBSD IP-tunneling
IP tunneling (IP encapsulation) is a technique to encapsulate IP datagram within IP datagram, which allows datagram destined for one IP address to be wrapped and redirected to another IP address. Many modern OS support IP tunneling, such as Linux, FreeBSD and Solaris.
Let’s do some test to setup normal Linux-Solaris and Linux-FreeBSD IPtunneling without LVS.
Testing environment
OS IP Address Tunnel IP Address
Fedora Core 5 Linux 10.0.0.1 172.16.0.1
Solaris 10 x86 10.0.0.2 172.16.0.2
FreeBSD 5.4 10.0.0.3 172.16.0.3
Config Linux box
In order to create an IP-Tunneling between Linux box and Solaris box, run command:
ip tunnel add tun0 mode ipip remote 10.0.0.2 local 10.0.0.1
ifconfig tun0 172.16.0.1 pointopoint 172.16.0.2
ip tunnel add tun1 mode ipip remote 10.0.0.3 local 10.0.0.1
ifconfig tun1 172.16.0.1 pointopoint 172.16.0.3
The ”tun0” is a virtual NIC device name. It can be replaced by other names, such as ”abc0” or ”def0”.
Config Solaris box
To create an IP-tunneling, we have two methods, First method of creating tunnel is using three commands as below.
/sbin/ifconfig ip.tun0 plumb
/sbin/ifconfig ip.tun0 tsrc 10.0.0.2 tdst 10.0.0.1
/sbin/ifconfig ip.tun0 172.16.0.2 172.16.0.1
It takes effect immediately.
Second method is creating /etc/hostname.ip.tun0 file which contains two lines as below.
tsrc 10.0.0.2 tdst 10.0.0.1
172.16.0.2 172.16.0.1 netmask 255.255.255.255 up
If we use the second method, the tunnel will exist automatically after we reboot system.
Config FreeBSD box
On FreeBSD box, we also have two methods to create an IP-Tunneling, one way is to run three commands:
ifconfig gif0 create
ifconfig gif0 tunnel 10.0.0.3 10.0.0.1
ifconfig gif0 inet 172.16.0.3 172.168.0.1 netmask 255.255.255.0
If we use the second method, the tunnel will exist automatically after we reboot system.
Add three lines on /etc/rc.conf file as below, the tunnel will exist automatically after we reboot system.
gif interfaces=”gif0”
gifconfig gif0=”10.0.0.3 10.0.0.1”
ifconfig gif0=”172.16.0.3 172.16.0.1 netmask 255.255.255.0”
Using IP-tunneling
In all the cases on Linux/FreeBSD/Solaris, the netmask value will be default if we ignore to set it.
After creating two tunnels, we can ping 10.0.0.2 and 10.0.0.3 at Linux box successfully, and ping 10.0.0.1 at Solaris or FreeBSD box successfully. All applications layers protocol will work correctly.
Working example of VS/TUN
In VS/TUN mode, because realserver don’t send any datagram to Load balancer, we need not create any tunnel on Linux box and must create proper tunnel on FreeBSD/Solaris box.
Topological graph
We can treat all machines on internet and have their real ip. The client can send their requests to VIP of LB Server. LB Server redirects the requests realserver through IP-tunneling. Realserver can send datagram to client.
Because both the gif0 of FreeBSD and ip.tun0 both are NO-ARP device, they are invisible to client.
Config FreeBSD box
On FreeBSD box, edit /etc/rc.conf as below:
...
ifconfig lnc0=”inet 192.168.0.170 netmask 0xffffff00”
gif interfaces=”gif0”
gifconfig gif0=”192.168.0.170 192.168.10.100”
ifconfig gif0=”192.168.10.200 192.168.10.99 netmask 0xffffffff”
apache enable=”YES”
...
192.168.10.99 looks like a terminal of this tunnel. It can be any idle IP and never use.
Config Solaris box
On Solaris , edit /etc/hostname.ip.tun0 as below:
tsrc 192.168.0.180 tdst 192.168.10.100
192.168.10.200 192.168.10.99 netmask 255.255.255.255 up
192.168.10.99 looks like a terminal of this tunnel. It can be any IP address and never use it.
Config Load balancer
Run ipvsadm command on Linux Box: Linux kernel must have ip forward enable.
Edit /etc/sysctl.conf and set to:
net.ipv4.ip forward = 1
While running command:
sysctl -p
It will take effect.
Run the scripts:
ipvsadm -C
ipvsadm -A -t 192.168.10.200:80 -s wlc
ipvsadm -a -t 192.168.10.200:80 -r 192.168.0.170 -i
ipvsadm -a -t 192.168.10.200:80 -r 192.168.0.180 -i
In this script, the port of LB Server and realerver must be same. Good idea is not setting the port of realserver.Now VS/TUN is finished. When a client access 192.168.10.200 at port 80, the datagram will be sent to 192.168.0.170 or 192.168.180 port 80.
Conclusions and Future Work
I believe AIX and HP-UX can also work with VS/TUN, I have no Aix or HP-UX servers. Who can help to provide me with such an environment?
References:
[1] http://www.linuxvirtualserver.org
[2] http://www.triload.com/en/manual/index.html
[3] http://blog.kovyrin.net/2006/03/17/how-to-create-ip-ip-tunnel-between-freebsd-and-linux/
[4] http://docs.sun.com/app/docs/doc/817-0573/6mgc65bcq?a=view
