lost and found ( for me ? )

configure transparent load balancing with HA Proxy 1.5.2 on CentOS7

Here are trial and error logs when configuring transparent load balance on HAProxy which is running within KVM.

KVM host : Ubuntu 14.04
HAProxy : CentOS7 running within KVM
Web Server : Ubuntu 14.04 running within KVM

Reference
http://blog.haproxy.com/2013/09/16/howto-transparent-proxying-and-binding-with-haproxy-and-aloha-load-balancer/

# tail -1 /etc/centos-release ; uname -ri
CentOS Linux release 7.0.1406 (Core)
3.10.0-123.13.2.el7.x86_64 x86_64

install haproxy
# yum install haproxy –y

how to start/stop haproxy.
# systemctl status haproxy.service
haproxy.service - HAProxy Load Balancer
  Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
  Active: inactive (dead)

#

# systemctl start haproxy.service

# systemctl status haproxy.service
haproxy.service - HAProxy Load Balancer
  Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
  Active: active (running) since Tue 2015-01-10 10:03:50 JST; 2s ago
Main PID: 17296 (haproxy-systemd)
  CGroup: /system.slice/haproxy.service
          |-17296 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy...
          |-17297 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
          `-17298 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

Jan 10 10:03:50 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Jan 10 10:03:50 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Jan 10 10:03:50 localhost.localdomain haproxy-systemd-wrapper[17296]: haproxy-systemd-wrapper: e...
Hint: Some lines were ellipsized, use -l to show in full.

# systemctl stop haproxy.service
[root@localhost ~]#
[root@localhost ~]# systemctl status haproxy.service
haproxy.service - HAProxy Load Balancer
  Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
  Active: inactive (dead)

[ configure HTTP load balance without transparent ]

<-- 192.168.122.0 --------->       <- 192.168.100.0 -->
client 122.1 ---- 122.198 ha proxy 100.146  ------- 100.195 server

haproxy.cfg looks like this
# cat /etc/haproxy/haproxy.cfg
backend web-backend
   balance roundrobin
   mode http
   server web1 192.168.100.195:80 check

frontend http
   bind *:80
   mode http
   #use_backend web-backend
   default_backend web-backend

restart haproxy
systemctl restart haproxy.service

# systemctl status haproxy.service
haproxy.service - HAProxy Load Balancer
  Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
  Active: active (running) since Tue 2015-01-10 10:18:54 JST; 9s ago
Main PID: 17387 (haproxy-systemd)
  CGroup: /system.slice/haproxy.service
          |-17387 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy...
          |-17388 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
          `-17389 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

Jan 10 10:18:54 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Jan 10 10:18:54 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Jan 10 10:18:54 localhost.localdomain haproxy-systemd-wrapper[17387]

access to the HAProxy’s IP from the client.

When HAProxy load balanced HTTP traffic to the backend server, source IP is NATed from client’s IP(192.168.122.1) to HA Proxy’s IP ( 192.168.100.146 )

[ configure HTTP transparent load balancing ]

confirm that HAProxy has been compiled with transparent support.

HAProxy shipped with CentOS7 distribution supports transparent proxy, so you do not need to compile HAProxy from the source.
# haproxy -vv
HA-Proxy version 1.5.2 2014/07/12
Copyright 2000-2014 Willy Tarreau <w@1wt.eu>

Build options :
 TARGET  = linux2628
 CPU     = generic
 CC      = gcc
 CFLAGS  = -O2 -g -fno-strict-aliasing
 OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1

Default settings :
 maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
     epoll : pref=300,  test result OK
      poll : pref=200,  test result OK
    select : pref=150,  test result OK

stop HAPRoxy
# systemctl stop haproxy.service

enable ip_forawd and ip_nonlocal_bind so that HAProxy can forward packets and bind IP addresses HA proxy does not have.
[root@localhost ~]#
[root@localhost ~]# echo 1 > /proc/sys/net/ipv4/ip_forward
[root@localhost ~]# echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind

next, configure iptables.

at first, I cleared all of iptables rule sets.
[root@localhost ~]# iptables -F -t mangle
[root@localhost ~]# iptables -F
[root@localhost ~]# iptables -F -t nat

configure iptables and ip.
 227  iptables -t mangle -N DIVERT
 228  iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
 229  iptables -t mangle -A DIVERT -j MARK --set-mark 1
 230  iptables -t mangle -A DIVERT -j ACCEPT
 231  ip rule add fwmark 1 lookup 100
 232  ip route add local 0.0.0.0/0 dev lo table 100

Here are outputs of iptables and ip route.

[root@localhost ~]# iptables -L -n -t mangle
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DIVERT     tcp  --  0.0.0.0/0            0.0.0.0/0            socket

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

Chain DIVERT (1 references)
target     prot opt source               destination
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK set 0x1
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0

[root@localhost ~]# ip route show
default via 192.168.122.1 dev eth1  proto static  metric 1024
192.168.100.0/24 dev ens5  proto kernel  scope link  src 192.168.100.146
192.168.122.0/24 dev eth1  proto kernel  scope link  src 192.168.122.198
[root@localhost ~]#
[root@localhost ~]# ip rule show
0:      from all lookup local
32765:  from all fwmark 0x1 lookup 100
32766:  from all lookup main
32767:  from all lookup default
[root@localhost ~]#
[root@localhost ~]# ip route show table 100
local default dev lo  scope host
[root@localhost ~]#

edit haproxy.cfg
backend web-backend
   balance roundrobin
   mode http
   source 0.0.0.0 usesrc clientip
   server web1 192.168.100.195:80 check

frontend http
   #bind *:80
   bind 192.168.122.198:80 transparent
   mode http
   #use_backend web-backend
   default_backend web-backend

start HAProxy.
can not start HAProxy..
[root@localhost haproxy]# systemctl start haproxy.service
[root@localhost haproxy]#
[root@localhost haproxy]# systemctl status haproxy.service
haproxy.service - HAProxy Load Balancer
  Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
  Active: inactive (dead)

Jan 10 10:45:46 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Jan 10 10:45:46 localhost.localdomain haproxy-systemd-wrapper[17601]: haproxy-systemd-wrapper: e...
Jan 10 10:45:46 localhost.localdomain haproxy-systemd-wrapper[17601]: [ALERT] 012/104546 (17602)...
Jan 10 10:45:46 localhost.localdomain haproxy-systemd-wrapper[17601]: haproxy-systemd-wrapper: e...
Jan 10 10:46:07 localhost.localdomain systemd[1]: Stopped HAProxy Load Balancer.
Jan 10 10:46:10 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Jan 10 10:46:10 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.

check the syslog.
# cat /var/log/messages

Jan 10 10:46:59 localhost systemd: Starting HAProxy Load Balancer...
Jan 10 10:46:59 localhost systemd: Started HAProxy Load Balancer.
Jan 10 10:46:59 localhost haproxy-systemd-wrapper: [ALERT] 012/104659 (17618) : [/usr/sbin/haproxy.main()] Some configuration options require full privileges, so global.uid cannot be changed.
Jan 10 10:46:59 localhost haproxy-systemd-wrapper: haproxy-systemd-wrapper: exit, haproxy RC=256

edit haproxy.cfg
   #user        haproxy
   #group       haproxy
   user        root
   group       root

start HAProxy
[root@localhost haproxy]# systemctl start haproxy.service
[root@localhost haproxy]#
[root@localhost haproxy]# systemctl status haproxy.service
haproxy.service - HAProxy Load Balancer
  Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
  Active: active (running) since Tue 2015-01-10 10:49:08 JST; 5s ago
Main PID: 17630 (haproxy-systemd)
  CGroup: /system.slice/haproxy.service
          |-17630 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy...
          |-17631 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
          `-17632 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

Jan 10 10:49:08 localhost.localdomain

access to HAProxy’s IP from the client.

on the HAProxy.
HAProxy load balanced HTTP traffic to the backend server with client IP.
# tcpdump -n -i ens5 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens5, link-type EN10MB (Ethernet), capture size 65535 bytes
11:43:24.789772 IP 192.168.122.1.50545 > 192.168.100.195.http: Flags [S], seq 4269756824, win 14600, options [mss 1460,sackOK,TS val 336384908 ecr 0,nop,wscale 7], length 0

on the server..
nnn, src IP is 192.168.100.1 which is virbr1 address on the KVM host..
Why??
11:43:59.974689 IP 192.168.100.1.56391 > 192.168.100.195.80: Flags [S], seq 4104026007, win 14600, options [mss 1460,sackOK,TS val 336421984 ecr 0,nop,wscale 7], length 0

HAProxy and the server are running within KVM.

on the KVM host.
# ip addr show virbr1
9: virbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
   link/ether 52:54:00:62:18:29 brd ff:ff:ff:ff:ff:ff
   inet 192.168.100.1/24 brd 192.168.100.255 scope global virbr1
      valid_lft forever preferred_lft forever

I guess something is wrong with KVM network.
Seen from the capture data collected on both HAProxy and the Web server, HAProxy load balanced HTTP traffic with client’s IP.
I’m using KVM’s default network ( not openvswitch )

How about using openvswitch for the bridge?

Reference

on the KVM host, define network “ovs-test100” to use openvswitch.
# cat /etc/libvirt/qemu/networks/ovstest-100.xml
<network>
 <name>ovstest-100</name>
 <forward mode='bridge'/>
 <bridge name='ovs-100'/>
 <vlan>
   <tag id='100'/>
 </vlan>
 <virtualport type='openvswitch'/>
</network>

# virsh net-start ovstest-100
Network ovstest-100 started

# virsh net-autostart ovstest-100
Network ovstest-100 marked as autostarted

# virsh net-dumpxml ovstest-100
<network>
 <name>ovstest-100</name>
 <uuid>db2fd0b8-2e00-4241-be8a-78c6f7c8e959</uuid>
 <forward mode='bridge'/>
 <bridge name='ovs-100'/>
 <vlan>
   <tag id='100'/>
 </vlan>
 <virtualport type='openvswitch'/>
</network>

add a bridge for the network “ovstest-100”.
# ovs-vsctl add-br ovs-100

edit VM’s xml file to use that network. ( virsh edit <VM> )
   </interface>
   <interface type='network'>
     <source network='ovstest-100'/>
     <model type='virtio'/>
   </interface>

start the VM.
# virsh start <VM>

Here is the xml file after booting the VM.
   </interface>
   <interface type='bridge'>
     <mac address='52:54:00:c4:39:76'/>
     <source bridge='ovs-100'/>
     <vlan>
       <tag id='100'/>
     </vlan>
     <virtualport type='openvswitch'>
       <parameters interfaceid='79370c51-11f5-44c6-89e2-0a4260413129'/>
     </virtualport>
     <target dev='vnet2'/>
     <model type='virtio'/>
     <alias name='net1'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
   </interface>

on the KVM.
# ovs-vsctl show
6b86c7ef-e656-430e-a09e-e9d3189dec52
   Bridge "ovsbr0"
       Port "em4"
           Interface "em4"
       Port "ovsbr0"
           Interface "ovsbr0"
               type: internal
   Bridge "ovs-100"
       Port "ovs-100"
           Interface "ovs-100"
               type: internal
       Port "vnet4"
           tag: 100
           Interface "vnet4"
       Port "vnet2"
           tag: 100
           Interface "vnet2"
   ovs_version: "2.0.2"

Here is the network topology.

   192.168.122.0/24                10.0.0.0/24
client 122.50 --- 122.198 ha proxy 10.10-------openvswitch-------10.100 server
                                
on the server, add the routing.
# route add -host 192.168.122.50 gw 10.0.0.10

on the client
# curl http://192.168.122.198 --interface 192.168.122.50

capture data on the HAProxy
192.168.122.50 : client IP
10.0.0.100 : server’s IP
Running as user "root" and group "root". This could be dangerous.
Capturing on 'eth1'
[ 1222.334216] device eth1 entered promiscuous mode
 1   0.000000 192.168.122.50 -> 10.0.0.100   TCP 74 41493 > 80 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=924748 TSecr=0 WS=128
 2   0.000792   10.0.0.100 -> 192.168.122.50 TCP 74 80 > 41493 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=85420 TSecr=924748 WS=128
 3   0.000846 192.168.122.50 -> 10.0.0.100   TCP 66 41493 > 80 [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=924749 TSecr=85420
 4   0.000892 192.168.122.50 -> 10.0.0.100   HTTP 197 GET / HTTP/1.1
 5   0.001013   10.0.0.100 -> 192.168.122.50 TCP 66 80 > 41493 [ACK] Seq=1 Ack=132 Win=30080 Len=0 TSval=85420 TSecr=924749

capture data on the server
14:28:56.561720 IP 192.168.122.50.49794 > 10.0.0.100.80: Flags [S], seq 1346616839, win 14600, options [mss 1460,sackOK,TS val 857775 ecr 0,nop,wscale 7], length 0
14:28:56.561834 IP 10.0.0.100.80 > 192.168.122.50.49794: Flags [S.], seq 1872910538, ack 1346616840, win 28960, options [mss 1460,sackOK,TS val 68668 ecr 857775,nop,wscale 7], length 0
14:28:56.562151 IP 192.168.122.50.49794 > 10.0.0.100.80: Flags [.], ack 1, win 115, options [nop,nop,TS val 857776 ecr 68668], length 0
14:28:56.562167 IP 192.168.122.50.49794 > 10.0.0.100.80: Flags [P.], seq 1:132, ack 1, win 115, options [nop,nop,TS val 857776 ecr 68668], length 131
14:28:56.562212 IP 10.0.0.100.80 > 192.168.122.50.49794: Flags [.], ack 132, win 235, options [nop,nop,TS val 68668 ecr 857776], length 0

I was able to confirm HAProxy’s transparent works well within KVM in combination with openvswitch.

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.