Nginx + Keepalived, high available load balance

Server environment

1
2
3
4
5
6
7
8
$ cat /etc/issue
Ubuntu 14.04.5 LTS

$ nginx -v
nginx version: nginx/1.4.6 (Ubuntu)

$ apachectl -V
Server version: Apache/2.4.7 (Ubuntu)
Server Keepalived Virtual IP Upstream IP Web Port
Nginx 1 Master 192.168.33.150 WebSite 1、WebSite 2 192.168.33.1 9001
Nginx 2 Backup 192.168.33.150 WebSite 1、WebSite 2 192.168.33.2 9002
WebSite 1 192.168.33.11 8011
WebSite 2 192.168.33.12 8011

Install Nginx

1
$ sudo apt-get install -y nginx

Nginx configuration

1
2
3
$ cd /etc/nginx/sites-enabled/
$ sudo rm -f default
$ sudo vim site.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
upstream apache {
server 192.168.33.11:8011 weight=1;
server 192.168.33.12:8011 weight=1;
}

server {
server_name _
listen 80;

location / {
proxy_pass http://apache;
proxy_set_header X-NGINX "NGINX-1"; // Just add a header info to distinguish
}
}

Restart nginx

1
$ sudo service nginx restart

Install Keepalived

1
$ sudo apt-get install -y keepalived

Config Keepalived

Nginx Master 192.168.33.1 :

1
$ sudo vim /etc/keepalived/keepalived.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
vrrp_script chk_nginx {
script "/etc/keepalived/check_nginx.sh" //check nginx process script
interval 2
weight -20
}

global_defs {
notification_email {
//something to notify by mail
}
}
vrrp_instance VI_1 {
state MASTER // Mark this one is the master server
interface eth0 // Your NIC 's name
virtual_router_id 51 // Virtual router, whether master or backup keepalived, it must be the same
mcast_src_ip 192.168.33.1 // Master nginx machine 's ip
priority 250 // Master priority must higher than backup
advert_int 1

authentication {
auth_type PASS
auth_pass 123456 // Password for communication in keepalived
}

track_script {
chk_nginx
}
virtual_ipaddress {
192.168.33.150 // Virtual public ip
}
}

Check nginx process script

1
$ sudo vim /etc/keepalived/check_nginx.sh

1
2
3
4
5
6
7
8
9
#!/bin/bash  
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ];then
/etc/init.d/nginx start
sleep 3
if [ `ps -C nginx --no-header |wc -l`-eq 0 ];then
killall keepalived
fi
fi

Nginx Slave 192.168.33.2 :

1
$ sudo vim /etc/keepalived/keepalived.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
vrrp_script chk_nginx {
script "/etc/keepalived/check_nginx.sh" //check nginx process script
interval 2
weight -20
}

global_defs {
notification_email {
//something to notify by mail
}
}
vrrp_instance VI_1 {
state BACKUP // Mark this one is the backup server
interface eth0 // Your NIC 's name
virtual_router_id 51 // Virtual router, whether master or backup keepalived, it must be the same
mcast_src_ip 192.168.33.2 // Slace nginx machine 's ip
priority 240 // Master priority must higher than backup
advert_int 1

authentication {
auth_type PASS
auth_pass 123456 // Password for communication in keepalived
}

track_script {
chk_nginx
}
virtual_ipaddress {
192.168.33.150 // Virtual public ip
}
}

Check nginx process script

1
$ sudo vim /etc/keepalived/check_nginx.sh

1
2
3
4
5
6
7
8
9
#!/bin/bash  
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ];then
/etc/init.d/nginx start
sleep 3
if [ `ps -C nginx --no-header |wc -l`-eq 0 ];then
killall keepalived
fi
fi

Start Keepalived And Check the process

1
2
3
$ sudo service keepalived start
$ ps aux | grep keepalived
$ ps -ef | grep nginx

Check Virtual IP (192.168.33.150) Has Been Bound

192.168.33.1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:80:6e:f8 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
valid_lft forever preferred_lft forever
inet 192.168.33.150/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe80:6ef8/64 scope link
valid_lft forever preferred_lft forever

Look at line 12, the virtual ip is bound on this machine.

Also try on backup server 192.168.33.2.

1
2
3
4
5
6
7
8
9
10
11
12
13
$ ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:80:6e:f8 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe80:6ef8/64 scope link
valid_lft forever preferred_lft forever

You can find that the virtual IP is not bound to the backup server.

Visit The Virtual Public IP

Visit this link several times to check if there is any change

1
$ curl http://192.168.33.150

Keepalived Non-Preempt Mode

Change Keepalived configuration

1
2
3
4
5
6
vrrp_instance VI_1 {  
state BACKUP // All machines' state change to 'BACKUP'
...
...
nopreempt // Add non-preempt mode keyword
}

The master keepalived machine is decided by priority

The Meaning Of Non-Preempt Mode

MASTER provides service, BACKUP waiting.
MASTER crash, BACKUP provides service instead of MASTER.
MASTER resume, MASTER waiting, BACKUP still provides service.
BACKUP crash, MASTER rebinds virtual IP and provides service.

Share