1. nginx 第三方高可用模块
| IP | 备注 |
|---|---|
| 10.0.0.63 | proxy |
| 10.0.0.64 | web1 |
| 10.0.0.65 | web2 |
这里会讲解一些nignx常用高可用方案,以及引入第三方高可用模块来了解nginx作为高可用服务它是如何运行的
2. 基础环境安装
10.0.0.63 nginx 增加 nginx_upstream_check_module模块
10.0.0.64 nginx
10.0.0.65 nginx
-------------------------------------------------------------
#3台都安装上nginxx:
useradd www -u 1200 -M -s /sbin/nologin
mkdir -p /var/log/nginx
yum install -y cmake pcre pcre-devel openssl openssl-devel gd-devel \
zlib-devel gcc gcc-c++ net-tools iproute telnet wget curl &&\
yum clean all && \
rm -rf /var/cache/yum/*
mkdir -p /server/tools
cd /server/tools
wget https://www.chenleilei.net/soft/nginx-1.16.1.tar.gz
tar xf nginx-1.16.1.tar.gz
cd nginx-1.16.1
./configure --prefix=/usr/local/nginx --with-http_image_filter_module --user=www --group=www \
--with-http_ssl_module --with-http_v2_module --with-http_stub_status_module \
--pid-path=/var/run/nginx/nginx.pid
make -j 4 && make install && \
rm -rf /usr/local/nginx/html/* && \
echo "nginx daemo hello" >/usr/local/nginx/html/index.html && \
echo "export PATH=$PATH:/usr/local/nginx/sbin" >>/etc/profile
chown -R www.www /var/log/nginx /usr/local/nginx
source /etc/profile
nginx -s reload
ps -ef|grep nginx
3. nginx的高可用配置
默认的nginx高可用是一个后端连接不可用时且没有健康检查,他会一台台轮询.在 如: 金钱交易的期间都会关闭这个默认高可用'proxy_next_upstream',原因就是,敏感交易有可能在默认时间未结束前得到了响应,而另一方面有其他服务器响应了该请求,可能会出现重复交易情况,所以建议关闭默认的高可用策略,从而改用更简单易用的其他高可用模块来实现,它就是 "ngx_http_proxy_module"
3.1 默认高可用策略介绍
server 10.0.0.10:80 max_fails=2 fail_timeout=30;
weight 权重
max_fails 失败次数后停止该服务器
fail_timeout 踢出后重新检测时间. 一般max_fails和fail_timeout一起启用的,也就是失败n次,等待N秒
backup 备用服务器
max_conns 允许最大连接数
slow_start 节点恢复后不立即加入到集群.
3.2 安装 ngx_upstream_check_module [10.0.0.63]
github: https://github.com/yaoweibin/nginx_upstream_check_module
[root@master tools]# git clone https://github.com/yaoweibin/nginx_upstream_check_module.git
[root@master tools]# ll
drwxr-xr-x 7 root root 4096 Apr 16 21:53 nginx_upstream_check_module
[root@master tools]# cd nginx_upstream_check_module/
[root@master nginx_upstream_check_module]# pwd
/server/tools/nginx_upstream_check_module #这个路径用于编译nginx新增模块使用.
# nginx_upstream_check_module 模块目录:
[root@master nginx_upstream_check_module]# ll
total 268
-rw-r--r-- 1 root root 0 Apr 16 21:53 CHANGES
-rw-r--r-- 1 root root 7921 Apr 16 21:53 check_1.11.1+.patch
-rw-r--r-- 1 root root 8330 Apr 16 21:53 check_1.11.5+.patch
-rw-r--r-- 1 root root 8060 Apr 16 21:53 check_1.12.1+.patch
-rw-r--r-- 1 root root 8054 Apr 16 21:53 check_1.14.0+.patch
-rw-r--r-- 1 root root 8409 Apr 16 21:53 check_1.16.1+.patch
-rw-r--r-- 1 root root 5483 Apr 16 21:53 check_1.2.1.patch
-rw-r--r-- 1 root root 7130 Apr 16 21:53 check_1.2.2+.patch
-rw-r--r-- 1 root root 7094 Apr 16 21:53 check_1.2.6+.patch
-rw-r--r-- 1 root root 6791 Apr 16 21:53 check_1.5.12+.patch
-rw-r--r-- 1 root root 8295 Apr 16 21:53 check_1.7.2+.patch
-rw-r--r-- 1 root root 8346 Apr 16 21:53 check_1.7.5+.patch
-rw-r--r-- 1 root root 8509 Apr 16 21:53 check_1.9.2+.patch
-rw-r--r-- 1 root root 6943 Apr 16 21:53 check.patch
-rw-r--r-- 1 root root 749 Apr 16 21:53 config
drwxr-xr-x 2 root root 43 Apr 16 21:53 doc
-rw-r--r-- 1 root root 1709 Apr 16 21:53 nginx-sticky-module.patch
drwxr-xr-x 2 root root 29 Apr 16 21:53 nginx-tests
-rw-r--r-- 1 root root 112085 Apr 16 21:53 ngx_http_upstream_check_module.c
-rw-r--r-- 1 root root 529 Apr 16 21:53 ngx_http_upstream_check_module.h
-rw-r--r-- 1 root root 2848 Apr 16 21:53 ngx_http_upstream_jvm_route_module.patch
-rw-r--r-- 1 root root 11509 Apr 16 21:53 README
drwxr-xr-x 6 root root 79 Apr 16 21:53 test
-rw-r--r-- 1 root root 3342 Apr 16 21:53 upstream_fair.patch
drwxr-xr-x 2 root root 81 Apr 16 21:53 util
#编译nginx:
1. 获取nginx原有编译参数:
configure arguments: --prefix=/usr/local/nginx --with-http_image_filter_module --user=www --group=www --with-http_ssl_module --with-http_v2_module --with-http_stub_status_module --pid-path=/var/run/nginx/nginx.pid
2. 同步模块代码:
cd /server/tools/nginx-1.16.1
patch -p1 < /server/tools/nginx_upstream_check_module/check_1.16.1+.patch
输出:
[root@master nginx-1.16.1]# patch -p1 < /server/tools/nginx_upstream_check_module/check_1.16.1+.patch
patching file src/http/modules/ngx_http_upstream_hash_module.c
patching file src/http/modules/ngx_http_upstream_ip_hash_module.c
patching file src/http/modules/ngx_http_upstream_least_conn_module.c
patching file src/http/ngx_http_upstream_round_robin.c
patching file src/http/ngx_http_upstream_round_robin.h
3. 重新编译nginx来添加模块:
[root@master ~ ]# cd /server/tools/nginx-1.16.1
[root@master nginx-1.16.1]# ./configure --prefix=/usr/local/nginx --with-http_image_filter_module --user=www --group=www --with-http_ssl_module --with-http_v2_module --with-http_stub_status_module --pid-path=/var/run/nginx/nginx.pid --add-module=/server/tools/nginx_upstream_check_module/
[root@master nginx-1.16.1]# make
make结束后拷贝新的nginx二进制文件替换老版本,这和版本升级流程一样
#拷贝二进制文件替换老的nginx二进制文件
[root@master nginx-1.16.1]# \cp -a objs/nginx /usr/local/nginx/sbin/
#查看进程:
[root@master nginx-1.16.1]# ps -ef|grep nginx
root 43754 1 0 22:17 ? 00:00:00 nginx: master process nginx
www 43755 43754 0 22:17 ? 00:00:00 nginx: worker process
root 43774 37915 0 22:17 pts/0 00:00:00 grep --color=auto nginx
#关闭nginx后重新启动nginx:
[root@master nginx-1.16.1]# pkill nginx
[root@master nginx-1.16.1]# nginx
#检查模块安装是否成功:
[root@master nginx-1.16.1]# nginx -V
nginx version: nginx/1.16.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)
built with OpenSSL 1.0.2k-fips 26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --with-http_image_filter_module --user=www --group=www --with-http_ssl_module --with-http_v2_module --with-http_stub_status_module --pid-path=/var/run/nginx/nginx.pid --add-module=/server/tools/nginx_upstream_check_module/
#可以看到已经安装了
顺便做一下语法高亮显示的小优化:
mkdir ~/.vim -p
cp -r contrib/vim/* ~/.vim/
4.第三方高可用模块nginx_ustream_check_mudule的配置使用
安装完了模块后,需要对nginx进行配置才能使得新模块得到应用
egrep -v "#|^$" /usr/local/nginx/conf/nginx.conf.default > /usr/local/nginx/conf/nginx.conf
vim /usr/local/nginx/conf/nginx.conf
配置为如下:
#-----------------------------------------------------------------------------#
worker_processes 1;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log logs/access.log main;
upstream web {
server 10.0.0.64;
server 10.0.0.65;
}
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen 80;
server_name locaalhost;
location / {
proxy_connect_timeout 2s;
proxy_read_timeout 2s;
proxy_next_upstream off;
proxy_pass http://web;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
#-----------------------------------------------------------------------------#
2. 修改添加模块配置项:
worker_processes 1;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log logs/access.log main;
upstream web {
server 10.0.0.64;
server 10.0.0.65;
#模块配置项 用于检查后端服务器是否正常
check interval=5000 rise=1 fall=3 timeout=4000;
check interval=3000 rise=2 fall=5 timeout=1000 type=ssl_hello;
check interval=3000 rise=2 fall=5 timeout=1000 type=http;
check_http_send "HEAD / HTTP/1.0\r\n\r\n";
check_http_expect_alive http_2xx http_3xx http_4xx http_5xx;
#模块配置项 用于检查后端服务器是否正常
}
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen 80;
server_name locaalhost;
location / {
proxy_connect_timeout 2s;
proxy_read_timeout 2s;
proxy_next_upstream off;
proxy_pass http://web;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
验证模块可用性:
1. 为后端nginx添加时别内容:
10.0.0.64服务器添加:
echo "10.0.0.64 nginx daemon" >index.html
10.0.0.65服务器添加:
echo "10.0.0.65 nginx daemon" >index.html
可以看出节点分配正常
4.1 后端健康检查测试
前端负载均衡 开启日志记录,获取访问的后端节点IP,便于查看配置是否正确,这个配置在企业中也是必备,务必掌握
http模块下修改默认日志记录为如下:
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" $request_time $upstream_response_time $upstream_addr $upstream_status';
access_log logs/access.log main;
配置完毕后,继续测试负载均衡模块的可用性
默认端口都开 检查日志访问情况:
10.0.0.1 - - [16/Apr/2020:23:11:29 +0800] "GET / HTTP/1.1" 200 23 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36" "-" 0.001 0.002 10.0.0.64:80 200
10.0.0.1 - - [16/Apr/2020:23:11:29 +0800] "GET / HTTP/1.1" 200 23 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36" "-" 0.001 0.001 10.0.0.65:80 200
10.0.0.1 - - [16/Apr/2020:23:11:30 +0800] "GET / HTTP/1.1" 200 23 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36" "-" 0.001 0.002 10.0.0.64:80 200
10.0.0.1 - - [16/Apr/2020:23:11:32 +0800] "GET / HTTP/1.1" 200 23 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36" "-" 0.001 0.001 10.0.0.65:80 200
#从这里可以看出,我们访问10.0.0.63负载均衡的时候,他会一次去访问10.0.0.64,一次去访问10.0.0.65, 照此依次轮询.
#现在关闭后端某个节点,用于测试模块是否正常检测:
2020/04/16 23:18:38 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:18:41 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:18:44 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:18:47 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:18:50 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:18:50 [error] 46559#0: disable check peer: 10.0.0.64:80 #<---- 移除了该节点
他会进行5次检测,发现后端节点无法访问,直接将故障节点移除集群.
同时当这个集群角色恢复后,该模块依然能够检测出并将其重新加入集群.
如下:
2020/04/16 23:20:53 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:20:56 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:21:03 [error] 46559#0: enable check peer: 10.0.0.64:80 #<---- 添加了该节点
