rabbitmq集群 && consul自动发现

工作中需要构建rabbitmq集群,前置”服务发现、高可用、LoadBalance”可以使用haproxy+keepalived,但是这种方案还是比较单一,consul在此功能上显得更加全面,consul基于go语言开发,轻量级,用于实现分布式系统的服务发现与配置。这里我们就用consul和rabbitmq集群的组合实现rabbitmq服务的”自动发现、高可用、LoadBalance”(可以以同样的方式实现consul+redis、consul+mysql、consul+apigateway等等)。如有疑问请联系我,大家共同学习!

概览

1.基础环境

  • ip:192.168.50.201 hostname:consulserver01 system:centos7
  • ip:192.168.50.202 hostname:consulserver02 system:centos7
  • ip:192.168.50.203 hostname:consulserver03 system:centos7
  • ip:192.168.50.204 hostname:consulclient01 system:centos7
  • ip:192.168.50.205 hostname:consulclient02 system:centos7
  • ip:192.168.50.206 hostname:consulclient03 system:centos7

2.全局图
mahua

rabbitmq集群搭建

1.更改hosts文件
在consulclient01、consulclient02、consulclient03机器分别运行

1
sudo vim /etc/hosts

加入如下内容

1
2
3
192.168.50.204 consulclient01
192.168.50.205 consulclient02
192.168.50.206 consulclient03

2.安装rabbitmq
在consulclient01、consulclient02、consulclient03机器分别运行

1
2
3
sudo yum -y install erlang
wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.6.6/rabbitmq-server-3.6.6-1.el7.noarch.rpm
sudo yum -y install rabbitmq-server-3.6.6-1.el7.noarch.rpm

2.配置rabbitmq
在consulclient01运行

1
sudo vim /etc/rabbitmq/rabbitmq.config

加入如下内容(这里根据实际机器情况调整各个参数值)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
[
{
rabbit, [
{tcp_listeners, [{"127.0.0.1", 5672}, {"192.168.50.204", 5672}]},
{cluster_partition_handling, pause_minority},
{log_levels, [{connection, info}, {channel, info}]},

{vm_memory_high_watermark, 0.5},
{vm_memory_high_watermark_paging_ratio, 0.5},
{memory_monitor_interval, 2500},

{disk_free_limit, "300MB"},
%% Set the default AMQP heartbeat delay (in seconds).
%%
{heartbeat, 60},

%% Set the max permissible size of an AMQP frame (in bytes).
%%
{frame_max, 131072},

%% Set the max frame size the server will accept before connection
%% tuning occurs
%%
{initial_frame_max, 4096},

%% Set the max permissible number of channels per connection.
%% 0 means "no limit".
%%
{channel_max, 512}
]
}
].

在consulclient02运行

1
sudo vim /etc/rabbitmq/rabbitmq.config

加入如下内容(这里根据实际机器情况调整各个参数值)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
[
{
rabbit, [
{tcp_listeners, [{"127.0.0.1", 5672}, {"192.168.50.205", 5672}]},
{cluster_partition_handling, pause_minority},
{log_levels, [{connection, info}, {channel, info}]},

{vm_memory_high_watermark, 0.5},
{vm_memory_high_watermark_paging_ratio, 0.5},
{memory_monitor_interval, 2500},

{disk_free_limit, "300MB"},
%% Set the default AMQP heartbeat delay (in seconds).
%%
{heartbeat, 60},

%% Set the max permissible size of an AMQP frame (in bytes).
%%
{frame_max, 131072},

%% Set the max frame size the server will accept before connection
%% tuning occurs
%%
{initial_frame_max, 4096},

%% Set the max permissible number of channels per connection.
%% 0 means "no limit".
%%
{channel_max, 512}
]
}
].

在consulclient03运行

1
sudo vim /etc/rabbitmq/rabbitmq.config

加入如下内容(这里根据实际机器情况调整各个参数值)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
[
{
rabbit, [
{tcp_listeners, [{"127.0.0.1", 5672}, {"192.168.50.206", 5672}]},
{cluster_partition_handling, pause_minority},
{log_levels, [{connection, info}, {channel, info}]},

{vm_memory_high_watermark, 0.5},
{vm_memory_high_watermark_paging_ratio, 0.5},
{memory_monitor_interval, 2500},

{disk_free_limit, "300MB"},
%% Set the default AMQP heartbeat delay (in seconds).
%%
{heartbeat, 60},

%% Set the max permissible size of an AMQP frame (in bytes).
%%
{frame_max, 131072},

%% Set the max frame size the server will accept before connection
%% tuning occurs
%%
{initial_frame_max, 4096},

%% Set the max permissible number of channels per connection.
%% 0 means "no limit".
%%
{channel_max, 512}
]
}
].

3.启动rabbitmq,启用管理界面
在consulclient01、consulclient02、consulclient03机器分别运行
启动rabbitmq

1
sudo service rabbitmq-server start

启动管理页面

1
sudo rabbitmq-plugins enable rabbitmq_management

设置开机启动

1
sudo systemctl enable rabbitmq-server

查看rabbitmq节点状态信息

1
sudo rabbitmqctl status

访问管理界面
http://192.168.50.204:15672
http://192.168.50.205:15672
http://192.168.50.206:15672

4.配置rabbitmq集群
把consulclient01机器上的/var/lib/rabbitmq/.erlang.cookie文件同步到consulclient02、consulclient03机器上,注意ower和权限配置保持一致

重新启动运行consulclient02、consulclient03节点

1
sudo service rabbitmq-server restart

将 consulclient02、consulclient03 与 consulclient01 组成集群(在consulclient02、consulclient03节点上运行下面的命令)

1
2
3
sudo rabbitmqctl stop_app
sudo rabbitmqctl join_cluster rabbit@consulclient01 # (默认都是磁盘节点,如果要使自己成为内存节点,则加上--ram参数)
sudo rabbitmqctl start_app

查看集群状态

1
sudo rabbitmqctl cluster_status

5.配置策略: 镜像模式(可选)
在任一集群节点上运行

1
sudo rabbitmqctl set_policy -p / ha-allqueue "^" '{"ha-mode":"all"}'

备注:这行命令在名称为”/“的vhost上创建了一个策略,策略名称为 ha-allqueue,策略模式为 all 即复制到所有节点,包含新增节点,策略正则表达式为“^”表示所有匹配所有队列名称

6.一般操作
新增用户

1
sudo rabbitmqctl  add_user  jack abc123abc

设置用户角色(administrator,monitoring,policymaker,management)

1
sudo rabbitmqctl  set_user_tags  jack  administrator

备注:这两步操作完成后,就可以使用用户来登录rabbitmq管理界面,在里面可以进行方便的其他各种操作

consul server节点

1.安装
在consulserver01、consulserver02、consulserver03机器分别运行

1
2
3
wget https://releases.hashicorp.com/consul/1.0.6/consul_1.0.6_linux_amd64.zip
unzip consul_1.0.6_linux_amd64.zip
sudo mv consul /usr/bin/

2.配置
创建目录,在consulserver01、consulserver02、consulserver03机器分别运行

1
2
3
4
5
6
sudo mkdir -p /etc/consul.d/
sudo mkdir -p /data/consul/
sudo mkdir -p /data/consul/shell
sudo mkdir -p /var/log/consul
sudo chown -R vagrant:vagrant /data/consul # 注意这里的用户名使用你机器上真正的用户名
sudo chown -R vagrant:vagrant /var/log/consul # 注意这里的用户名使用你机器上真正的用户名

在consulserver01节点上运行

1
sudo vim /etc/consul.d/server.json

加入如下配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
"data_dir": "/data/consul",
"datacenter": "dc1",
"log_level": "INFO",
"server": true,
"bootstrap_expect": 2,
"bind_addr": "192.168.50.201",
"client_addr": "192.168.50.201",
"ports": {
"dns": 53
},
"recursors": ["8.8.8.8", "8.8.4.4"],
"ui":true
}

在consulserver02节点上运行

1
sudo vim /etc/consul.d/server.json

加入如下配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
"data_dir": "/data/consul",
"datacenter": "dc1",
"log_level": "INFO",
"server": true,
"bootstrap_expect": 2,
"bind_addr": "192.168.50.202",
"client_addr": "192.168.50.202",
"start_join": ["192.168.50.201", "192.168.50.203"],
"retry_join": ["192.168.50.201", "192.168.50.203"],
"ports": {
"dns": 53
},
"recursors": ["8.8.8.8", "8.8.4.4"],
"ui":false
}

在consulserver03节点上运行

1
sudo vim /etc/consul.d/server.json

加入如下配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
"data_dir": "/data/consul",
"datacenter": "dc1",
"log_level": "INFO",
"server": true,
"bootstrap_expect": 2,
"bind_addr": "192.168.50.203",
"client_addr": "192.168.50.203",
"start_join": ["192.168.50.201", "192.168.50.202"],
"retry_join": ["192.168.50.201", "192.168.50.202"],
"ports": {
"dns": 53
},
"recursors": ["8.8.8.8", "8.8.4.4"],
"ui":false
}

3.启动consul server
在consulserver01、consulserver02、consulserver03机器依次运行

1
sudo consul agent -config-dir=/etc/consul.d >/var/log/consul/consul.log 2>&1 &

查看/var/log/consul/consul.log是否正常启动

4.更新第一个consul server节点配置
在consulserver01节点上运行

1
sudo vim /etc/consul.d/server.json

加入如下配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
"data_dir": "/data/consul",
"datacenter": "dc1",
"log_level": "INFO",
"server": true,
"bootstrap_expect": 2,
"bind_addr": "192.168.50.201",
"client_addr": "192.168.50.201",
"start_join": ["192.168.50.202", "192.168.50.203"],
"retry_join": ["192.168.50.202", "192.168.50.203"],
"ports": {
"dns": 53
},
"recursors": ["8.8.8.8", "8.8.4.4"],
"ui":true
}

更新此节点配置

1
sudo consul reload -http-addr=192.168.50.201:8500

5.检查集群是否正常
在任一节点上运行

1
sudo consul members -http-addr=192.168.50.201:8500

还可以通过集群的ui管理界面来查看,地址为:
http://192.168.50.201:8500/ui/
http://192.168.50.202:8500/ui/
http://192.168.50.203:8500/ui/

备注:线上系统可以部署nginx服务,然后在nginx中加上basic auth,再反向代理到此管理界面,供外网访问

consul client节点

1.安装
在consulclient01、consulclient02、consulclient03机器分别运行

1
2
3
wget https://releases.hashicorp.com/consul/1.0.6/consul_1.0.6_linux_amd64.zip
unzip consul_1.0.6_linux_amd64.zip
sudo mv consul /usr/local/bin/

2.配置
创建目录,在consulclient01、consulclient02、consulclient03机器分别运行

1
2
3
4
5
6
sudo mkdir -p /etc/consul.d/
sudo mkdir -p /data/consul/
sudo mkdir -p /data/consul/shell
sudo mkdir -p /var/log/consul
sudo chown -R vagrant:vagrant /data/consul # 注意这里的用户名使用你机器上真正的用户名
sudo chown -R vagrant:vagrant /var/log/consul # 注意这里的用户名使用你机器上真正的用户名

在consulclient01节点上运行

1
sudo vim /etc/consul.d/client.json

加入如下配置

1
2
3
4
5
6
7
8
9
10
{
"data_dir": "/data/consul",
"datacenter": "dc1",
"enable_script_checks": true,
"bind_addr": "192.168.50.204",
"retry_join": ["192.168.50.201", "192.168.50.202", "192.168.50.203"],
"retry_interval": "30s",
"rejoin_after_leave": true,
"start_join": ["192.168.50.201", "192.168.50.202", "192.168.50.203"]
}

在consulclient02节点上运行

1
sudo vim /etc/consul.d/client.json

加入如下配置

1
2
3
4
5
6
7
8
9
10
{
"data_dir": "/data/consul",
"datacenter": "dc1",
"enable_script_checks": true,
"bind_addr": "192.168.50.205",
"retry_join": ["192.168.50.201", "192.168.50.202", "192.168.50.203"],
"retry_interval": "30s",
"rejoin_after_leave": true,
"start_join": ["192.168.50.201", "192.168.50.202", "192.168.50.203"]
}

在consulclient03节点上运行

1
sudo vim /etc/consul.d/client.json

加入如下配置

1
2
3
4
5
6
7
8
9
10
{
"data_dir": "/data/consul",
"datacenter": "dc1",
"enable_script_checks": true,
"bind_addr": "192.168.50.206",
"retry_join": ["192.168.50.201", "192.168.50.202", "192.168.50.203"],
"retry_interval": "30s",
"rejoin_after_leave": true,
"start_join": ["192.168.50.201", "192.168.50.202", "192.168.50.203"]
}

3.启动consul client
在consulclient01、consulclient02、consulclient03机器分别运行

1
nohup consul agent -config-dir=/etc/consul.d >/var/log/consul/consul.log 2>&1 &

查看/var/log/consul/consul.log是否正常启动

配置rabbitmq节点发现服务

1.配置节点发现服务
在consulclient01节点上运行

1
sudo vim /etc/consul.d/rabbitmq.json

加入如下配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
"services": [
{
"name": "rabbitmq",
"tags": [
"rabbitmq"
],
"address": "192.168.50.204",
"port": 5672,
"checks": [
{
"args": ["/data/consul/shell/check_rabbitmq.sh"],
"interval": "15s"
}
]
}
]
}

在consulclient02节点上运行

1
sudo vim /etc/consul.d/rabbitmq.json

加入如下配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
"services": [
{
"name": "rabbitmq",
"tags": [
"rabbitmq"
],
"address": "192.168.50.205",
"port": 5672,
"checks": [
{
"args": ["/data/consul/shell/check_rabbitmq.sh"],
"interval": "15s"
}
]
}
]
}

在consulclient03节点上运行

1
sudo vim /etc/consul.d/rabbitmq.json

加入如下配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
"services": [
{
"name": "rabbitmq",
"tags": [
"rabbitmq"
],
"address": "192.168.50.206",
"port": 5672,
"checks": [
{
"args": ["/data/consul/shell/check_rabbitmq.sh"],
"interval": "15s"
}
]
}
]
}

2.配置检测rabbitmq脚本
在consulclient01、consulclient02、consulclient03机器分别运行

1
sudo vim /data/consul/shell/check_rabbitmq.sh

加入如下内容

1
2
3
4
5
6
7
8
9
#!/bin/bash
status=`sudo /usr/sbin/rabbitmqctl node_health_check`
health=`sudo /usr/sbin/rabbitmqctl node_health_check |grep -Ec 'Health check passed'`
echo $status

if [ $health -ne 1 ]
then
exit 2
fi

加上可执行权限

1
sudo chmod a+x /data/consul/shell/check_rabbitmq.sh

3.consul client重新加载配置
在各个节点上运行

1
2
3
consul reload -http-addr=127.0.0.1:8500
consul reload -http-addr=127.0.0.1:8500
consul reload -http-addr=127.0.0.1:8500

客户端配置

  • 客户端需要如何连接rabbitmq服务呢,下面我们在客户端机器上指定dns解析服务器,来使用consul的dns解析,连接到rabbitmq集群.(consul提供了dns解析服务,如果上面的操作都没问题的话,consul已经把”rabbitmq.service.consul”解析到了consulclient01、consulclient02、consulclient03这三台机器)

1.更改resolv配置

1
sudo vim /etc/resolv.conf

在头部加入

1
2
3
nameserver 192.168.50.201
nameserver 192.168.50.202
nameserver 192.168.50.203

2.测试
这里就可以使用consul的域名解析服务了
备注:默认可以使用域名rabbitmq.service.consul或者rabbitmq.rabbitmq.service.consul,如果在上面的service定义中定义了多个tag,那么可以使用多个tag的域名,示例:tagname.rabbitmq.service.consul

1
2
nslookup rabbitmq.service.consul
nslookup rabbitmq.rabbitmq.service.consul

1
2
ping rabbitmq.service.consul
ping rabbitmq.rabbitmq.service.consul

至此,客户端就可以使用rabbitmq.service.consul或rabbitmq.rabbitmq.service.consul域名来访问rabbitmq集群。
并且如果rabbitmq集群有节点减少或增加,都不需要变动客户端代码。

坚持原创技术分享,您的支持将鼓励我继续创作!