Openstack-Mitaka 高可用之 Pacemaker+corosync+pcs 高可用集群

admin

介绍及特点
    Pacemaker：工作在资源分配层，提供资源管理器的功能
. J* ^2 U' _: r) m2 d9 Q8 h7 Q6 |    Corosync：提供集群的信息层功能，传递心跳信息和集群事务信息
    Pacemaker + Corosync 就可以实现高可用集群架构
1 _4 _2 k# Z! r) j/ F5 k
集群搭建
以下三个节点都需要执行：
, R: G+ z! s! g6 i, D" M
# yum install pcs -y
0 h* b4 A7 `! A# systemctl start  pcsd ; systemctl enable pcsd
# echo 'hacluster' | passwd --stdin hacluster
6 _0 O/ ~" u4 y& I7 p! D2 s, L# yum install haproxy  rsyslog -y
# echo 'net.ipv4.ip_nonlocal_bind = 1' >> /etc/sysctl.conf        # 启动服务的时候，允许忽视VIP的存在
" Z5 s3 R( s1 B# echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf        # 开启内核转发功能
) v/ ~" R7 P% c; w7 Q& w# sysctl -p
6 S4 v) h* |) Q/ x/ {9 A
; [. o) t; _/ C* o在任意节点创建用于haproxy监控Mariadb的用户
/ o% S, h0 F0 o: P8 |MariaDB [(none)]> CREATE USER 'haproxy'@'%' ;
3 w8 }3 U& w& p6 w配置haproxy用于负载均衡器
1 v' N+ u5 ^  [* ]& Q' J
9 m' R$ X' ?2 ?' @, @[root@controller1 ~]# egrep -v "^#|^$" /etc/haproxy/haproxy.cfg
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
( n& S- w& S) r  {" q; Z    user        haproxy
0 o3 q4 b* I9 f7 e& z/ w    group       haproxy
" {8 y' C' n  g$ N    daemon
! L( d7 e6 e+ C1 \    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
defaults
    mode                    http
: n6 ?3 i# D/ e    log                     global
$ G& a( Z  c9 E, W8 L9 u7 d    option                  httplog
    option                  dontlognull
( E( F4 @5 H" e4 p8 g. p    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
# e( B! Y; f) d; b7 ^  k    timeout http-request    10s
" ?7 i# p' k( Q& l, f' `$ y2 I    timeout queue           1m
    timeout connect         10s
1 F/ W' T' h1 ^8 i! q, `5 s& I, ^0 e    timeout client          1m
, _( [% V9 p/ |/ d    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 4000
listen galera_cluster
    mode tcp            
    bind 192.168.0.10:3306
3 J5 p0 y6 T' }    balance source
6 x) q2 k3 W% N" G: v    option mysql-check user haproxy
    server controller1 192.168.0.11:3306 check inter 2000 rise 3 fall 3 backup
1 i9 {% B9 B! w2 J% D, e2 @    server controller2 192.168.0.12:3306 check inter 2000 rise 3 fall 3
    server controller3 192.168.0.13:3306 check inter 2000 rise 3 fall 3 backup
1 C# U; D+ c4 @$ f* u$ Z/ u
6 p5 ~/ Z, T9 I1 z! Tlisten memcache_cluster
- R' u1 S9 s" M2 @- T/ W    mode tcp
8 A. |$ C; l% I$ t6 n$ U, b1 W- @    bind 192.168.0.10:11211
    balance source
    option tcplog
    server controller1 192.168.0.11:11211 check inter 2000 rise 3 fall 3
    server controller2 192.168.0.12:11211 check inter 2000 rise 3 fall 3
    server controller3 192.168.0.13:11211 check inter 2000 rise 3 fall 3

' Y' \0 d( l6 I, k0 O, c
3 q3 e+ d$ E* a- u$ G  t* ^9 N: O注意：
    （1）确保haproxy配置无误，建议首先修改ip和端口启动测试是否成功。
    （2）Mariadb-Galera和rabbitmq默认监听到 0.0.0.0 修改调整监听到本地 192.168.0.x
7 [+ P" I  X$ P) l5 @/ s: c    （3）将haproxy正确的配置拷贝到其他节点，无需手动启动haproxy服务
0 F! ^* W7 B# C7 |% P为haproxy配置日志（所有controller节点执行）：

# b+ {7 y: }4 L0 _, M. a9 o" E" Q# vim /etc/rsyslog.conf
2 g9 `' D4 N% Q7 R: T…
  _) G2 l6 q; a( p. M5 e1 H$ModLoad imudp
$UDPServerRun 514
…
1 _6 j% |* R" ^: dlocal2.*                                                /var/log/haproxy/haproxy.log
…
! F) q- }+ q" |) i* A0 D/ f
# mkdir -pv /var/log/haproxy/
: h9 |- p5 ~6 w' O/ o, Gmkdir: created directory ‘/var/log/haproxy/’
  W" X& g+ p3 x6 F" z
$ v$ `1 ~- t" F% _, G* d8 T% e5 e# systemctl restart rsyslog

1 |" p4 C: t' k* X. D4 f! C2 x1 o/ G3 \启动haproxy进行验证操作：
# p% h: K6 S' d
# systemctl start haproxy
[root@controller1 ~]# netstat -ntplu | grep ha
, @7 S( H: U, x; F# ^" ?& h; b( \tcp        0      0 192.168.0.10:3306       0.0.0.0:*               LISTEN      15467/haproxy      
tcp        0      0 192.168.0.10:11211      0.0.0.0:*               LISTEN      15467/haproxy      
udp        0      0 0.0.0.0:43268           0.0.0.0:*                           15466/haproxy

( A: N- f8 R/ I" ]验证成功，关闭haproxy
" x2 m, l+ f# X# ~# systemctl stop haproxy

9 i6 w/ L" ]. `# ]
: M' x7 }, ?( I0 R( f6 b在controller1节点上执行：
5 M4 R1 J/ k- S* K) {[root@controller1 ~]# pcs cluster auth controller1 controller2 controller3 -u hacluster -p hacluster --force
( [2 ~8 P/ z9 p3 k: m4 `controller3: Authorized
  B  ^# ^/ _, Q, Q  ~controller2: Authorized
controller1: Authorized
创建集群：
7 v! w6 z# Y9 P% D
[root@controller1 ~]# pcs cluster setup --name openstack-cluster controller1 controller2 controller3  --force
Destroying cluster on nodes: controller1, controller2, controller3...
controller3: Stopping Cluster (pacemaker)...
controller2: Stopping Cluster (pacemaker)...
; ^3 a7 N* {% o) f' V$ H( j+ pcontroller1: Stopping Cluster (pacemaker)...
controller3: Successfully destroyed cluster
. X$ f3 z1 y7 w0 @controller1: Successfully destroyed cluster
controller2: Successfully destroyed cluster

5 @/ l* p( }' R, c1 S, vSending 'pacemaker_remote authkey' to 'controller1', 'controller2', 'controller3'
3 l8 c2 S( {# [6 }controller3: successful distribution of the file 'pacemaker_remote authkey'
controller1: successful distribution of the file 'pacemaker_remote authkey'
controller2: successful distribution of the file 'pacemaker_remote authkey'
* Y: _7 L5 a8 o% v$ w- D. ?1 ^: dSending cluster config files to the nodes...
3 d' h$ \% d3 D( e6 O/ Lcontroller1: Succeeded
controller2: Succeeded
+ h0 J" J, r; Q! Q9 Dcontroller3: Succeeded

Synchronizing pcsd certificates on nodes controller1, controller2, controller3...
2 i4 p3 e- R0 `controller3: Success
4 l! l& @- ^. u% E& ^5 kcontroller2: Success
controller1: Success
* D' P) v5 E2 P, x% ^3 A5 pRestarting pcsd on the nodes in order to reload the certificates...
controller3: Success
controller2: Success
+ A" S% T+ y& A* ocontroller1: Success

  {8 g: r* i& u4 [% w+ H: p/ t2 N# {3 x启动集群的所有节点：

[root@controller1 ~]# pcs cluster start --all
controller2: Starting Cluster...
  ]3 U: ]0 ^, T  T3 N1 e2 p5 V- J0 ?controller1: Starting Cluster...
controller3: Starting Cluster...
5 Y! H1 r! u/ V  i( e[root@controller1 ~]# pcs cluster enable --all
controller1: Cluster Enabled
controller2: Cluster Enabled
5 z6 T* i# E  a) p9 ^5 Mcontroller3: Cluster Enabled
- Y1 h% l# A+ j: M* ^, t& g0 H
查看集群信息：

[root@controller1 ~]# pcs status
/ ~& W/ o  M- p/ SCluster name: openstack-cluster
WARNING: no stonith devices and stonith-enabled is not false
Stack: corosync
Current DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
/ W, D3 \: ~& R" A' s# Z+ MLast updated: Thu Nov 30 19:30:43 2017
9 x( A- Z' Z- [( YLast change: Thu Nov 30 19:30:17 2017 by hacluster via crmd on controller3

- u0 F, B6 _! ^, S4 ?+ W3 nodes configured
* m5 O' f, R' y+ C! p0 resources configured

Online: [ controller1 controller2 controller3 ]

No resources

8 O% T* ~! W% k' O8 a" `
; v9 N# m# G3 q" IDaemon Status:
  corosync: active/enabled
. K. z) ^2 b8 Q6 t1 ?  Z, D  pacemaker: active/enabled
$ t' A! Y! W9 s7 c4 Y% q5 ?  pcsd: active/enabled
3 O' s) O9 @4 y6 \9 x/ ?: E[root@controller1 ~]# pcs cluster status
Cluster Status:
Stack: corosync
2 D, ^" d' ^# i Current DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Thu Nov 30 19:30:52 2017
' n! I- @9 N' g7 \: O! Y Last change: Thu Nov 30 19:30:17 2017 by hacluster via crmd on controller3
/ d2 d9 o1 X, s' m8 a4 g 3 nodes configured
1 W0 d5 C0 W8 ?$ d! A 0 resources configured
- c2 A' e. g1 @! {) F+ c* o6 v. T* v
PCSD Status:
  controller2: Online
  controller3: Online
  controller1: Online

9 i% \$ w% N2 q& ^5 y9 `三个节点都在线
8 M# b7 k9 g5 y: e$ s默认的表决规则建议集群中的节点个数为奇数且不低于3。当集群只有2个节点，其中1个节点崩坏，由于不符合默认的表决规则， 集群资源不发生转移，集群整体仍不可用。no-quorum-policy="ignore"可以解决此双节点的问题，但不要用于生产环境。换句话说，生 产环境还是至少要3节点。
pe-warn-series-max、pe-input-series-max、pe-error-series-max代表日志深度。
cluster-recheck-interval是节点重新检查的频率。
, x- {0 |" I$ |8 L3 c, l[root@controller1 ~]#  pcs property set pe-warn-series-max=1000 pe-input-series-max=1000 pe-error-series-max=1000 cluster-recheck-interval=5min
禁用stonith：
$ g0 p- X# ~1 T2 [8 h# ?& R( Vstonith是一种能够接受指令断电的物理设备，环境无此设备，如果不关闭该选项，执行pcs命令总是含其报错信息。
% X# A, ]! n+ I' ?9 E- G[root@controller1 ~]# pcs property set stonith-enabled=false
二个节点时，忽略节点quorum功能：
[root@controller1 ~]# pcs property set no-quorum-policy=ignore
. C$ b0 q  p6 T/ c+ S验证集群配置信息
2 G, J: p( f; z/ B0 k9 h* f[root@controller1 ~]# crm_verify -L -V
为集群配置虚拟 ip
[root@controller1 ~]# pcs resource create ClusterIP ocf:heartbeat:IPaddr2 \
ip="192.168.0.10" cidr_netmask=32 nic=eno16777736 op monitor interval=30s
$ O8 p) o+ M( O9 d, R到此，Pacemaker+corosync 是为 haproxy服务的，添加haproxy资源到pacemaker集群
6 z2 X/ @8 w1 b0 p1 `[root@controller1 ~]# pcs resource create lb-haproxy systemd:haproxy --clone
* |* g5 f" Z/ i5 u; o0 S3 k. N8 U说明：创建克隆资源，克隆的资源会在全部节点启动。这里haproxy会在三个节点自动启动。
4 D1 x7 @( e. i- K$ W查看Pacemaker资源情况
[root@controller1 ~]# pcs resource
ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller1        # 心跳的资源绑定在第三个节点的
Clone Set: lb-haproxy-clone [lb-haproxy]        # haproxy克隆资源
     Started: [ controller1 controller2 controller3 ]
注意：这里一定要进行资源绑定，否则每个节点都会启动haproxy，造成访问混乱
将这两个资源绑定到同一个节点上
3 a, S- ~  E+ M* l[root@controller1 ~]# pcs constraint colocation add lb-haproxy-clone ClusterIP INFINITY
- Q# e& A- U3 v4 g绑定成功
[root@controller1 ~]# pcs resource
2 |0 U& h& y/ H' m$ ^ ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller3
Clone Set: lb-haproxy-clone [lb-haproxy]
     Started: [ controller1]
! v: n; F) W. g/ q( \0 q/ T     Stopped: [ controller2 controller3 ]
配置资源的启动顺序，先启动vip，然后haproxy再启动，因为haproxy是监听到vip
[root@controller1 ~]# pcs constraint order ClusterIP then lb-haproxy-clone
手动指定资源到某个默认节点，因为两个资源绑定关系，移动一个资源，另一个资源自动转移。
) M8 w8 ?: X' n! P3 t3 t* D' H
' ~% p; L; r3 L[root@controller1 ~]# pcs constraint location ClusterIP prefers controller1
[root@controller1 ~]# pcs resource
ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller1
Clone Set: lb-haproxy-clone [lb-haproxy]
  u& Z( S! k( Z$ Y$ s! x' m     Started: [ controller1 ]
     Stopped: [ controller2 controller3 ]
[root@controller1 ~]# pcs resource defaults resource-stickiness=100        # 设置资源粘性，防止自动切回造成集群不稳定
  T1 @3 ?& i& A5 j! Z现在vip已经绑定到controller1节点
" J$ ~$ Q' Z2 d6 a2 o8 J* Y" A[root@controller1 ~]# ip a | grep global
    inet 192.168.0.11/24 brd 192.168.0.255 scope global eno16777736
$ `! N" c/ \5 P. k4 @; M    inet 192.168.0.10/32 brd 192.168.0.255 scope global eno16777736
    inet 192.168.118.11/24 brd 192.168.118.255 scope global eno33554992

尝试通过vip连接数据库
Controller1:
- r# P  L& O6 \4 i3 S
# t0 `' M3 h& M2 f7 w7 U[root@controller1 haproxy]# mysql -ugalera -pgalera -h 192.168.0.10

5 k) x  O6 ]" ?: |
& H0 {. _, z1 _6 z4 | Controller2:
( e$ u' |4 L) x$ `4 o
　
高可用配置成功。

* Q7 ?, h3 c& o+ D测试高可用是否正常
2 L( C2 s  i- |! d2 C" z在controller1节点上直接执行 poweroff -f
2 i: l- o3 W/ I/ i  h# V' E[root@controller1 ~]# poweroff -f
vip很快就转移到controller2节点上

6 ?+ s; v( P: l再次尝试访问数据库


; i: e6 k+ Y+ V6 I; x6 o 无任何问题，测试成功。
查看集群信息：
4 N1 c% Y5 ^0 u) H& y# s
( ^4 }% v1 I/ L[root@controller2 ~]# pcs status
2 c, o3 z0 `! PCluster name: openstack-cluster
Stack: corosync
! b! K" \; o* _6 S  a. W0 E) [Current DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Thu Nov 30 23:57:28 2017
1 V, k+ u) {' B* A- n7 V& KLast change: Thu Nov 30 23:54:11 2017 by root via crm_attribute on controller1
+ ~0 g! m" o& g8 w" e! e6 V& ^
2 G/ k* K2 {  i1 o) {4 {3 nodes configured
4 resources configured

Online: [ controller2 controller3 ]
# }8 D5 u4 v. f% y  ^! Y2 e3 N6 pOFFLINE: [ controller1 ]            # controller1 已经下线

Full list of resources:

ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller2
3 v3 ]+ ^0 P! Z( H6 |# H Clone Set: lb-haproxy-clone [lb-haproxy]
* {5 H, y2 S) Z) e" N     Started: [ controller2 ]
     Stopped: [ controller1 controller3 ]

Daemon Status:
3 f$ a- f6 {1 E+ O8 }% f) D  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled