Openstack-Mitaka 高可用之 Pacemaker+corosync+pcs 高可用集群

admin

介绍及特点
3 _9 q* G$ o; ?    Pacemaker：工作在资源分配层，提供资源管理器的功能
    Corosync：提供集群的信息层功能，传递心跳信息和集群事务信息
6 K7 r  C6 C; Q+ d7 D: Z5 e3 n# n( N    Pacemaker + Corosync 就可以实现高可用集群架构
1 n& C. f/ U8 }; Z; ^$ u& W
集群搭建
以下三个节点都需要执行：

# yum install pcs -y
* i) |3 J" B, Y9 k( J9 W# systemctl start  pcsd ; systemctl enable pcsd
# echo 'hacluster' | passwd --stdin hacluster
# yum install haproxy  rsyslog -y
# echo 'net.ipv4.ip_nonlocal_bind = 1' >> /etc/sysctl.conf        # 启动服务的时候，允许忽视VIP的存在
# echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf        # 开启内核转发功能
' h# z. d, i; G( P# sysctl -p
$ K. g7 ?% m& O. A5 Q
4 B. S% K" x' Y7 u  w2 X2 X' _2 F2 L3 N在任意节点创建用于haproxy监控Mariadb的用户
% ~7 C' L1 v% O: xMariaDB [(none)]> CREATE USER 'haproxy'@'%' ;
' M6 @+ u0 U  f( b4 p: O配置haproxy用于负载均衡器

  }: W3 S7 k! p3 B+ u5 C( t[root@controller1 ~]# egrep -v "^#|^$" /etc/haproxy/haproxy.cfg
    log         127.0.0.1 local2
! b, p. X8 z# p: d# r1 E    chroot      /var/lib/haproxy
" x) i/ d7 g/ K' w    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    # turn on stats unix socket
# s' A6 f& s" g( a% r6 F+ e; s    stats socket /var/lib/haproxy/stats
defaults
    mode                    http
9 \% K0 }" Y5 M+ s+ N: }' `: J7 O    log                     global
# \* r5 @: x$ ]: V8 r1 k) e    option                  httplog
    option                  dontlognull
  w2 s9 S9 U- G% z    option http-server-close
& j5 }) o9 Z/ f" H4 v    option forwardfor       except 127.0.0.0/8
    option                  redispatch
+ E/ M+ c* `& T- R+ c( X    retries                 3
    timeout http-request    10s
; z, c; g/ z0 n( U6 n    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
! D' @- u; Q8 Z( w( J, B# P    maxconn                 4000
listen galera_cluster
+ p4 a+ A* H3 M/ `3 Z    mode tcp            
; w/ u$ Q: N; f9 c. R    bind 192.168.0.10:3306
    balance source
( Z- o( e9 F! d/ m    option mysql-check user haproxy
" B1 @) y8 l7 C  x' r6 c    server controller1 192.168.0.11:3306 check inter 2000 rise 3 fall 3 backup
* p1 G+ h6 y4 s+ O/ H    server controller2 192.168.0.12:3306 check inter 2000 rise 3 fall 3
    server controller3 192.168.0.13:3306 check inter 2000 rise 3 fall 3 backup
7 X! I) S# J; W2 D. v
* U2 d8 q4 V& s" x& P# ^4 ylisten memcache_cluster
$ P" C0 T- i5 k% D" Z* z* x7 ~    mode tcp
    bind 192.168.0.10:11211
    balance source
/ y$ I2 i! u) K8 s    option tcplog
* s$ v% f" ]3 T. S3 ]    server controller1 192.168.0.11:11211 check inter 2000 rise 3 fall 3
0 e5 H# H. i  L0 p6 D' A    server controller2 192.168.0.12:11211 check inter 2000 rise 3 fall 3
5 _& k, ?, N& w1 ]/ \' K    server controller3 192.168.0.13:11211 check inter 2000 rise 3 fall 3

- {; T# e7 F# F: N$ J5 [5 {# a! s
注意：
    （1）确保haproxy配置无误，建议首先修改ip和端口启动测试是否成功。
/ `; }* Y2 u0 k+ v    （2）Mariadb-Galera和rabbitmq默认监听到 0.0.0.0 修改调整监听到本地 192.168.0.x
    （3）将haproxy正确的配置拷贝到其他节点，无需手动启动haproxy服务
6 _; j( T+ ^* D为haproxy配置日志（所有controller节点执行）：
6 _, p$ W9 }4 Q
# vim /etc/rsyslog.conf
…
$ModLoad imudp
9 D8 s# V, C; Z. L" ?. u$UDPServerRun 514
$ H/ k; w3 T) Z5 W…
local2.*                                                /var/log/haproxy/haproxy.log
…
4 G% h3 [3 _0 N
! Q8 U6 p5 z# Z( n# mkdir -pv /var/log/haproxy/
mkdir: created directory ‘/var/log/haproxy/’
1 J4 ~" w) y: @2 e$ n+ ?* h  |
( j' c' {$ l# W2 p2 G) b( B# systemctl restart rsyslog

启动haproxy进行验证操作：

6 K/ j- m8 ^1 ?) U6 R1 W4 ~. W# systemctl start haproxy
[root@controller1 ~]# netstat -ntplu | grep ha
tcp        0      0 192.168.0.10:3306       0.0.0.0:*               LISTEN      15467/haproxy      
2 Y4 X' a4 i$ S0 Q- E1 G1 A8 gtcp        0      0 192.168.0.10:11211      0.0.0.0:*               LISTEN      15467/haproxy      
" ~  b( A0 v4 ^+ zudp        0      0 0.0.0.0:43268           0.0.0.0:*                           15466/haproxy
1 h4 R6 a$ `' `6 Y& x
/ H2 Y0 h, s* T4 L0 l验证成功，关闭haproxy
# systemctl stop haproxy


在controller1节点上执行：
[root@controller1 ~]# pcs cluster auth controller1 controller2 controller3 -u hacluster -p hacluster --force
6 ]" U1 T. E5 l1 ~$ a; Ncontroller3: Authorized
controller2: Authorized
1 ?: Z7 y) L* Y, b4 ucontroller1: Authorized
创建集群：
3 U/ m0 `" y  i# o0 Q* V& L
, Y; G' T* I% L) f. i[root@controller1 ~]# pcs cluster setup --name openstack-cluster controller1 controller2 controller3  --force
. [$ N  B- N: f5 V0 l$ B$ jDestroying cluster on nodes: controller1, controller2, controller3...
controller3: Stopping Cluster (pacemaker)...
controller2: Stopping Cluster (pacemaker)...
+ ^9 A& o, y# s- zcontroller1: Stopping Cluster (pacemaker)...
controller3: Successfully destroyed cluster
controller1: Successfully destroyed cluster
. X6 B: v# v; k2 i, @$ scontroller2: Successfully destroyed cluster
" W1 J7 k/ r6 W
1 X; \- c9 i* bSending 'pacemaker_remote authkey' to 'controller1', 'controller2', 'controller3'
+ o, Z: v( h3 h1 r2 Zcontroller3: successful distribution of the file 'pacemaker_remote authkey'
controller1: successful distribution of the file 'pacemaker_remote authkey'
' C: i8 a( S9 C' Q( @6 s% ]. Y+ ccontroller2: successful distribution of the file 'pacemaker_remote authkey'
Sending cluster config files to the nodes...
- p4 _% n! z$ \5 d) B1 qcontroller1: Succeeded
controller2: Succeeded
controller3: Succeeded
/ \7 V+ w2 \' `! n& M: N* W8 c
Synchronizing pcsd certificates on nodes controller1, controller2, controller3...
controller3: Success
" F0 \! G) P" y# G- y8 Zcontroller2: Success
controller1: Success
Restarting pcsd on the nodes in order to reload the certificates...
, V, Q3 j+ N$ A9 W% Q, ncontroller3: Success
controller2: Success
* E- v- e* _& F- S# m/ n6 ^' Lcontroller1: Success
( m! ^; I7 L. a
' ~) E3 P" \/ }) U, B* E启动集群的所有节点：
) P2 ]( H9 M! L4 F( X
$ \7 Y$ E3 N6 g6 V, @' \- v[root@controller1 ~]# pcs cluster start --all
controller2: Starting Cluster...
controller1: Starting Cluster...
/ g# s+ E. o* |5 W5 Fcontroller3: Starting Cluster...
[root@controller1 ~]# pcs cluster enable --all
$ }" n/ v9 f* m% a: T6 T: Ncontroller1: Cluster Enabled
controller2: Cluster Enabled
/ b: S# E0 d' X. |) A! vcontroller3: Cluster Enabled

查看集群信息：
3 J  _* ^  r: N% q3 Z
[root@controller1 ~]# pcs status
# w% n* t5 F' H5 }0 p9 RCluster name: openstack-cluster
WARNING: no stonith devices and stonith-enabled is not false
Stack: corosync
Current DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Thu Nov 30 19:30:43 2017
$ K# S% [7 i! k: N2 K' y4 l9 mLast change: Thu Nov 30 19:30:17 2017 by hacluster via crmd on controller3
. N* f2 r% \: ~
( K0 J( p7 J# k+ [' u# o2 b+ P: U3 nodes configured
/ N5 c4 a2 \; g) n' {2 |4 U" h0 resources configured
2 a8 E0 W5 Z% G8 [
Online: [ controller1 controller2 controller3 ]

" B( P8 y+ ^; j% Z! Z5 xNo resources


Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
1 D& E& j/ h! b( N[root@controller1 ~]# pcs cluster status
Cluster Status:
Stack: corosync
- g- p' D+ w$ l Current DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Thu Nov 30 19:30:52 2017
Last change: Thu Nov 30 19:30:17 2017 by hacluster via crmd on controller3
3 nodes configured
4 k& {5 b& v4 S4 }- o  `# H 0 resources configured

PCSD Status:
' j" D6 Z1 D; Q3 B6 i% B( P  controller2: Online
  controller3: Online
  controller1: Online

/ n+ D5 E, Y% U8 X( b三个节点都在线
默认的表决规则建议集群中的节点个数为奇数且不低于3。当集群只有2个节点，其中1个节点崩坏，由于不符合默认的表决规则， 集群资源不发生转移，集群整体仍不可用。no-quorum-policy="ignore"可以解决此双节点的问题，但不要用于生产环境。换句话说，生 产环境还是至少要3节点。
pe-warn-series-max、pe-input-series-max、pe-error-series-max代表日志深度。
  _9 _* r6 n8 p% T9 o- s3 [/ Bcluster-recheck-interval是节点重新检查的频率。
[root@controller1 ~]#  pcs property set pe-warn-series-max=1000 pe-input-series-max=1000 pe-error-series-max=1000 cluster-recheck-interval=5min
# g7 j6 ]. |4 N4 w2 u; U禁用stonith：
7 j, C' I" |+ _( U4 Ystonith是一种能够接受指令断电的物理设备，环境无此设备，如果不关闭该选项，执行pcs命令总是含其报错信息。
! ^; N% d9 F: t+ S; @[root@controller1 ~]# pcs property set stonith-enabled=false
二个节点时，忽略节点quorum功能：
, ?+ h: D& V0 w' b7 ?9 ~[root@controller1 ~]# pcs property set no-quorum-policy=ignore
" l" l& F1 i3 ^7 x# U5 N+ k验证集群配置信息
[root@controller1 ~]# crm_verify -L -V
3 k( h$ @  ?6 q, Q为集群配置虚拟 ip
3 d! l/ l. Y! ]& T( s[root@controller1 ~]# pcs resource create ClusterIP ocf:heartbeat:IPaddr2 \
  r5 b# H; B( i ip="192.168.0.10" cidr_netmask=32 nic=eno16777736 op monitor interval=30s
到此，Pacemaker+corosync 是为 haproxy服务的，添加haproxy资源到pacemaker集群
[root@controller1 ~]# pcs resource create lb-haproxy systemd:haproxy --clone
- t2 q$ `9 R! u0 t说明：创建克隆资源，克隆的资源会在全部节点启动。这里haproxy会在三个节点自动启动。
6 U( S8 S$ A9 M- R- y7 F查看Pacemaker资源情况
$ K7 M- u8 O) f/ L; R[root@controller1 ~]# pcs resource
ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller1        # 心跳的资源绑定在第三个节点的
Clone Set: lb-haproxy-clone [lb-haproxy]        # haproxy克隆资源
* x7 e5 C# ^3 N     Started: [ controller1 controller2 controller3 ]
2 H8 L. l' b4 y- {: W6 ~注意：这里一定要进行资源绑定，否则每个节点都会启动haproxy，造成访问混乱
  a# V, r# D1 ^# W将这两个资源绑定到同一个节点上
[root@controller1 ~]# pcs constraint colocation add lb-haproxy-clone ClusterIP INFINITY
绑定成功
0 _( Z# f+ L& C, ^[root@controller1 ~]# pcs resource
ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller3
5 w# [0 E# h# ?; H7 S" f+ T Clone Set: lb-haproxy-clone [lb-haproxy]
/ [8 Q7 }7 ?& ^2 }1 s     Started: [ controller1]
1 |7 u- g6 d" V3 H) U& \     Stopped: [ controller2 controller3 ]
9 V5 X: T3 H! q3 h  l配置资源的启动顺序，先启动vip，然后haproxy再启动，因为haproxy是监听到vip
3 j1 |- U, r0 a4 c; L[root@controller1 ~]# pcs constraint order ClusterIP then lb-haproxy-clone
手动指定资源到某个默认节点，因为两个资源绑定关系，移动一个资源，另一个资源自动转移。
! T" V. l% x, w" s1 Q7 I& e, I% g
[root@controller1 ~]# pcs constraint location ClusterIP prefers controller1
% k* S$ v* W3 T* P! {& @$ M[root@controller1 ~]# pcs resource
ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller1
$ m% f: {( u) k5 D Clone Set: lb-haproxy-clone [lb-haproxy]
     Started: [ controller1 ]
5 h2 i6 k; J" S     Stopped: [ controller2 controller3 ]
( j) Q7 m. i! |: |[root@controller1 ~]# pcs resource defaults resource-stickiness=100        # 设置资源粘性，防止自动切回造成集群不稳定
+ G: k% |) o( v+ t8 l现在vip已经绑定到controller1节点
" |. k. f" [5 r7 q; L[root@controller1 ~]# ip a | grep global
3 f; [' D3 ]% n' l3 S    inet 192.168.0.11/24 brd 192.168.0.255 scope global eno16777736
* H" ]6 }% J  d( F    inet 192.168.0.10/32 brd 192.168.0.255 scope global eno16777736
+ B& y& K! O7 u    inet 192.168.118.11/24 brd 192.168.118.255 scope global eno33554992
; [1 G/ G' X3 m( Q0 b: v2 C
& p+ n, C; L" Q+ n# g- _  B+ c尝试通过vip连接数据库
# a# k* K/ @8 X+ m/ lController1:
- @$ p+ a' P( T: Z5 }, H, Y
[root@controller1 haproxy]# mysql -ugalera -pgalera -h 192.168.0.10


Controller2:
, a0 E7 N: R3 u% K) D
　
高可用配置成功。

9 v6 |2 S$ w9 j2 o& ~测试高可用是否正常
8 f3 v* E* d. u  W6 q4 J% r在controller1节点上直接执行 poweroff -f
6 t* S/ U* r. \[root@controller1 ~]# poweroff -f
vip很快就转移到controller2节点上
6 G" A7 a- g- w6 g* h) y5 U
再次尝试访问数据库


无任何问题，测试成功。
4 @& y& j3 I7 V1 d* V8 W6 l查看集群信息：

[root@controller2 ~]# pcs status
Cluster name: openstack-cluster
Stack: corosync
4 g: G! s5 u8 t2 [Current DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
& f  h/ Y4 Y' g: e* J3 vLast updated: Thu Nov 30 23:57:28 2017
Last change: Thu Nov 30 23:54:11 2017 by root via crm_attribute on controller1

6 s! F4 l5 I/ r7 T0 ]; m3 nodes configured
( i( k7 x; y. V! h3 [' s4 T5 q4 resources configured
; F: P# V# M4 `4 [: k, l8 B/ [1 N. r: h
Online: [ controller2 controller3 ]
, z8 X' H! M, i; w$ lOFFLINE: [ controller1 ]            # controller1 已经下线
7 m% |& A, E& [
Full list of resources:
9 [2 X( ^, _, D- b, V  T, S2 e
# s4 [7 F& N! y7 ]$ u; T! t ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller2
* E: ]& K: ~: M/ V* { Clone Set: lb-haproxy-clone [lb-haproxy]
1 W% Z5 c/ I7 T# c6 _0 l- @1 w     Started: [ controller2 ]
$ C! h, l$ ~5 Y2 ?6 E) e     Stopped: [ controller1 controller3 ]
1 g2 N0 _0 E* F8 c
Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
5 w+ [" X6 R7 S; X  pcsd: active/enabled