记一次mysq(mariadb)l数据库恢复成功过程 recover参数，rsync数据同步，数据恢复失败后重新覆盖文件恢复

admin

[root@controller1 _data]# kolla-ansible -i multinode mariadb_recovery
# M5 @' R& a! `Attempting to restart mariadb cluster : ansible-playbook -i multinode -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla  -e action=deploy -e common_run=true /usr/share/kolla-ansible/ansible/mariadb_recovery.yml
& A. K" e5 i' H! m4 [5 a2 S/usr/lib/python2.7/site-packages/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
, S! v' r% k# a& x' ?$ v- t  from cryptography.exceptions import InvalidSignature
- i7 I5 B3 k% e3 ^: T0 r3 C[WARNING]: Unable to parse /var/lib/docker/volumes/mariadb/_data/multinode as an inventory source
; R& E1 V# X1 F5 j[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
# H; r! U* z; K* @/ r4 O& `: M[WARNING]: Could not match supplied host pattern, ignoring: mariadb

PLAY [Detect openstack_release variable] **************************************************************************************************************************
8 l$ Y8 Y: D7 ]# Yskipping: no hosts matched
PLAY [Recover mariadb] ********************************************************************************************************************************************
7 `0 Q$ N! C5 j- W) X) zskipping: no hosts matched
PLAY RECAP ********************************************************************************************************************************************************
+ i: n2 E  w) p; ]' G$ P[root@controller1 _data]# cd
$ a# L% k9 N3 U: I* y# B[root@controller1 ~]# kolla-ansible -i multinode mariadb_recovery
Attempting to restart mariadb cluster : ansible-playbook -i multinode -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla  -e action=deploy -e common_run=true /usr/share/kolla-ansible/ansible/mariadb_recovery.yml
3 J: |$ ?+ t) I! {$ |" f/usr/lib/python2.7/site-packages/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography.exceptions import InvalidSignature
$ F- v. L! f7 \& R, D8 s& H[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
& B- ?5 |( H3 _! u0 H  A' l[WARNING]: Found variable using reserved name: action
9 c1 n- @9 H6 C2 P  {% R; b8 C; I- f1 E
' d2 R3 c6 p9 _  I& }  A( R8 I  CPLAY [Detect openstack_release variable] **************************************************************************************************************************
( t6 X, X! ?% m* A; u1 w* u) q* vTASK [Get current kolla-ansible version number] *******************************************************************************************************************
) K( A* d5 {8 A! h4 yskipping: [controller1]
3 x2 i& c* |: w1 Oskipping: [controller2]
. [! X5 f) S8 P5 N) `( w/ l! Q+ cskipping: [controller3]
$ m' p7 r- E1 {, yTASK [Set openstack_release variable] *****************************************************************************************************************************
! m- Z  L) b6 W. L" l9 s# Sskipping: [controller1]
skipping: [controller2]
skipping: [controller3]
PLAY [Recover mariadb] ********************************************************************************************************************************************
' G  ~' n% c# t# e. ^) GTASK [Gathering Facts] ********************************************************************************************************************************************
7 W' ^- e" Z# T! m1 d% o" f! _  Cok: [controller1]
2 |- c# ]' a1 _8 Sok: [controller3]
  [* u8 R3 k+ E" L0 e/ Gok: [controller2]
! ^! }& X5 s0 a. Y. gTASK [common : include] *******************************************************************************************************************************************
skipping: [controller1]
skipping: [controller2]
skipping: [controller3]
TASK [common : Registering common role has run] *******************************************************************************************************************
2 J0 `' m" r% w: t5 f) g7 _1 V# m. Bskipping: [controller1]
skipping: [controller2]
skipping: [controller3]
5 y% B1 P6 x/ H2 c9 Z, E+ ITASK [mariadb : include] ******************************************************************************************************************************************
included: /usr/share/kolla-ansible/ansible/roles/mariadb/tasks/deploy.yml for controller1, controller2, controller3
TASK [mariadb : include] ******************************************************************************************************************************************
& r9 K+ F- T( V4 d: d6 K, r+ cincluded: /usr/share/kolla-ansible/ansible/roles/mariadb/tasks/config.yml for controller1, controller2, controller3
! K! i8 q2 v4 y: _+ JTASK [mariadb : Ensuring config directories exist] ****************************************************************************************************************
ok: [controller2] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})
ok: [controller3] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})
- t! M: ^( `& Iok: [controller1] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})
3 S/ D% ^- s- b* e9 ^* aTASK [mariadb : Copying over config.json files for services] ******************************************************************************************************
ok: [controller2]
ok: [controller3]
ok: [controller1]
TASK [mariadb : Copying over galera.cnf] **************************************************************************************************************************
8 Z* I) S: k& h: J/ e3 a/ H[WARNING]: _execute_module no longer honors the tmp parameter. Action plugins should set self._connection._shell.tmpdir to share the tmpdir
[WARNING]: _execute_module no longer honors the tmp parameter. Action plugins should set self._connection._shell.tmpdir to share the tmpdir
( ~( I) g! @8 \; g[WARNING]: _execute_module no longer honors the tmp parameter. Action plugins should set self._connection._shell.tmpdir to share the tmpdir
ok: [controller3]
! A; p# \6 j0 N; b! |ok: [controller2]
ok: [controller1]
TASK [mariadb : Copying over wsrep-notify.sh] *********************************************************************************************************************
7 C( C0 l( C& \* gok: [controller2] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})
ok: [controller1] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})
% i; ?8 i- E: G0 a# Gok: [controller3] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})
" _9 u6 I& ]8 [. V5 l/ f6 D0 D; U1 jTASK [mariadb : Check mariadb containers] *************************************************************************************************************************
1 H" ?# Z' E6 M8 ychanged: [controller3] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})
changed: [controller1] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})
- r; ?9 ]8 t. J+ I; G2 Y% wchanged: [controller2] => (item={'value': {u'enabled': True, u'image': u'kolla/centos-binary-mariadb:queens', u'group': u'mariadb', u'volumes': [u'/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro', u'/etc/localtime:/etc/localtime:ro', u'mariadb:/var/lib/mysql', u'kolla_logs:/var/log/kolla/'], u'container_name': u'mariadb'}, 'key': u'mariadb'})
TASK [mariadb : include] ******************************************************************************************************************************************
4 P9 M+ Z+ c* [# @: F$ Iincluded: /usr/share/kolla-ansible/ansible/roles/mariadb/tasks/bootstrap.yml for controller1, controller2, controller3
3 \4 Z! \3 G- q  ATASK [mariadb : set_fact] *****************************************************************************************************************************************
ok: [controller1]
( A# ?) ]( ~) W* W3 L2 |  h6 Sok: [controller2]
  O) O' v" U% |$ I' F4 Tok: [controller3]
4 K  C# r) `' nTASK [mariadb : include] ******************************************************************************************************************************************
included: /usr/share/kolla-ansible/ansible/roles/mariadb/tasks/lookup_cluster.yml for controller1, controller2, controller3
TASK [mariadb : Cleaning up temp file on localhost] ***************************************************************************************************************
3 e% C4 |1 q8 Dok: [controller1 -> localhost]
TASK [mariadb : Creating temp file on localhost] ******************************************************************************************************************
ok: [controller1 -> localhost]
TASK [mariadb : Creating mariadb volume] **************************************************************************************************************************
ok: [controller1]
ok: [controller2]
ok: [controller3]
: K& `8 h$ A" v; TTASK [mariadb : Writing hostname of host with existing cluster files to temp file] ********************************************************************************
' q; M( I- S% B' `' \( A- B% {[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version
2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
' u5 y- ^& B5 t: O! X[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version
2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
5 e! V* e7 }) N! m0 P2 n3 Y[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version
2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
2 L" ~9 J1 l' U1 q6 ?/ uok: [controller1 -> localhost]
ok: [controller2 -> localhost]
5 Z5 M5 _4 ]% F3 a( nok: [controller3 -> localhost]
TASK [mariadb : Registering host from temp file] ******************************************************************************************************************
  r0 W) w6 Z: G" r# i8 K2 Iok: [controller1]
' n6 s$ y# P: [$ w6 Iok: [controller2]
# [- W, n' E( N' m0 q$ Vok: [controller3]
# O$ V0 H& h3 G& D  w( p$ W. WTASK [mariadb : Cleaning up temp file on localhost] ***************************************************************************************************************
ok: [controller1 -> localhost]
0 z/ P( u5 L  I) O% }9 a: P& [% JTASK [mariadb : include] ******************************************************************************************************************************************
skipping: [controller1]
6 c# V* H1 A" C7 X3 _skipping: [controller2]
5 [5 Q5 W: R! |! v4 X' H% eskipping: [controller3]
TASK [mariadb : include] ******************************************************************************************************************************************
included: /usr/share/kolla-ansible/ansible/roles/mariadb/tasks/recover_cluster.yml for controller1, controller2, controller3
& F4 Z5 P. V) B9 g+ x; WTASK [mariadb : fail] *********************************************************************************************************************************************
/ q3 I9 U: j2 q- jskipping: [controller1]
0 @' V5 t% ]! _/ d0 [skipping: [controller2]
" I) s  P8 K8 G. w: a! W; Y7 Gskipping: [controller3]
TASK [mariadb : Cleaning up temp file on mariadb hosts] ***********************************************************************************************************
ok: [controller2]
/ s- B' B. j* N3 M( Eok: [controller1]
ok: [controller3]
TASK [mariadb : Cleaning up temp file on localhost] ***************************************************************************************************************
+ U% v4 w, T6 u# r0 m" n  k/ Zok: [controller1 -> localhost]
TASK [mariadb : Stop MariaDB containers] **************************************************************************************************************************
7 p% Y4 B9 |; \2 N* Wok: [controller2]
/ t# `% I) @' B: Aok: [controller1]
ok: [controller3]
TASK [mariadb : Run MariaDB wsrep recovery] ***********************************************************************************************************************
4 N. `. G4 V9 Xchanged: [controller1]
1 q; ?$ K; q# }changed: [controller2]
1 Y6 }" m6 \: J/ i) z- O( Z2 Kchanged: [controller3]
TASK [mariadb : Copying MariaDB log file to /tmp] *****************************************************************************************************************
changed: [controller3]
2 s! h+ c$ ]6 r! T; xchanged: [controller2]
6 T6 ?: ?0 t: Jchanged: [controller1]
$ }7 o- D  b. G, o/ uTASK [mariadb : Get MariaDB wsrep recovery seqno] *****************************************************************************************************************
changed: [controller2]
) H* u7 W# j3 ^% ichanged: [controller1]
changed: [controller3]
& D6 Z% o/ N. o9 A% ^4 u: jTASK [mariadb : Removing MariaDB log file from /tmp] **************************************************************************************************************
1 `/ O+ B% c1 A8 M2 f% ^ok: [controller2]
# x: |5 d6 {2 N" T0 q4 Zok: [controller1]
/ Y4 N7 H2 L- ?6 z0 qok: [controller3]
TASK [mariadb : Registering MariaDB seqno variable] ***************************************************************************************************************
4 ~$ Q/ C. x" X) D/ G! ^* tok: [controller1]
ok: [controller2]
6 U! [$ G# V# @% v$ z7 Gok: [controller3]
9 l5 g" H+ J# k0 aTASK [mariadb : Comparing seqno value on all mariadb hosts] *******************************************************************************************************
$ f5 _* _9 s" I3 cok: [controller2] => (item=controller1)
: R1 e+ m# [9 b7 B! r# X  Wok: [controller1] => (item=controller1)
ok: [controller3] => (item=controller1)
) w" o8 Z3 S" l2 B. K) t) bok: [controller2] => (item=controller2)
ok: [controller3] => (item=controller2)
ok: [controller1] => (item=controller2)
; O4 K: {3 b9 ~1 A$ [ok: [controller2] => (item=controller3)
ok: [controller3] => (item=controller3)
- G5 ]! u( T6 ?* @; lok: [controller1] => (item=controller3)
. h& M/ T( i" V; u- tTASK [mariadb : Writing hostname of host with the largest seqno to temp file] *************************************************************************************
skipping: [controller3]
ok: [controller1 -> localhost]
ok: [controller2 -> localhost]
4 X, Y1 V  N7 VTASK [mariadb : Registering mariadb_recover_inventory_name from temp file] ****************************************************************************************
0 S* P3 J! h: d! n$ l& ?" {ok: [controller1]
, y2 ?% j) h$ A% R$ @2 Rok: [controller2]
; ?- z2 q% i! j+ J+ E, t3 Iok: [controller3]
5 |# d5 B* p$ b# X( CTASK [mariadb : set_fact] *****************************************************************************************************************************************
changed: [controller1]
7 }8 G; h* r6 _+ @4 A! ychanged: [controller2]
changed: [controller3]
$ V+ Q" v  n3 @! _TASK [mariadb : Copying grastate.dat file from MariaDB container in bootstrap host] *******************************************************************************
skipping: [controller1]
skipping: [controller3]
ok: [controller2]
TASK [mariadb : Set grastate.dat file from MariaDB container in bootstrap host] ***********************************************************************************
skipping: [controller1]
; U1 U+ i! S& j* K# q; Q! d& Uskipping: [controller3]
6 Y4 Y. T& j; A. H; Zchanged: [controller2]
2 J8 }5 j6 \* Y, rTASK [mariadb : Copying grastate.dat file to mariadb container] ***************************************************************************************************
) N. E- S$ p6 h* J9 B  @) Kskipping: [controller1]
skipping: [controller3]
ok: [controller2]
TASK [mariadb : Starting first MariaDB container] *****************************************************************************************************************
skipping: [controller1]
5 R. X. @8 s$ \+ U. z( Oskipping: [controller3]
- ?" e( r! e4 u! x& g/ achanged: [controller2]
TASK [mariadb : Wait for first MariaDB container] *****************************************************************************************************************
skipping: [controller1]
+ G. u- l4 I: ?) H9 r9 o5 bskipping: [controller3]
1 D# v& A0 h3 Z4 D[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version
( {, N) o- s+ x8 p- E6 h8 l, b9 J2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
1 M0 ?$ N$ g7 |$ q4 g0 U1 tFAILED - RETRYING: Wait for first MariaDB container (10 retries left).
: h* H4 |" q, ?! j9 d* @; Qok: [controller2]
TASK [mariadb : Set first MariaDB container as primary] ***********************************************************************************************************
skipping: [controller1]
skipping: [controller3]
' M9 Q8 b# D; f, _1 [/ V/ ^$ |changed: [controller2]
5 w7 C; i# a$ x& q4 I! dTASK [mariadb : Wait for MariaDB to become operational] ***********************************************************************************************************
9 B7 z5 m0 C4 k7 z1 q0 E& }skipping: [controller1]
skipping: [controller3]
' `% s( U! \; ?/ Y# h3 ^  z5 jok: [controller2]
TASK [mariadb : Restart slave MariaDB container] ******************************************************************************************************************
1 M8 k9 Q2 e) e. h! w) ?# Zskipping: [controller2]
changed: [controller3]
2 m  V6 _7 }6 Z1 {changed: [controller1]
# O5 q! o/ }1 o, dTASK [mariadb : Wait for slave MariaDB] ***************************************************************************************************************************
9 M4 Y# `6 ^- eskipping: [controller2]
[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version
2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
# d# u8 Y% r9 Q, h7 Z. q8 P: zFAILED - RETRYING: Wait for slave MariaDB (10 retries left).
# h+ l5 P) I& z! l* C& V' ~3 ]1 a* D[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version
2 ^& u: J) H/ P! y, q8 l. f% r2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
* L/ ~4 u3 Z. \, @FAILED - RETRYING: Wait for slave MariaDB (10 retries left).
$ e8 r0 h1 L* ]3 Sok: [controller3]
+ o2 h6 V( E* C' F' |ok: [controller1]
[WARNING]: flush_handlers task does not support when conditional

RUNNING HANDLER [mariadb : restart slave mariadb] *****************************************************************************************************************
1 I) I& Q9 j, y5 r% X, w! V. [skipping: [controller3]
- V8 D* W% h- }9 ^! Xskipping: [controller1]
skipping: [controller2]
RUNNING HANDLER [mariadb : restart master mariadb] ****************************************************************************************************************
% A! k, \5 e2 R3 w. ~1 `* pskipping: [controller3]
) y% K2 \5 r$ ^9 wskipping: [controller1]
skipping: [controller2]
  f& D/ t2 s9 x& i: s3 N# M9 l+ @( K  wTASK [mariadb : include] ******************************************************************************************************************************************
included: /usr/share/kolla-ansible/ansible/roles/mariadb/tasks/register.yml for controller1, controller2, controller3
9 }9 ~7 c* G" nTASK [mariadb : Creating haproxy mysql user] **********************************************************************************************************************
ok: [controller1]
TASK [mariadb : Cleaning up facts] ********************************************************************************************************************************
# ^% {' y1 W# M1 `. v; M9 [. tok: [controller1]
ok: [controller2]
, u# R! T" `: w: Fok: [controller3]
TASK [mariadb : include] ******************************************************************************************************************************************
$ h# g* O0 ~  S! Cincluded: /usr/share/kolla-ansible/ansible/roles/mariadb/tasks/check.yml for controller1, controller2, controller3
) h* i  D7 h$ X+ Z& ?+ O  n+ STASK [mariadb : Waiting for MariaDB service to be ready through VIP] **********************************************************************************************
[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version
% [$ c5 Y( j4 d: x, U2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
9 J' C. u- {5 y2 G$ h/ Vok: [controller1]
[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version
2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
% _# M8 D9 B+ @, U; g0 j% }) V1 J( S[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version
2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
3 f3 W2 K/ H" K; ~- p6 U- A( \0 \0 r/ Mok: [controller2]
ok: [controller3]
PLAY RECAP ********************************************************************************************************************************************************
& n1 r# c: T( d% Hcontroller1                : ok=37   changed=6    unreachable=0    failed=0    skipped=15   rescued=0    ignored=0   
controller2                : ok=37   changed=8    unreachable=0    failed=0    skipped=10   rescued=0    ignored=0   
controller3                : ok=31   changed=6    unreachable=0    failed=0    skipped=16   rescued=0    ignored=0   

3 q7 Q; O3 H( m* U  ?恢复正常。

* z, @  O' X# P# V; j; O$ p
有时候通过一些特殊手段可以帮助我们解决一些特别的问题，与其长时间尝试老方法，不如来点新的方式试试或许不一样的效果。

1 x" j. l9 X+ \1 L, t0 D
我们首先是因为数据库恢复出问题，无法恢复数据，， 检查数据的库的值，

cat /var/lib/docker/volumes/mariadb/_data/grastate.dat

中seqno值，若该值在所有节点中存在唯一得最大值
然后我们保留这个节点上数据，
删除其他节点上的数据文件
0 Q/ |- n, u4 {3 R" D: U' w9 O) ?然后通过scp或者rsync同步保留节点的数据文件  （根据个人喜好同步文件，本人建议使用rsync）
3 N* {7 X6 P4 @$ h, y! s$ `1 {* K: R然后再执行kolla-ansible数据库mariadb_recovery恢复命令：
* q: @2 D  W2 K5 J8 v) ~2 u+ L


  N$ e5 \9 i0 T0 q
差不多就可以完成数据库集群恢复了。