Volume f6c2389d-44ca-4dce-9d71-361bbeb112cd did not finish being created even af

admin

Volume f6c2389d-44ca-4dce-9d71-361bbeb112cd did not finish being created even after we waited 865 seconds or 121 attempts. And its status is creating.

7 ]; j8 E0 y; ?0 x
4 B+ z+ A, l2 J9 {/ g
3 h- e. K8 d$ `. V, N' @$ J
/ b( t* y7 y  n6 Z3 T( n" i; J  l' P修改nova-compute.conf文件
5 s  E* N# z) e: f* r3 ]

On each compute node, edit the Nova configuration file:
' A9 r9 @5 p$ Z7 X5 ~! U: K* ^Raw
* I1 A  i4 t7 }* {+ A* F# vi /var/lib/config-data/nova_libvirt/etc/nova/nova.conf
; M- [* l: {7 R( r! RFind the following two parameters and increase them:
Raw
block_device_allocate_retries=120  <---- Default is 60
1 u5 H; M3 Q# f( A) ublock_device_allocate_retries_interval=6  <---- Default is 3
- ?1 c: s: i; dRestart the nova_compute container on each compute node:
Raw
# docker restart nova_compute
7 v; j  e" i* J! R, I: C: _Once all of the nova_compute containers have fully restarted, retry the instance creation.
NOTE: If editing the Nova configuration files is not feasible, then it is also possible to create the Cinder volume first, and then use it to boot the Nova instance. The following is an example:
3 W  y6 r; D, O% |/ f% dRaw
# openstack volume create --size 60 --image big_image boot_volume
# openstack server create --flavor 2 --volume boot_volume big_server
Applying the changes through the templates
0 n& i6 J9 w( Z- In order to apply this through the templates, add the following to an environment file such as network-environment.yaml:
Raw
  ComputeParameters:
    nova::block_device_allocate_retries: 120
' Q( `! X- x) J( }    nova::block_device_allocate_retries_interval: 6
  ControllerParameters:
    cinder::rpc_response_timeout: 1800

admin

block_device_allocate_retries = 180
block_device_allocate_retries_interval = 12

admin

[root@compute01 ~]# ceph -s
* v& V, }' E* p( q  cluster:
& e8 R8 M: O2 @. y+ l    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
' i# R8 l0 u: ~    health: HEALTH_OK
5 u1 V) O, R8 |- \4 p0 g
) v  C+ I3 R+ z+ Q4 k5 M" [& ^  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
+ w; `& _1 `; l8 x" r( ^    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

# B: X  ~( V' c! K, y; H+ K& B  data:
    pools:   7 pools, 1552 pgs
! ~) D+ X7 S; c; z0 X    objects: 8.26k objects, 42 GiB
    usage:   99 GiB used, 12 TiB / 13 TiB avail
6 E& E# q8 {7 z- A& u3 J    pgs:     1552 active+clean

( u1 z" f# C$ j  C$ T# L  io:
    client:   11 MiB/s rd, 1 op/s rd, 0 op/s wr
# d6 E% ?6 {8 u
[root@compute01 ~]# ceph -s
  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

6 z# F4 t1 ^8 z2 @, g6 x  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
) E! Z3 @' E+ L& v! Z" c
  data:
    pools:   7 pools, 1552 pgs
- f7 a# ^* w# D1 a+ |) {$ r    objects: 8.26k objects, 42 GiB
7 X# D! F4 U9 K, U7 C6 [& r/ q    usage:   99 GiB used, 12 TiB / 13 TiB avail
9 t$ B% U" p( [1 {4 H2 N( h: m+ b    pgs:     1552 active+clean

  io:
0 a$ U: k% r7 t! q( y    client:   11 MiB/s rd, 1 op/s rd, 0 op/s wr


' l& I; V5 e# D) e, C
主要原因是ceph的写入慢导致的问题，现在通过调整ceph cinder用户添加一个rx的权限，数据上来了。
* X2 X( f0 p" a" f4 K7 V1 {# ~allow rx pool=images

& c2 m' O  o/ e5 [# j6 L5 D[root@compute01 ~]# ceph -s
9 \8 a9 N7 U6 y# D  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
$ Y5 O* Y+ R4 f: T! F" O, v2 e, G    health: HEALTH_OK

: X3 F- v) _- G3 x% l0 K  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
! ], w7 U; Q5 ?& F' D( ^$ _7 E    mgr: compute03(active, since 24h), standbys: compute02, compute01
7 I6 G! [$ t, ?8 ?2 l( t% L    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

+ z# m' D9 f% Q0 p- x$ R  data:
0 h' A6 x" K0 h* h3 u! Y    pools:   7 pools, 1552 pgs
    objects: 8.30k objects, 42 GiB
# ~+ ~5 i& N+ O    usage:   99 GiB used, 12 TiB / 13 TiB avail
( u5 ~% A) t0 V6 G" K    pgs:     1552 active+clean

8 t( v6 S# W2 G" k4 J. l% ]0 T  io:
( a4 |3 I/ y7 t& C+ X5 h$ k    client:   13 MiB/s rd, 12 MiB/s wr, 7 op/s rd, 15 op/s wr
- o. T- n7 e/ t
) H& k) j5 U" B! B" |& `+ Z[root@compute01 ~]# ceph -s
( x. n5 I' ^8 U* J1 x  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
" z7 E6 |; Z/ P. `    health: HEALTH_OK
: ^6 s  @- T" t8 v2 e
/ ^1 `& K. i, l' u- D1 }2 O! {  services:
4 v: ?) c% T, C    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
' P0 `1 M" O7 M! x    mgr: compute03(active, since 24h), standbys: compute02, compute01
3 ^& g$ h7 W1 [1 ?    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
% C6 L4 J# A, s; J) t$ L; D
7 H" C- G8 X7 k* ~3 I5 A  data:
6 v8 ^4 o) P! A  A    pools:   7 pools, 1552 pgs
; P# F& C: I. h6 L9 z7 D. X* t    objects: 8.43k objects, 42 GiB
( C+ S+ q4 d! r    usage:   100 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean

  f  k3 V) e# s" `* ?  io:
5 L( Y: k+ I+ h5 g$ ]8 w  I    client:   13 MiB/s rd, 52 MiB/s wr, 24 op/s rd, 110 op/s wr
9 y/ s- v' f/ V5 P# ^, y* a2 R8 _3 S. R
[root@compute01 ~]# ceph -s
  cluster:
  g1 E3 ?. s: |6 z) h5 q/ W    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

% O% d& b; `5 [0 h- j% z  services:
( m3 ^- H$ w% V6 a2 B# N    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
6 c' L2 w9 I3 z+ a3 d. S6 W+ T    mgr: compute03(active, since 24h), standbys: compute02, compute01
6 T( P+ r5 P! {7 p4 D$ I6 H    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

6 a0 e* V. ^+ w' v7 {  data:
    pools:   7 pools, 1552 pgs
- j, P  W7 A5 Z/ _9 z# s8 A. x    objects: 8.43k objects, 42 GiB
; x  P% X) V& S    usage:   100 GiB used, 12 TiB / 13 TiB avail
7 E+ I+ m5 }8 D8 o    pgs:     1552 active+clean

$ d% q0 V  `7 O2 ?  io:
    client:   13 MiB/s rd, 52 MiB/s wr, 24 op/s rd, 110 op/s wr
% |7 Q0 b- ]7 C- L2 l+ ?$ O
[root@compute01 ~]# ceph -s
  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK
/ l; |- O+ H/ F% \9 X- x1 P
' `( x& n" G, K+ ]  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
$ S) l- H0 t  U. L6 @4 X4 _7 \    mgr: compute03(active, since 24h), standbys: compute02, compute01
" X/ r9 x$ g4 B9 P    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
2 g' ~1 ~3 k% ^6 ~. s' T3 `
: z; Z- g& v/ p5 O5 U  data:
; e) i" D& Q% c. z4 o/ w    pools:   7 pools, 1552 pgs
    objects: 8.43k objects, 42 GiB
    usage:   100 GiB used, 12 TiB / 13 TiB avail
' L8 M: `" |" l8 W3 C" |+ B    pgs:     1552 active+clean
: I* J5 N- j" a4 F5 U/ ]9 @
' Y  k. V+ b! S! b3 U  io:
$ R9 `. ~' {$ V8 G/ Z! F2 W3 U    client:   8.7 MiB/s rd, 52 MiB/s wr, 23 op/s rd, 110 op/s wr

[root@compute01 ~]# ceph -s
: g, y2 ]* {3 R. r5 t7 u  cluster:
. g! S' g# q& @) d. ?( e5 E2 S- \" t    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
% L) A5 m! e/ L: `    health: HEALTH_OK
; \' d" q  O+ n; z) F6 L
  S( n4 n" r9 Z2 k' U6 Q( s  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
- _; n: |) y: R5 _5 l3 A) d' N    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

. L! {! |7 V  J; d% J  data:
# \# P" s, P0 k) m) m: g& a% F    pools:   7 pools, 1552 pgs
6 e" Z5 Q& D- o% b" n, E1 S    objects: 8.50k objects, 43 GiB
    usage:   101 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean
1 e( p1 Z" D7 M
  io:
    client:   8.7 MiB/s rd, 73 MiB/s wr, 23 op/s rd, 225 op/s wr

) A& s" p5 W& o( }0 v  u# u+ y, x- u[root@compute01 ~]# ceph -s
5 [$ q, p* I$ N' Y1 n7 o+ O& P% V+ [  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
: T. w; k9 E3 f- J' {
  data:
# y, d" d: B' Z1 \6 n6 N5 d& ?    pools:   7 pools, 1552 pgs
4 u! p* Y; T0 Z- B3 ]8 g# r2 y    objects: 8.50k objects, 43 GiB
    usage:   101 GiB used, 12 TiB / 13 TiB avail
0 Q2 D6 ~& L8 w3 o    pgs:     1552 active+clean
. F! I' ^. c! H8 p# W
, d$ @( ?+ }& ]+ ~, ^2 |6 ^  io:
( s* B2 D; c7 a; H    client:   8.7 MiB/s rd, 73 MiB/s wr, 23 op/s rd, 225 op/s wr
+ I, @9 }: X1 O  n- a
[root@compute01 ~]# ceph -s
& }' z% L0 [+ T3 q( N! I  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

1 V0 b2 N2 [1 |7 \' s  services:
; p  e* F9 V: R" h    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
+ E7 J( O7 a  }9 t    mgr: compute03(active, since 24h), standbys: compute02, compute01
    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

& v7 l- e% S* Q' E' v  data:
! w: ]9 @- o9 h4 v$ r    pools:   7 pools, 1552 pgs
+ T) u1 \  T) n0 Y; Y* Y    objects: 8.56k objects, 43 GiB
  i( b+ z4 ^7 v2 ]4 W    usage:   101 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean

  io:
0 \* |( Q1 K4 E) ]+ k6 g    client:   13 MiB/s rd, 92 MiB/s wr, 23 op/s rd, 267 op/s wr

! L1 c$ s5 R" l5 J: _8 R* A[root@compute01 ~]# ceph -s
  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
- U- [% x" H9 b, U- ^- }0 \. o    health: HEALTH_OK

1 s* V# A7 ]- |8 J& d0 r( b  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
2 k1 r+ r4 [, r5 b  K# ~    mgr: compute03(active, since 24h), standbys: compute02, compute01
    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

  data:
: m" t( x7 l: ]  Q. u    pools:   7 pools, 1552 pgs
8 P& c, ^/ {  X3 Y    objects: 8.68k objects, 43 GiB
$ d4 S9 E! ^: V4 P- i- C$ {. M    usage:   102 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean

  io:
  w. o; {1 V6 D% |    client:   7.3 MiB/s rd, 39 MiB/s wr, 4 op/s rd, 32 op/s wr

1 L: R" ~1 t" Q3 |' @[root@compute01 ~]# ceph -s
  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
8 O6 G: i# v7 s' U7 N3 M6 m( M, A    mgr: compute03(active, since 24h), standbys: compute02, compute01
: f) s. S7 A- @; u( i9 \    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

  data:
! U2 c$ ]3 E" ?7 N' a) I1 T    pools:   7 pools, 1552 pgs
8 X0 ]2 l: P0 h7 d+ r) T1 x    objects: 8.68k objects, 43 GiB
* W' u- B7 j: S    usage:   102 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean
1 ]/ z4 ?( q+ @! ?- L
  io:
    client:   11 MiB/s rd, 15 MiB/s wr, 4 op/s rd, 7 op/s wr
7 C; s: D. U- d6 q