Volume f6c2389d-44ca-4dce-9d71-361bbeb112cd did not finish being created even af

admin

Volume f6c2389d-44ca-4dce-9d71-361bbeb112cd did not finish being created even after we waited 865 seconds or 121 attempts. And its status is creating.


; y: G5 B. C. y% {4 `

修改nova-compute.conf文件
/ P& {) _6 ]# {2 e: r% [9 Y

0 k6 r& C+ g, i  E& F7 E  f9 COn each compute node, edit the Nova configuration file:
$ o+ B' x# [! p/ q) z* z1 Z, WRaw
- H/ ~9 r2 ^; |  w# C# vi /var/lib/config-data/nova_libvirt/etc/nova/nova.conf
Find the following two parameters and increase them:
Raw
% l$ S0 `+ v1 c! M/ B! \# W3 oblock_device_allocate_retries=120  <---- Default is 60
block_device_allocate_retries_interval=6  <---- Default is 3
# @& P- A4 r- H: ~  S  SRestart the nova_compute container on each compute node:
" t. r/ _( W( r% C2 [' C1 HRaw
% q7 d: O3 D- Y; {0 Z" Z3 A# docker restart nova_compute
+ M0 Z* C1 R7 fOnce all of the nova_compute containers have fully restarted, retry the instance creation.
NOTE: If editing the Nova configuration files is not feasible, then it is also possible to create the Cinder volume first, and then use it to boot the Nova instance. The following is an example:
Raw
# openstack volume create --size 60 --image big_image boot_volume
4 q7 H7 @/ O2 W; d% Z# openstack server create --flavor 2 --volume boot_volume big_server
Applying the changes through the templates
4 k. b8 j+ k' ]6 g9 g2 `5 H3 G4 w- In order to apply this through the templates, add the following to an environment file such as network-environment.yaml:
. S7 `# s# c' \5 l6 bRaw
  ComputeParameters:
    nova::block_device_allocate_retries: 120
    nova::block_device_allocate_retries_interval: 6
: v/ ]4 ^6 @" t0 a; E  ControllerParameters:
    cinder::rpc_response_timeout: 1800

admin

block_device_allocate_retries = 180
# B+ O% R- J' v/ U9 Z& Qblock_device_allocate_retries_interval = 12

admin

[root@compute01 ~]# ceph -s
/ X: H! x. ^$ W2 K  cluster:
$ R' G8 V& z' ?( {/ O7 m    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK
2 G8 {* q4 o/ ^- g( G. C* f  o
# n' W, q/ p9 ?" f; p2 D4 {  services:
2 O( {% ?9 N" a6 M, u2 g5 M0 G    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
2 H8 o! J' A/ A% c$ K. m# ?    mgr: compute03(active, since 24h), standbys: compute02, compute01
  B4 D  J  g/ u# ^& N6 ?    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
2 ?8 d8 c1 _: o4 w/ u; G. s
  data:
    pools:   7 pools, 1552 pgs
  ^2 E5 X7 R; M2 h+ v+ i2 w) j    objects: 8.26k objects, 42 GiB
' ]. w% q) }; D0 x& S; |    usage:   99 GiB used, 12 TiB / 13 TiB avail
2 Y" q9 k8 d7 ?    pgs:     1552 active+clean
1 u6 h; s/ W: V/ A5 h
  io:
2 r. s; E- K6 U$ @3 b6 T4 J( N' ?7 P: U    client:   11 MiB/s rd, 1 op/s rd, 0 op/s wr

. O/ `8 g" L& N4 h; n/ p4 T[root@compute01 ~]# ceph -s
3 t+ h; ^  o6 m5 p  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
4 Z, W  R6 l, g! T0 P+ W+ o! \) l    health: HEALTH_OK
. f' w0 ]$ n" F: {: P2 ?% `/ ?- E
. l/ E, u! e8 b# f! S  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

  data:
    pools:   7 pools, 1552 pgs
9 a' Q$ B" a8 K0 n    objects: 8.26k objects, 42 GiB
2 X6 ^0 j3 `& R& j7 N3 r7 P  m' }    usage:   99 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean
# q8 r( P8 C6 ]# ]7 e8 c9 r
  io:
    client:   11 MiB/s rd, 1 op/s rd, 0 op/s wr
! f# {* K- R8 _% J
# v) V4 h2 N) z8 h) ~

主要原因是ceph的写入慢导致的问题，现在通过调整ceph cinder用户添加一个rx的权限，数据上来了。
9 _' @) G; \4 S8 O& o9 sallow rx pool=images

4 R( w2 o/ h4 L; W4 q[root@compute01 ~]# ceph -s
  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

' A* T2 N# a) A+ T- t  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
& F) u$ K4 z  q, r- D4 |) ~& w$ b% n
  data:
( `: B5 Q9 b8 z7 K& Q7 i    pools:   7 pools, 1552 pgs
    objects: 8.30k objects, 42 GiB
    usage:   99 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean
" c0 J1 p- ^* V9 G
) I2 |  {* c; D6 k" S# F" ^0 i  io:
    client:   13 MiB/s rd, 12 MiB/s wr, 7 op/s rd, 15 op/s wr
; |1 e9 }! m! N7 G
[root@compute01 ~]# ceph -s
% U, B, O' P* U. B1 p- J  cluster:
: O) [9 c2 w- t7 }6 ~    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
4 c/ w0 s: X; r  Z% s+ @( X+ A    health: HEALTH_OK
( L0 [. Y4 ~3 t
( k$ @; M1 d5 V  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
4 @/ A; b, ^) a8 [- f  R8 A    mgr: compute03(active, since 24h), standbys: compute02, compute01
    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
- n" C" W. [9 h; j( S# h
  data:
    pools:   7 pools, 1552 pgs
    objects: 8.43k objects, 42 GiB
  O4 J% |. G0 e' b    usage:   100 GiB used, 12 TiB / 13 TiB avail
# f4 L4 u' G) y) b* |    pgs:     1552 active+clean
; A6 B3 B1 S5 F
2 k  m+ H# j. e4 }7 `  io:
" _' d( w( A: A9 _5 Z5 o  k7 {8 e    client:   13 MiB/s rd, 52 MiB/s wr, 24 op/s rd, 110 op/s wr

% N- U, G% [& C7 D4 N8 y[root@compute01 ~]# ceph -s
8 n* r2 u0 Q* ~  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
' }/ |5 o( ~( F! \3 @$ B    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
' Q. c! k; n' Y' O+ ^) C    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
% V) y; M+ ~& X. t* n, H" N: T
  data:
    pools:   7 pools, 1552 pgs
    objects: 8.43k objects, 42 GiB
/ w! I' L1 j& ]( d* V1 R% a* H    usage:   100 GiB used, 12 TiB / 13 TiB avail
8 u- K: P( L1 W- e* ?    pgs:     1552 active+clean

! D$ }- T* c$ b2 E+ F1 H  io:
1 Q$ q& F) n+ h# p; |/ p    client:   13 MiB/s rd, 52 MiB/s wr, 24 op/s rd, 110 op/s wr
* Y- M1 V" u2 m% r) M$ J
) |, n2 M7 r3 X7 a# Y[root@compute01 ~]# ceph -s
2 Q/ W9 n; V- U$ i  cluster:
8 D. C1 p3 r, N' V% e  t4 f( d/ g    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK
" f4 V+ X( h" h/ |
  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
2 n" J' t) M! @1 W) u2 v9 n
& s" k3 f6 b% ~' O% N( w+ a7 p" {  data:
    pools:   7 pools, 1552 pgs
; X6 \2 L4 [0 o) o    objects: 8.43k objects, 42 GiB
    usage:   100 GiB used, 12 TiB / 13 TiB avail
2 b" P4 g+ Y& ?    pgs:     1552 active+clean
" C$ \( I4 L' w- n
, @2 Y/ y) ^0 S  ]# v( H  io:
    client:   8.7 MiB/s rd, 52 MiB/s wr, 23 op/s rd, 110 op/s wr

- {' K& Y' }# `' e3 {& u' i[root@compute01 ~]# ceph -s
  cluster:
& @8 Y. `6 g" o5 W) ~0 P( \    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

2 \" A+ @, h$ b# R2 W  services:
2 N! b: E! z3 @5 g  V+ J, Y    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
" L8 o8 H- Z2 |9 L% H4 Y9 L
  data:
    pools:   7 pools, 1552 pgs
7 C. R" s7 a  i: @2 Z6 t+ F    objects: 8.50k objects, 43 GiB
8 c$ U. _! z* d% N    usage:   101 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean
  K( @0 x$ S2 R+ }2 O2 R
; s, A! i1 s. P5 w: z8 U; I  io:
; \6 c- J7 D, x7 O) A: |2 o, x    client:   8.7 MiB/s rd, 73 MiB/s wr, 23 op/s rd, 225 op/s wr
  @2 C$ q* H2 |1 p, A
[root@compute01 ~]# ceph -s
  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK
& n; s4 {2 D6 Q$ |
  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
/ Y! S" n: C- s2 a: |. A
  data:
    pools:   7 pools, 1552 pgs
    objects: 8.50k objects, 43 GiB
, i: B/ b. S' f    usage:   101 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean
& Z8 s4 r+ h5 {" a; Q+ b7 P$ t
  io:
& K$ S, V0 {3 X6 P) X6 o    client:   8.7 MiB/s rd, 73 MiB/s wr, 23 op/s rd, 225 op/s wr

[root@compute01 ~]# ceph -s
2 B  V+ E7 g: q+ L2 x  cluster:
5 }" Y3 N7 J* n7 U' M; A* w5 w6 x* ~    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
( t2 v, M" G4 P) k    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
! n! y/ Y9 [/ k. |    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

  data:
    pools:   7 pools, 1552 pgs
2 h: A* b3 {/ T! t1 q8 J! D    objects: 8.56k objects, 43 GiB
( Y9 i* M, x: d& S3 R" b. O! W! e    usage:   101 GiB used, 12 TiB / 13 TiB avail
1 Z" d' u5 @5 X    pgs:     1552 active+clean
/ O- M  \1 n. `8 r+ c
  io:
    client:   13 MiB/s rd, 92 MiB/s wr, 23 op/s rd, 267 op/s wr
0 r4 i$ e) n4 Y# M+ B
7 w  z+ Q1 b8 e% s[root@compute01 ~]# ceph -s
2 W/ E0 p* o* x8 L7 [  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK
& b* v. J  F0 I
8 ~+ n/ r9 k1 M. G& q  services:
2 g7 Q' |. ?$ K, B& {* m, ^    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
6 T! A* `% y6 J1 X  `% t2 l- q& \7 x    mgr: compute03(active, since 24h), standbys: compute02, compute01
5 r5 E8 C) N' o    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
& \' y! h" n3 p! `! G; k
5 i1 E# n3 H# r1 Z* D5 P  data:
    pools:   7 pools, 1552 pgs
    objects: 8.68k objects, 43 GiB
    usage:   102 GiB used, 12 TiB / 13 TiB avail
  ?: X% u- K" Y. A5 @* U/ m& o    pgs:     1552 active+clean

: M# A% ]; j" N' N' b7 U  io:
    client:   7.3 MiB/s rd, 39 MiB/s wr, 4 op/s rd, 32 op/s wr
! R4 ]" c% x, }4 y
[root@compute01 ~]# ceph -s
7 Q4 b5 M* ^$ _" r4 y7 X3 G  cluster:
& r! }7 U/ Q5 t: ~# M1 \    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK
2 l/ j* Q/ [+ Z# |4 t; L  {. u
0 }$ {. R( k% _6 ~8 H  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
0 A8 ]. x( O; P( H    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
! q) u- _: ]% \+ s3 o, X% H
" ?% f# Z/ v% X! Q9 E7 t  data:
+ \2 j- D& p& j0 v" U" f/ |    pools:   7 pools, 1552 pgs
; @5 \6 D- |/ |. j, j    objects: 8.68k objects, 43 GiB
( C! O. B+ i: c+ `+ p2 ~& G    usage:   102 GiB used, 12 TiB / 13 TiB avail
2 I% h9 w% r2 i( ~    pgs:     1552 active+clean
0 {) g9 v4 }0 s4 e2 V* Y* J. U
8 y6 Y% y4 {$ A  Y& |( D, B  io:
    client:   11 MiB/s rd, 15 MiB/s wr, 4 op/s rd, 7 op/s wr
7 l! z/ x7 U( ]; H* i% ?7 W7 B& T