occur QEMU error when volume attach

아래의 경우는 ceph2.1에 packstack 환경으로 일반적이지 않을수 있으나 참고할 필요는 있다.

OpenStack과 Ceph를 연동한 환경에서 아래와 같이 볼륨 attach시에 error 가발생하는 경우가 있다.

실제에러는 cinder-volume log에서 발생된다.

instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf] Attaching volume 8b4088ae-eb34-4076-bea0-7ad5c250d5de to /dev/vdb

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [req-a5e2fc3f-551e-4bb4-8217-8ed28d0706ce 06b90330d8e24091b0a06c121176c882 4f1ed33b6f22471596cf499fb87eae5a - - -] [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf] Failed to attach volume at mountpoint: /dev/vdb

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf] Traceback (most recent call last):

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1347, in attach_volume

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]     guest.attach_device(conf, persistent=True, live=live)

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 250, in attach_device

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]     self._domain.attachDeviceFlags(conf.to_xml(), flags=flags)

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]     result = proxy_call(self._autowrap, f, *args, **kwargs)

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]     rv = execute(f, *args, **kwargs)

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]     six.reraise(c, e, tb)

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]     rv = meth(*args, **kwargs)

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf] libvirtError: internal error: unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk1' could not be initialized

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]

이와 함께 libvirt 로그 상에는 다음과 같은 에러로그가 출력된다.

error reading header from volume-8b4088ae-eb34-4076-bea0-7ad5c250d5de: Function not implemented

error reading header from volume-3cb07ffc-3b6f-4ded-841c-42d9a30ff1e5: Function not implemented

error reading header from volume-8b4088ae-eb34-4076-bea0-7ad5c250d5de: Function not implemented

error reading header from volume-8b4088ae-eb34-4076-bea0-7ad5c250d5de: Function not implemented

error reading header from volume-8b4088ae-eb34-4076-bea0-7ad5c250d5de: Function not implemented

당시 ceph의 volume 정보를 확인해보면 format이 2 이고 feature가 layering 으로 되어 있음을 확인할 수 있다.

[root@com02 ~]# rbd info volumes/volume-c46fbf21-babf-47d7-95a1-f43ffc1322a0

rbd image 'volume-c46fbf21-babf-47d7-95a1-f43ffc1322a0':

        size 10240 MB in 2560 objects

        order 22 (4096 kB objects)

        block_name_prefix: rbd_data.10c282e9b53db

        format: 2

        features: layering

        flags:

해결방법은 다음 설정을 ceph.conf 상에 추가한다.

실제 아래 설정은 nova-compute 및 cinder가 동작하는 노드에 수정하면 된다.

  [client]

  rbd default features = 1

참고로 rbd feature의 number는 다음과 같은 뜻을 의미한다.

* 1 : layering

* 2 : striping 

* 4 : exclusive-lock

* 8 : object-map 

- 참조 : https://www.sebastien-han.fr/blog/2015/07/06/ceph-enable-the-object-map-feature/