error about security group in use

아래와 같은 상황으로 stack delete 가 계속 fail 되었다.

Heat로 VM을 생성하는 과정중에 error 가 발생되었고 에러가 발생된 VM을 수동으로 삭제한후 아래와 같은 stack delete 에러가 발생되었다.

2017-07-12 11:06:30.735 13244 INFO heat.engine.stack [-] Stack DELETE FAILED (AutoScaling-group-zr4gbmlkvzy3): Resource DELETE failed: Conflict: resources.apdlebkgvxmw.resources.sec_group: Security Group 6d2585fc-4e52-4371-aeb2-ada3ff645c64 in use.

Neutron server returns request_ids: ['req-a91a2956-5d52-4cb3-b736-bc1db4497116']

2017-07-12 11:06:31.700 13247 INFO heat.engine.resource [-] deleting SecurityGroup "sec_group" [6d2585fc-4e52-4371-aeb2-ada3ff645c64] Stack "AutoScaling-group-zr4gbmlkvzy3-apdlebkgvxmw-hu4oxpmmivm6" [fe8befb1-8f66-4e86-acea-35ce3ea2dbc0]

2017-07-12 11:06:31.824 13247 INFO heat.engine.resource [-] DELETE: SecurityGroup "sec_group" [6d2585fc-4e52-4371-aeb2-ada3ff645c64] Stack "AutoScaling-group-zr4gbmlkvzy3-apdlebkgvxmw-hu4oxpmmivm6" [fe8befb1-8f66-4e86-acea-35ce3ea2dbc0]

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource Traceback (most recent call last):

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 708, in _action_recorder

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource     yield

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1483, in delete

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource     yield self.action_handler_task(action, *action_args)

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 297, in wrapper

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource     step = next(subtask)

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 750, in action_handler_task

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource     handler_data = handler(*args)

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/neutron/security_group.py", line 241, in handle_delete

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource     self.client().delete_security_group(self.resource_id)

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 97, in with_params

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource     ret = self.function(instance, *args, **kwargs)

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 876, in delete_security_group

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource     return self.delete(self.security_group_path % (security_group))

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in delete

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource     headers=headers, params=params)

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 335, in retry_request

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource     headers=headers, params=params)

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 298, in do_request

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 273, in _handle_fault_response

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 84, in exception_handler_v20

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource     request_ids=request_ids)

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource Conflict: Security Group 6d2585fc-4e52-4371-aeb2-ada3ff645c64 in use.

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource Neutron server returns request_ids: ['req-bdea436e-3dab-424b-a736-94c43cf6677e']

2017-07-12 11:06:31.824 13247 ERROR heat.engine.resource

2017-07-12 11:06:31.943 13247 INFO heat.engine.stack [-] Stack DELETE FAILED (AutoScaling-group-zr4gbmlkvzy3-apdlebkgvxmw-hu4oxpmmivm6): Resource DELETE failed: Conflict: resources.sec_group: Security Group 6d2585fc-4e52-4371-aeb2-ada3ff645c64 in use.

실제 security group 삭제에 실패가 나면서 발생되는 문제였다.

하지만 command를 이용해 직접 삭제를 해봐도 아래와 같은 에러가 발생되고 삭제가 되지 않았다.

[stack@director9 ~]$ nova-secgroup-delete 6d2585fc-4e52-4371-aeb2-ada3ff645c64

Security Group 6d2585fc-4e52-4371-aeb2-ada3ff645c64 in use.

[stack@director9 ~]$ neutron security-group-delete 6d2585fc-4e52-4371-aeb2-ada3ff645c64

Security Group 6d2585fc-4e52-4371-aeb2-ada3ff645c64 in use.

Neutron server returns request_ids: ['req-b7baf6fe-ac49-4cb2-a2cf-197969538dbd']

실제 확인해보니 network에 port 가 삭제가되지 않아 발생된 문제였다.

아래와 같이 controller에 DB를 직접 연결하여 select query를 해보니 해당하는 secgroup에 매칭되는 port가 남아있었다.

MariaDB [ovs_neutron]> select * from securitygroupportbindings where security_group_id='6d2585fc-4e52-4371-aeb2-ada3ff645c64';

+--------------------------------------+--------------------------------------+

| port_id                              | security_group_id                    |

+--------------------------------------+--------------------------------------+

| aa4f464e-cc35-4d7d-95a8-62d11e455e7c | 6d2585fc-4e52-4371-aeb2-ada3ff645c64 |

| d6702191-3f4f-4392-a87b-fa3bcf9f0c2b | 6d2585fc-4e52-4371-aeb2-ada3ff645c64 |

+--------------------------------------+--------------------------------------+

해당 하는 port를 horizon에 network으로 이동하여 port를 delete 한후에 다시 stack delete를 수행한 결과 정상적으로 stack이 삭제되었다.

참고사이트

- https://ask.openstack.org/en/question/97045/delete-hanging-security-group/