libibverbs/comp_mgr: fix hang during destruction