librdmacm: fix event reporting when destroying listen