Cannot immediately overwrite the peer address when we parse