libbinder: Inet SocketAddr contains raw port

In preparation for being able to call 'getaddrinfo' from inside this
object, which is required to setup dynamic threads. As a side-effect,
this also saves a heap allocation in the happy path.

Bug: 185167543
Test: N/A
Change-Id: I689d083876acf58d8d868d81d66508b91914b8a9