Remove non-determinism in socket unit test.

It is possible that there is some delay when a server socket is closed.
Due to this it is possible on slower machines to fail the
noConnectWhenNotCallingStart test (esp. on mac build bots that are under
heavy load).

We now remove this part of the test as the eventual part is hard to
measure.

Test: ran unit tests
Bug: 197374965
Change-Id: Icb5092fe9aceee174f507152d9a6c861f86c24cf