Add -lm to the default libs for Linux & Darwin

libm is a default library for device builds, so default it for host
builds as well.

Also removes duplicate additions of -ldl, -lpthread, -lm and -lrt.

Test: m host
Change-Id: I6a07e12053090eb6997b79d4091c28ac9a9022de