Remove return value of BTM_VendorSpecificCommand function

The return value of BTM_VendorSpecificCommand is currently just an
information wether the callback was passed to it or not, but many places
through stack use this value as error indication. Remove it.

Test: unit tests pass
Change-Id: Idf73fc49fb25e52c3aeb64f2305b4498ecacd9a4