Fix logics for floating-point comparision in VTS test.

Set the acceptable error range based on both absolute tolerance and
relative tolerance.

Currently, absolute tolerance is set to 1e-5 for FP32 and 5 epsilon
(~5e-3) for FP16 relaxed computation. The relative tolerance is set to
5ULP of the corresponding precision. Add a TODO mark for potential
future adjustment on error limit based on testing.

Bug: 111768023

Test: none
Change-Id: Idedcec3e09fd7de9696811b93c81d0f180e896ef