SF: Test coverage for OutputLayer::updateGeometryState

Adds test coverage for the new updateGeometryState and its child functions.

Also fixes an incorrect computation in the implementation as detected by
the tests.

Test: atest libsurfaceflinger_unittest libcompositionengine_test
Bug: 121291683
Change-Id: I8fa544fefea78effd3947ffebd0665799de0eaea