Add unit tests for invalid transfomations in Layer

This CL extracts common set up logic for layer related tests to another
file so that it can be shared among tests.

Bug: 210837722
Test: atest libsurfaceflinger_unittest
Change-Id: I9fb08ad164d57369f503a35eca907393b1969547