Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit c3ed75cf authored by TreeHugger Robot's avatar TreeHugger Robot Committed by Automerger Merge Worker
Browse files

Merge changes from topic "touch-rotation-precision" into udc-dev am: b2fd17a7

parents d020b562 b2fd17a7
Loading
Loading
Loading
Loading
+113 −0
Original line number Diff line number Diff line
# Input Coordinate Processing in InputFlinger

This document aims to illustrate why we need to take care when converting
between the discrete and continuous coordinate spaces, especially when
performing rotations.

The Linux evdev protocol works over **discrete integral** values. The same is
true for displays, which output discrete pixels. WindowManager also tracks
window bounds in pixels in the rotated logical display.

However, our `MotionEvent` APIs
report **floating point** axis values in a **continuous space**. This disparity
is important to note when working in InputFlinger, which has to make sure the
discrete raw coordinates are  converted to the continuous space correctly in all
scenarios.

## Disparity between continuous and discrete coordinates during rotation

Let's consider an example of device that has a 3 x 4 screen.

### Natural orientation:  No rotation

If the user interacts with the highlighted pixel, the touchscreen would report
the discreet coordinates (0, 2).

```
     ┌─────┬─────┬─────┐
     │ 0,0 │ 1,0 │ 2,0 │
     ├─────┼─────┼─────┤
     │ 0,1 │ 1,1 │ 2,1 │
     ├─────┼─────┼─────┤
     │█0,2█│ 1,2 │ 2,2 │
     ├─────┼─────┼─────┤
     │ 0,3 │ 1,3 │ 2,3 │
     └─────┴─────┴─────┘
```

When converted to the continuous space, the point (0, 2) corresponds to the
location shown below.

```
     0     1     2     3
  0  ┌─────┬─────┬─────┐
     │     │     │     │
  1  ├─────┼─────┼─────┤
     │     │     │     │
  2  █─────┼─────┼─────┤
     │     │     │     │
  3  ├─────┼─────┼─────┤
     │     │     │     │
  4  └─────┴─────┴─────┘
```

### Rotated orientation: 90-degree counter-clockwise rotation

When the device is rotated and the same place on the touchscreen is touched, the
input device will still report the same coordinates of (0, 2).

In the rotated display, that now corresponds to the pixel (2, 2).

```
     ┌─────┬─────┬─────┬─────┐
     │ 0,0 │ 1,0 │ 2,0 │ 3,0 │
     ├─────┼─────┼─────┼─────┤
     │ 0,1 │ 1,1 │ 2,1 │ 3,1 │
     ├─────┼─────┼─────┼─────┤
     │ 0,2 │ 1,2 │█2,2█│ 3,2 │
     └─────┴─────┴─────┴─────┘
```

*It is important to note that rotating the device 90 degrees is NOT equivalent
to rotating the continuous coordinate space by 90 degrees.*

The point (2, 2) now corresponds to a different location in the continuous space
than before, even though the user was interacting at the same place on the
touchscreen.

```
     0     1     2     3     4
  0  ┌─────┬─────┬─────┬─────┐
     │     │     │     │     │
  1  ├─────┼─────┼─────┼─────┤
     │     │     │     │     │
  2  ├─────┼─────█─────┼─────┤
     │     │     │     │     │
  3  └─────┴─────┴─────┴─────┘
```

If we were to simply (incorrectly) rotate the continuous space from before by
90 degrees, the touched point would correspond to the location (2, 3), shown
below. This new point is outside the bounds of the display, since it does not
correspond to any pixel at that location.

It should be impossible for a touchscreen to generate points outside the bounds
of the display, because we assume that the area of the touchscreen maps directly
to the area of the display. Therefore, that point is an invalid coordinate that
cannot be generated by an input device.

```
     0     1     2     3     4
  0  ┌─────┬─────┬─────┬─────┐
     │     │     │     │     ╏
  1  ├─────┼─────┼─────┼─────┤
     │     │     │     │     ╏
  2  ├─────┼─────┼─────┼─────┤
     │     │     │     │     ╏
  3  └-----┴-----█-----┴-----┘
```

The same logic applies to windows as well. When performing hit tests to
determine if a point in the continuous space falls inside a window's bounds,
hit test must be performed in the correct orientation, since points on the right
and bottom edges of the window do not fall within the window bounds.
+54 −29
Original line number Diff line number Diff line
@@ -842,38 +842,60 @@ void TouchInputMapper::initializeOrientedRanges() {
}

void TouchInputMapper::computeInputTransforms() {
    const ui::Size rawSize{mRawPointerAxes.getRawWidth(), mRawPointerAxes.getRawHeight()};
    constexpr auto isRotated = [](const ui::Transform::RotationFlags& rotation) {
        return rotation == ui::Transform::ROT_90 || rotation == ui::Transform::ROT_270;
    };

    ui::Size rotatedRawSize = rawSize;
    if (mInputDeviceOrientation == ui::ROTATION_270 || mInputDeviceOrientation == ui::ROTATION_90) {
        std::swap(rotatedRawSize.width, rotatedRawSize.height);
    }
    const auto rotationFlags = ui::Transform::toRotationFlags(-mInputDeviceOrientation);
    mRawRotation = ui::Transform{rotationFlags};
    // See notes about input coordinates in the inputflinger docs:
    // //frameworks/native/services/inputflinger/docs/input_coordinates.md

    // Step 1: Undo the raw offset so that the raw coordinate space now starts at (0, 0).
    ui::Transform undoRawOffset;
    undoRawOffset.set(-mRawPointerAxes.x.minValue, -mRawPointerAxes.y.minValue);

    // Step 2: Rotate the raw coordinates to the expected orientation.
    ui::Transform rotate;
    // When rotating raw coordinates, the raw size will be used as an offset.
    // Account for the extra unit added to the raw range when the raw size was calculated.
    rotate.set(rotationFlags, rotatedRawSize.width - 1, rotatedRawSize.height - 1);

    // Step 3: Scale the raw coordinates to the display space.
    ui::Transform scaleToDisplay;
    const float xScale = static_cast<float>(mDisplayBounds.width) / rotatedRawSize.width;
    const float yScale = static_cast<float>(mDisplayBounds.height) / rotatedRawSize.height;
    scaleToDisplay.set(xScale, 0, 0, yScale);

    mRawToDisplay = (scaleToDisplay * (rotate * undoRawOffset));

    // Calculate the transform that takes raw coordinates to the rotated display space.
    ui::Transform displayToRotatedDisplay;
    displayToRotatedDisplay.set(ui::Transform::toRotationFlags(-mViewport.orientation),
                                mViewport.deviceWidth, mViewport.deviceHeight);
    mRawToRotatedDisplay = displayToRotatedDisplay * mRawToDisplay;
    ui::Transform undoOffsetInRaw;
    undoOffsetInRaw.set(-mRawPointerAxes.x.minValue, -mRawPointerAxes.y.minValue);

    // Step 2: Rotate the raw coordinates to account for input device orientation. The coordinates
    // will now be in the same orientation as the display in ROTATION_0.
    // Note: Negating an ui::Rotation value will give its inverse rotation.
    const auto inputDeviceOrientation = ui::Transform::toRotationFlags(-mParameters.orientation);
    const ui::Size orientedRawSize = isRotated(inputDeviceOrientation)
            ? ui::Size{mRawPointerAxes.getRawHeight(), mRawPointerAxes.getRawWidth()}
            : ui::Size{mRawPointerAxes.getRawWidth(), mRawPointerAxes.getRawHeight()};
    // When rotating raw values, account for the extra unit added when calculating the raw range.
    const auto orientInRaw = ui::Transform(inputDeviceOrientation, orientedRawSize.width - 1,
                                           orientedRawSize.height - 1);

    // Step 3: Rotate the raw coordinates to account for the display rotation. The coordinates will
    // now be in the same orientation as the rotated display. There is no need to rotate the
    // coordinates to the display rotation if the device is not orientation-aware.
    const auto viewportRotation = ui::Transform::toRotationFlags(-mViewport.orientation);
    const auto rotatedRawSize = mParameters.orientationAware && isRotated(viewportRotation)
            ? ui::Size{orientedRawSize.height, orientedRawSize.width}
            : orientedRawSize;
    // When rotating raw values, account for the extra unit added when calculating the raw range.
    const auto rotateInRaw = mParameters.orientationAware
            ? ui::Transform(viewportRotation, rotatedRawSize.width - 1, rotatedRawSize.height - 1)
            : ui::Transform();

    // Step 4: Scale the raw coordinates to the display space.
    // - Here, we assume that the raw surface of the touch device maps perfectly to the surface
    //   of the display panel. This is usually true for touchscreens.
    // - From this point onward, we are no longer in the discrete space of the raw coordinates but
    //   are in the continuous space of the logical display.
    ui::Transform scaleRawToDisplay;
    const float xScale = static_cast<float>(mViewport.deviceWidth) / rotatedRawSize.width;
    const float yScale = static_cast<float>(mViewport.deviceHeight) / rotatedRawSize.height;
    scaleRawToDisplay.set(xScale, 0, 0, yScale);

    // Step 5: Undo the display rotation to bring us back to the un-rotated display coordinate space
    // that InputReader uses.
    const auto undoRotateInDisplay =
            ui::Transform(viewportRotation, mViewport.deviceWidth, mViewport.deviceHeight)
                    .inverse();

    // Now put it all together!
    mRawToRotatedDisplay = (scaleRawToDisplay * (rotateInRaw * (orientInRaw * undoOffsetInRaw)));
    mRawToDisplay = (undoRotateInDisplay * mRawToRotatedDisplay);
    mRawRotation = ui::Transform{mRawToDisplay.getOrientation()};
}

void TouchInputMapper::configureInputDevice(nsecs_t when, bool* outResetNeeded) {
@@ -949,6 +971,9 @@ void TouchInputMapper::configureInputDevice(nsecs_t when, bool* outResetNeeded)
            mPhysicalFrameInRotatedDisplay = {mViewport.physicalLeft, mViewport.physicalTop,
                                              mViewport.physicalRight, mViewport.physicalBottom};

            // TODO(b/257118693): Remove the dependence on the old orientation/rotation logic that
            //     uses mInputDeviceOrientation. The new logic uses the transforms calculated in
            //     computeInputTransforms().
            // InputReader works in the un-rotated display coordinate space, so we don't need to do
            // anything if the device is already orientation-aware. If the device is not
            // orientation-aware, then we need to apply the inverse rotation of the display so that
+201 −8
Original line number Diff line number Diff line
@@ -6754,28 +6754,30 @@ public:
    // The values inside DisplayViewport are expected to be pre-rotated. This updates the current
    // DisplayViewport to pre-rotate the values. The viewport's physical display will be set to the
    // rotated equivalent of the given un-rotated physical display bounds.
    void configurePhysicalDisplay(ui::Rotation orientation, Rect naturalPhysicalDisplay) {
    void configurePhysicalDisplay(ui::Rotation orientation, Rect naturalPhysicalDisplay,
                                  int32_t naturalDisplayWidth = DISPLAY_WIDTH,
                                  int32_t naturalDisplayHeight = DISPLAY_HEIGHT) {
        uint32_t inverseRotationFlags;
        auto width = DISPLAY_WIDTH;
        auto height = DISPLAY_HEIGHT;
        auto rotatedWidth = naturalDisplayWidth;
        auto rotatedHeight = naturalDisplayHeight;
        switch (orientation) {
            case ui::ROTATION_90:
                inverseRotationFlags = ui::Transform::ROT_270;
                std::swap(width, height);
                std::swap(rotatedWidth, rotatedHeight);
                break;
            case ui::ROTATION_180:
                inverseRotationFlags = ui::Transform::ROT_180;
                break;
            case ui::ROTATION_270:
                inverseRotationFlags = ui::Transform::ROT_90;
                std::swap(width, height);
                std::swap(rotatedWidth, rotatedHeight);
                break;
            case ui::ROTATION_0:
                inverseRotationFlags = ui::Transform::ROT_0;
                break;
        }
        const ui::Transform rotation(inverseRotationFlags, width, height);
        const ui::Transform rotation(inverseRotationFlags, rotatedWidth, rotatedHeight);
        const Rect rotatedPhysicalDisplay = rotation.transform(naturalPhysicalDisplay);
        std::optional<DisplayViewport> internalViewport =
@@ -6794,8 +6796,8 @@ public:
        v.physicalRight = rotatedPhysicalDisplay.right;
        v.physicalBottom = rotatedPhysicalDisplay.bottom;
        v.deviceWidth = width;
        v.deviceHeight = height;
        v.deviceWidth = rotatedWidth;
        v.deviceHeight = rotatedHeight;
        v.isActive = true;
        v.uniqueId = UNIQUE_ID;
@@ -6909,6 +6911,197 @@ TEST_F(TouchDisplayProjectionTest, EmitsTouchDownAfterEnteringPhysicalDisplay) {
    }
}
// --- TouchscreenPrecisionTests ---
// This test suite is used to ensure that touchscreen devices are scaled and configured correctly
// in various orientations and with different display rotations. We configure the touchscreen to
// have a higher resolution than that of the display by an integer scale factor in each axis so that
// we can enforce that coordinates match precisely as expected.
class TouchscreenPrecisionTestsFixture : public TouchDisplayProjectionTest,
                                         public ::testing::WithParamInterface<ui::Rotation> {
public:
    void SetUp() override {
        SingleTouchInputMapperTest::SetUp();
        // Prepare the raw axes to have twice the resolution of the display in the X axis and
        // four times the resolution of the display in the Y axis.
        prepareButtons();
        mFakeEventHub->addAbsoluteAxis(EVENTHUB_ID, ABS_X, PRECISION_RAW_X_MIN, PRECISION_RAW_X_MAX,
                                       0, 0);
        mFakeEventHub->addAbsoluteAxis(EVENTHUB_ID, ABS_Y, PRECISION_RAW_Y_MIN, PRECISION_RAW_Y_MAX,
                                       0, 0);
    }
    static const int32_t PRECISION_RAW_X_MIN = TouchInputMapperTest::RAW_X_MIN;
    static const int32_t PRECISION_RAW_X_MAX = PRECISION_RAW_X_MIN + DISPLAY_WIDTH * 2 - 1;
    static const int32_t PRECISION_RAW_Y_MIN = TouchInputMapperTest::RAW_Y_MIN;
    static const int32_t PRECISION_RAW_Y_MAX = PRECISION_RAW_Y_MIN + DISPLAY_HEIGHT * 4 - 1;
    static const std::array<Point, 4> kRawCorners;
};
const std::array<Point, 4> TouchscreenPrecisionTestsFixture::kRawCorners = {{
        {PRECISION_RAW_X_MIN, PRECISION_RAW_Y_MIN}, // left-top
        {PRECISION_RAW_X_MAX, PRECISION_RAW_Y_MIN}, // right-top
        {PRECISION_RAW_X_MAX, PRECISION_RAW_Y_MAX}, // right-bottom
        {PRECISION_RAW_X_MIN, PRECISION_RAW_Y_MAX}, // left-bottom
}};
// Tests for how the touchscreen is oriented relative to the natural orientation of the display.
// For example, if a touchscreen is configured with an orientation of 90 degrees, it is a portrait
// touchscreen panel that is used on a device whose natural display orientation is in landscape.
TEST_P(TouchscreenPrecisionTestsFixture, OrientationPrecision) {
    enum class Orientation {
        ORIENTATION_0 = ui::toRotationInt(ui::ROTATION_0),
        ORIENTATION_90 = ui::toRotationInt(ui::ROTATION_90),
        ORIENTATION_180 = ui::toRotationInt(ui::ROTATION_180),
        ORIENTATION_270 = ui::toRotationInt(ui::ROTATION_270),
        ftl_last = ORIENTATION_270,
    };
    using Orientation::ORIENTATION_0, Orientation::ORIENTATION_90, Orientation::ORIENTATION_180,
            Orientation::ORIENTATION_270;
    static const std::map<Orientation, std::array<vec2, 4> /*mappedCorners*/> kMappedCorners = {
            {ORIENTATION_0, {{{0, 0}, {479.5, 0}, {479.5, 799.75}, {0, 799.75}}}},
            {ORIENTATION_90, {{{0, 479.5}, {0, 0}, {799.75, 0}, {799.75, 479.5}}}},
            {ORIENTATION_180, {{{479.5, 799.75}, {0, 799.75}, {0, 0}, {479.5, 0}}}},
            {ORIENTATION_270, {{{799.75, 0}, {799.75, 479.5}, {0, 479.5}, {0, 0}}}},
    };
    const auto touchscreenOrientation = static_cast<Orientation>(ui::toRotationInt(GetParam()));
    // Configure the touchscreen as being installed in the one of the four different orientations
    // relative to the display.
    addConfigurationProperty("touch.deviceType", "touchScreen");
    addConfigurationProperty("touch.orientation", ftl::enum_string(touchscreenOrientation).c_str());
    prepareDisplay(ui::ROTATION_0);
    SingleTouchInputMapper& mapper = addMapperAndConfigure<SingleTouchInputMapper>();
    // If the touchscreen is installed in a rotated orientation relative to the display (i.e. in
    // orientations of either 90 or 270) this means the display's natural resolution will be
    // flipped.
    const bool displayRotated =
            touchscreenOrientation == ORIENTATION_90 || touchscreenOrientation == ORIENTATION_270;
    const int32_t width = displayRotated ? DISPLAY_HEIGHT : DISPLAY_WIDTH;
    const int32_t height = displayRotated ? DISPLAY_WIDTH : DISPLAY_HEIGHT;
    const Rect physicalFrame{0, 0, width, height};
    configurePhysicalDisplay(ui::ROTATION_0, physicalFrame, width, height);
    const auto& expectedPoints = kMappedCorners.at(touchscreenOrientation);
    const float expectedPrecisionX = displayRotated ? 4 : 2;
    const float expectedPrecisionY = displayRotated ? 2 : 4;
    // Test all four corners.
    for (int i = 0; i < 4; i++) {
        const auto& raw = kRawCorners[i];
        processDown(mapper, raw.x, raw.y);
        processSync(mapper);
        const auto& expected = expectedPoints[i];
        ASSERT_NO_FATAL_FAILURE(mFakeListener->assertNotifyMotionWasCalled(
                AllOf(WithMotionAction(AMOTION_EVENT_ACTION_DOWN),
                      WithCoords(expected.x, expected.y),
                      WithPrecision(expectedPrecisionX, expectedPrecisionY))))
                << "Failed to process raw point (" << raw.x << ", " << raw.y << ") "
                << "with touchscreen orientation "
                << ftl::enum_string(touchscreenOrientation).c_str() << ", expected point ("
                << expected.x << ", " << expected.y << ").";
        processUp(mapper);
        processSync(mapper);
        ASSERT_NO_FATAL_FAILURE(mFakeListener->assertNotifyMotionWasCalled(
                AllOf(WithMotionAction(AMOTION_EVENT_ACTION_UP),
                      WithCoords(expected.x, expected.y))));
    }
}
TEST_P(TouchscreenPrecisionTestsFixture, RotationPrecisionWhenOrientationAware) {
    static const std::map<ui::Rotation /*rotation*/, std::array<vec2, 4> /*mappedCorners*/>
            kMappedCorners = {
                    {ui::ROTATION_0, {{{0, 0}, {479.5, 0}, {479.5, 799.75}, {0, 799.75}}}},
                    {ui::ROTATION_90, {{{0.5, 0}, {480, 0}, {480, 799.75}, {0.5, 799.75}}}},
                    {ui::ROTATION_180, {{{0.5, 0.25}, {480, 0.25}, {480, 800}, {0.5, 800}}}},
                    {ui::ROTATION_270, {{{0, 0.25}, {479.5, 0.25}, {479.5, 800}, {0, 800}}}},
            };
    const ui::Rotation displayRotation = GetParam();
    addConfigurationProperty("touch.deviceType", "touchScreen");
    prepareDisplay(displayRotation);
    SingleTouchInputMapper& mapper = addMapperAndConfigure<SingleTouchInputMapper>();
    const auto& expectedPoints = kMappedCorners.at(displayRotation);
    // Test all four corners.
    for (int i = 0; i < 4; i++) {
        const auto& expected = expectedPoints[i];
        const auto& raw = kRawCorners[i];
        processDown(mapper, raw.x, raw.y);
        processSync(mapper);
        ASSERT_NO_FATAL_FAILURE(mFakeListener->assertNotifyMotionWasCalled(
                AllOf(WithMotionAction(AMOTION_EVENT_ACTION_DOWN),
                      WithCoords(expected.x, expected.y), WithPrecision(2, 4))))
                << "Failed to process raw point (" << raw.x << ", " << raw.y << ") "
                << "with display rotation " << ui::toCString(displayRotation)
                << ", expected point (" << expected.x << ", " << expected.y << ").";
        processUp(mapper);
        processSync(mapper);
        ASSERT_NO_FATAL_FAILURE(mFakeListener->assertNotifyMotionWasCalled(
                AllOf(WithMotionAction(AMOTION_EVENT_ACTION_UP),
                      WithCoords(expected.x, expected.y))));
    }
}
TEST_P(TouchscreenPrecisionTestsFixture, RotationPrecisionOrientationAwareInOri270) {
    static const std::map<ui::Rotation /*orientation*/, std::array<vec2, 4> /*mappedCorners*/>
            kMappedCorners = {
                    {ui::ROTATION_0, {{{799.75, 0}, {799.75, 479.5}, {0, 479.5}, {0, 0}}}},
                    {ui::ROTATION_90, {{{800, 0}, {800, 479.5}, {0.25, 479.5}, {0.25, 0}}}},
                    {ui::ROTATION_180, {{{800, 0.5}, {800, 480}, {0.25, 480}, {0.25, 0.5}}}},
                    {ui::ROTATION_270, {{{799.75, 0.5}, {799.75, 480}, {0, 480}, {0, 0.5}}}},
            };
    const ui::Rotation displayRotation = GetParam();
    addConfigurationProperty("touch.deviceType", "touchScreen");
    addConfigurationProperty("touch.orientation", "ORIENTATION_270");
    SingleTouchInputMapper& mapper = addMapperAndConfigure<SingleTouchInputMapper>();
    // Ori 270, so width and height swapped
    const Rect physicalFrame{0, 0, DISPLAY_HEIGHT, DISPLAY_WIDTH};
    prepareDisplay(displayRotation);
    configurePhysicalDisplay(displayRotation, physicalFrame, DISPLAY_HEIGHT, DISPLAY_WIDTH);
    const auto& expectedPoints = kMappedCorners.at(displayRotation);
    // Test all four corners.
    for (int i = 0; i < 4; i++) {
        const auto& expected = expectedPoints[i];
        const auto& raw = kRawCorners[i];
        processDown(mapper, raw.x, raw.y);
        processSync(mapper);
        ASSERT_NO_FATAL_FAILURE(mFakeListener->assertNotifyMotionWasCalled(
                AllOf(WithMotionAction(AMOTION_EVENT_ACTION_DOWN),
                      WithCoords(expected.x, expected.y), WithPrecision(4, 2))))
                << "Failed to process raw point (" << raw.x << ", " << raw.y << ") "
                << "with display rotation " << ui::toCString(displayRotation)
                << ", expected point (" << expected.x << ", " << expected.y << ").";
        processUp(mapper);
        processSync(mapper);
        ASSERT_NO_FATAL_FAILURE(mFakeListener->assertNotifyMotionWasCalled(
                AllOf(WithMotionAction(AMOTION_EVENT_ACTION_UP),
                      WithCoords(expected.x, expected.y))));
    }
}
// Run the precision tests for all rotations.
INSTANTIATE_TEST_SUITE_P(TouchscreenPrecisionTests, TouchscreenPrecisionTestsFixture,
                         ::testing::Values(ui::ROTATION_0, ui::ROTATION_90, ui::ROTATION_180,
                                           ui::ROTATION_270),
                         [](const testing::TestParamInfo<ui::Rotation>& testParamInfo) {
                             return ftl::enum_string(testParamInfo.param);
                         });
// --- ExternalStylusFusionTest ---
class ExternalStylusFusionTest : public SingleTouchInputMapperTest {
+6 −0
Original line number Diff line number Diff line
@@ -176,4 +176,10 @@ MATCHER_P(WithDownTime, downTime, "InputEvent with specified downTime") {
    return arg.downTime == downTime;
}

MATCHER_P2(WithPrecision, xPrecision, yPrecision, "MotionEvent with specified precision") {
    *result_listener << "expected x-precision " << xPrecision << " and y-precision " << yPrecision
                     << ", but got " << arg.xPrecision << " and " << arg.yPrecision;
    return arg.xPrecision == xPrecision && arg.yPrecision == yPrecision;
}

} // namespace android