Merge changes from topic "touch-rotation-precision" into udc-dev (b2fd17a7) · Commits · e / os / android_frameworks_native

services/inputflinger/docs/input_coordinates.md

0 → 100644

+113 −0

Original line number	Diff line number	Diff line
		# Input Coordinate Processing in InputFlinger

		This document aims to illustrate why we need to take care when converting
		between the discrete and continuous coordinate spaces, especially when
		performing rotations.

		The Linux evdev protocol works over discrete integral values. The same is
		true for displays, which output discrete pixels. WindowManager also tracks
		window bounds in pixels in the rotated logical display.

		However, our `MotionEvent` APIs
		report floating point axis values in a continuous space. This disparity
		is important to note when working in InputFlinger, which has to make sure the
		discrete raw coordinates are converted to the continuous space correctly in all
		scenarios.

		## Disparity between continuous and discrete coordinates during rotation

		Let's consider an example of device that has a 3 x 4 screen.

		### Natural orientation: No rotation

		If the user interacts with the highlighted pixel, the touchscreen would report
		the discreet coordinates (0, 2).

		```
		┌─────┬─────┬─────┐
		│ 0,0 │ 1,0 │ 2,0 │
		├─────┼─────┼─────┤
		│ 0,1 │ 1,1 │ 2,1 │
		├─────┼─────┼─────┤
		│█0,2█│ 1,2 │ 2,2 │
		├─────┼─────┼─────┤
		│ 0,3 │ 1,3 │ 2,3 │
		└─────┴─────┴─────┘
		```

		When converted to the continuous space, the point (0, 2) corresponds to the
		location shown below.

		```
		0 1 2 3
		0 ┌─────┬─────┬─────┐
		│ │ │ │
		1 ├─────┼─────┼─────┤
		│ │ │ │
		2 █─────┼─────┼─────┤
		│ │ │ │
		3 ├─────┼─────┼─────┤
		│ │ │ │
		4 └─────┴─────┴─────┘
		```

		### Rotated orientation: 90-degree counter-clockwise rotation

		When the device is rotated and the same place on the touchscreen is touched, the
		input device will still report the same coordinates of (0, 2).

		In the rotated display, that now corresponds to the pixel (2, 2).

		```
		┌─────┬─────┬─────┬─────┐
		│ 0,0 │ 1,0 │ 2,0 │ 3,0 │
		├─────┼─────┼─────┼─────┤
		│ 0,1 │ 1,1 │ 2,1 │ 3,1 │
		├─────┼─────┼─────┼─────┤
		│ 0,2 │ 1,2 │█2,2█│ 3,2 │
		└─────┴─────┴─────┴─────┘
		```

		*It is important to note that rotating the device 90 degrees is NOT equivalent
		to rotating the continuous coordinate space by 90 degrees.*

		The point (2, 2) now corresponds to a different location in the continuous space
		than before, even though the user was interacting at the same place on the
		touchscreen.

		```
		0 1 2 3 4
		0 ┌─────┬─────┬─────┬─────┐
		│ │ │ │ │
		1 ├─────┼─────┼─────┼─────┤
		│ │ │ │ │
		2 ├─────┼─────█─────┼─────┤
		│ │ │ │ │
		3 └─────┴─────┴─────┴─────┘
		```

		If we were to simply (incorrectly) rotate the continuous space from before by
		90 degrees, the touched point would correspond to the location (2, 3), shown
		below. This new point is outside the bounds of the display, since it does not
		correspond to any pixel at that location.

		It should be impossible for a touchscreen to generate points outside the bounds
		of the display, because we assume that the area of the touchscreen maps directly
		to the area of the display. Therefore, that point is an invalid coordinate that
		cannot be generated by an input device.

		```
		0 1 2 3 4
		0 ┌─────┬─────┬─────┬─────┐
		│ │ │ │ ╏
		1 ├─────┼─────┼─────┼─────┤
		│ │ │ │ ╏
		2 ├─────┼─────┼─────┼─────┤
		│ │ │ │ ╏
		3 └-----┴-----█-----┴-----┘
		```

		The same logic applies to windows as well. When performing hit tests to
		determine if a point in the continuous space falls inside a window's bounds,
		hit test must be performed in the correct orientation, since points on the right
		and bottom edges of the window do not fall within the window bounds.

services/inputflinger/reader/mapper/TouchInputMapper.cpp

+54 −29

Original line number	Diff line number	Diff line
		@@ -842,38 +842,60 @@ void TouchInputMapper::initializeOrientedRanges() {
		}

		void TouchInputMapper::computeInputTransforms() {
		const ui::Size rawSize{mRawPointerAxes.getRawWidth(), mRawPointerAxes.getRawHeight()};
		constexpr auto isRotated = [](const ui::Transform::RotationFlags& rotation) {
		return rotation == ui::Transform::ROT_90 \|\| rotation == ui::Transform::ROT_270;
		};

		ui::Size rotatedRawSize = rawSize;
		if (mInputDeviceOrientation == ui::ROTATION_270 \|\| mInputDeviceOrientation == ui::ROTATION_90) {
		std::swap(rotatedRawSize.width, rotatedRawSize.height);
		}
		const auto rotationFlags = ui::Transform::toRotationFlags(-mInputDeviceOrientation);
		mRawRotation = ui::Transform{rotationFlags};
		// See notes about input coordinates in the inputflinger docs:
		// //frameworks/native/services/inputflinger/docs/input_coordinates.md

		// Step 1: Undo the raw offset so that the raw coordinate space now starts at (0, 0).
		ui::Transform undoRawOffset;
		undoRawOffset.set(-mRawPointerAxes.x.minValue, -mRawPointerAxes.y.minValue);

		// Step 2: Rotate the raw coordinates to the expected orientation.
		ui::Transform rotate;
		// When rotating raw coordinates, the raw size will be used as an offset.
		// Account for the extra unit added to the raw range when the raw size was calculated.
		rotate.set(rotationFlags, rotatedRawSize.width - 1, rotatedRawSize.height - 1);

		// Step 3: Scale the raw coordinates to the display space.
		ui::Transform scaleToDisplay;
		const float xScale = static_cast<float>(mDisplayBounds.width) / rotatedRawSize.width;
		const float yScale = static_cast<float>(mDisplayBounds.height) / rotatedRawSize.height;
		scaleToDisplay.set(xScale, 0, 0, yScale);

		mRawToDisplay = (scaleToDisplay * (rotate * undoRawOffset));

		// Calculate the transform that takes raw coordinates to the rotated display space.
		ui::Transform displayToRotatedDisplay;
		displayToRotatedDisplay.set(ui::Transform::toRotationFlags(-mViewport.orientation),
		mViewport.deviceWidth, mViewport.deviceHeight);
		mRawToRotatedDisplay = displayToRotatedDisplay * mRawToDisplay;
		ui::Transform undoOffsetInRaw;
		undoOffsetInRaw.set(-mRawPointerAxes.x.minValue, -mRawPointerAxes.y.minValue);

		// Step 2: Rotate the raw coordinates to account for input device orientation. The coordinates
		// will now be in the same orientation as the display in ROTATION_0.
		// Note: Negating an ui::Rotation value will give its inverse rotation.
		const auto inputDeviceOrientation = ui::Transform::toRotationFlags(-mParameters.orientation);
		const ui::Size orientedRawSize = isRotated(inputDeviceOrientation)
		? ui::Size{mRawPointerAxes.getRawHeight(), mRawPointerAxes.getRawWidth()}
		: ui::Size{mRawPointerAxes.getRawWidth(), mRawPointerAxes.getRawHeight()};
		// When rotating raw values, account for the extra unit added when calculating the raw range.
		const auto orientInRaw = ui::Transform(inputDeviceOrientation, orientedRawSize.width - 1,
		orientedRawSize.height - 1);

		// Step 3: Rotate the raw coordinates to account for the display rotation. The coordinates will
		// now be in the same orientation as the rotated display. There is no need to rotate the
		// coordinates to the display rotation if the device is not orientation-aware.
		const auto viewportRotation = ui::Transform::toRotationFlags(-mViewport.orientation);
		const auto rotatedRawSize = mParameters.orientationAware && isRotated(viewportRotation)
		? ui::Size{orientedRawSize.height, orientedRawSize.width}
		: orientedRawSize;
		// When rotating raw values, account for the extra unit added when calculating the raw range.
		const auto rotateInRaw = mParameters.orientationAware
		? ui::Transform(viewportRotation, rotatedRawSize.width - 1, rotatedRawSize.height - 1)
		: ui::Transform();

		// Step 4: Scale the raw coordinates to the display space.
		// - Here, we assume that the raw surface of the touch device maps perfectly to the surface
		// of the display panel. This is usually true for touchscreens.
		// - From this point onward, we are no longer in the discrete space of the raw coordinates but
		// are in the continuous space of the logical display.
		ui::Transform scaleRawToDisplay;
		const float xScale = static_cast<float>(mViewport.deviceWidth) / rotatedRawSize.width;
		const float yScale = static_cast<float>(mViewport.deviceHeight) / rotatedRawSize.height;
		scaleRawToDisplay.set(xScale, 0, 0, yScale);

		// Step 5: Undo the display rotation to bring us back to the un-rotated display coordinate space
		// that InputReader uses.
		const auto undoRotateInDisplay =
		ui::Transform(viewportRotation, mViewport.deviceWidth, mViewport.deviceHeight)
		.inverse();

		// Now put it all together!
		mRawToRotatedDisplay = (scaleRawToDisplay * (rotateInRaw * (orientInRaw * undoOffsetInRaw)));
		mRawToDisplay = (undoRotateInDisplay * mRawToRotatedDisplay);
		mRawRotation = ui::Transform{mRawToDisplay.getOrientation()};
		}

		void TouchInputMapper::configureInputDevice(nsecs_t when, bool* outResetNeeded) {
		@@ -949,6 +971,9 @@ void TouchInputMapper::configureInputDevice(nsecs_t when, bool* outResetNeeded)
		mPhysicalFrameInRotatedDisplay = {mViewport.physicalLeft, mViewport.physicalTop,
		mViewport.physicalRight, mViewport.physicalBottom};

		// TODO(b/257118693): Remove the dependence on the old orientation/rotation logic that
		// uses mInputDeviceOrientation. The new logic uses the transforms calculated in
		// computeInputTransforms().
		// InputReader works in the un-rotated display coordinate space, so we don't need to do
		// anything if the device is already orientation-aware. If the device is not
		// orientation-aware, then we need to apply the inverse rotation of the display so that

services/inputflinger/tests/InputReader_test.cpp

+201 −8

Original line number	Diff line number	Diff line
		@@ -6754,28 +6754,30 @@ public:
		// The values inside DisplayViewport are expected to be pre-rotated. This updates the current
		// DisplayViewport to pre-rotate the values. The viewport's physical display will be set to the
		// rotated equivalent of the given un-rotated physical display bounds.
		void configurePhysicalDisplay(ui::Rotation orientation, Rect naturalPhysicalDisplay) {
		void configurePhysicalDisplay(ui::Rotation orientation, Rect naturalPhysicalDisplay,
		int32_t naturalDisplayWidth = DISPLAY_WIDTH,
		int32_t naturalDisplayHeight = DISPLAY_HEIGHT) {
		uint32_t inverseRotationFlags;
		auto width = DISPLAY_WIDTH;
		auto height = DISPLAY_HEIGHT;
		auto rotatedWidth = naturalDisplayWidth;
		auto rotatedHeight = naturalDisplayHeight;
		switch (orientation) {
		case ui::ROTATION_90:
		inverseRotationFlags = ui::Transform::ROT_270;
		std::swap(width, height);
		std::swap(rotatedWidth, rotatedHeight);
		break;
		case ui::ROTATION_180:
		inverseRotationFlags = ui::Transform::ROT_180;
		break;
		case ui::ROTATION_270:
		inverseRotationFlags = ui::Transform::ROT_90;
		std::swap(width, height);
		std::swap(rotatedWidth, rotatedHeight);
		break;
		case ui::ROTATION_0:
		inverseRotationFlags = ui::Transform::ROT_0;
		break;
		}

		const ui::Transform rotation(inverseRotationFlags, width, height);
		const ui::Transform rotation(inverseRotationFlags, rotatedWidth, rotatedHeight);
		const Rect rotatedPhysicalDisplay = rotation.transform(naturalPhysicalDisplay);

		std::optional<DisplayViewport> internalViewport =
		@@ -6794,8 +6796,8 @@ public:
		v.physicalRight = rotatedPhysicalDisplay.right;
		v.physicalBottom = rotatedPhysicalDisplay.bottom;

		v.deviceWidth = width;
		v.deviceHeight = height;
		v.deviceWidth = rotatedWidth;
		v.deviceHeight = rotatedHeight;

		v.isActive = true;
		v.uniqueId = UNIQUE_ID;
		@@ -6909,6 +6911,197 @@ TEST_F(TouchDisplayProjectionTest, EmitsTouchDownAfterEnteringPhysicalDisplay) {
		}
		}

		// --- TouchscreenPrecisionTests ---

		// This test suite is used to ensure that touchscreen devices are scaled and configured correctly
		// in various orientations and with different display rotations. We configure the touchscreen to
		// have a higher resolution than that of the display by an integer scale factor in each axis so that
		// we can enforce that coordinates match precisely as expected.
		class TouchscreenPrecisionTestsFixture : public TouchDisplayProjectionTest,
		public ::testing::WithParamInterface<ui::Rotation> {
		public:
		void SetUp() override {
		SingleTouchInputMapperTest::SetUp();

		// Prepare the raw axes to have twice the resolution of the display in the X axis and
		// four times the resolution of the display in the Y axis.
		prepareButtons();
		mFakeEventHub->addAbsoluteAxis(EVENTHUB_ID, ABS_X, PRECISION_RAW_X_MIN, PRECISION_RAW_X_MAX,
		0, 0);
		mFakeEventHub->addAbsoluteAxis(EVENTHUB_ID, ABS_Y, PRECISION_RAW_Y_MIN, PRECISION_RAW_Y_MAX,
		0, 0);
		}

		static const int32_t PRECISION_RAW_X_MIN = TouchInputMapperTest::RAW_X_MIN;
		static const int32_t PRECISION_RAW_X_MAX = PRECISION_RAW_X_MIN + DISPLAY_WIDTH * 2 - 1;
		static const int32_t PRECISION_RAW_Y_MIN = TouchInputMapperTest::RAW_Y_MIN;
		static const int32_t PRECISION_RAW_Y_MAX = PRECISION_RAW_Y_MIN + DISPLAY_HEIGHT * 4 - 1;

		static const std::array<Point, 4> kRawCorners;
		};

		const std::array<Point, 4> TouchscreenPrecisionTestsFixture::kRawCorners = {{
		{PRECISION_RAW_X_MIN, PRECISION_RAW_Y_MIN}, // left-top
		{PRECISION_RAW_X_MAX, PRECISION_RAW_Y_MIN}, // right-top
		{PRECISION_RAW_X_MAX, PRECISION_RAW_Y_MAX}, // right-bottom
		{PRECISION_RAW_X_MIN, PRECISION_RAW_Y_MAX}, // left-bottom
		}};

		// Tests for how the touchscreen is oriented relative to the natural orientation of the display.
		// For example, if a touchscreen is configured with an orientation of 90 degrees, it is a portrait
		// touchscreen panel that is used on a device whose natural display orientation is in landscape.
		TEST_P(TouchscreenPrecisionTestsFixture, OrientationPrecision) {
		enum class Orientation {
		ORIENTATION_0 = ui::toRotationInt(ui::ROTATION_0),
		ORIENTATION_90 = ui::toRotationInt(ui::ROTATION_90),
		ORIENTATION_180 = ui::toRotationInt(ui::ROTATION_180),
		ORIENTATION_270 = ui::toRotationInt(ui::ROTATION_270),
		ftl_last = ORIENTATION_270,
		};
		using Orientation::ORIENTATION_0, Orientation::ORIENTATION_90, Orientation::ORIENTATION_180,
		Orientation::ORIENTATION_270;
		static const std::map<Orientation, std::array<vec2, 4> /mappedCorners/> kMappedCorners = {
		{ORIENTATION_0, {{{0, 0}, {479.5, 0}, {479.5, 799.75}, {0, 799.75}}}},
		{ORIENTATION_90, {{{0, 479.5}, {0, 0}, {799.75, 0}, {799.75, 479.5}}}},
		{ORIENTATION_180, {{{479.5, 799.75}, {0, 799.75}, {0, 0}, {479.5, 0}}}},
		{ORIENTATION_270, {{{799.75, 0}, {799.75, 479.5}, {0, 479.5}, {0, 0}}}},
		};

		const auto touchscreenOrientation = static_cast<Orientation>(ui::toRotationInt(GetParam()));

		// Configure the touchscreen as being installed in the one of the four different orientations
		// relative to the display.
		addConfigurationProperty("touch.deviceType", "touchScreen");
		addConfigurationProperty("touch.orientation", ftl::enum_string(touchscreenOrientation).c_str());
		prepareDisplay(ui::ROTATION_0);

		SingleTouchInputMapper& mapper = addMapperAndConfigure<SingleTouchInputMapper>();

		// If the touchscreen is installed in a rotated orientation relative to the display (i.e. in
		// orientations of either 90 or 270) this means the display's natural resolution will be
		// flipped.
		const bool displayRotated =
		touchscreenOrientation == ORIENTATION_90 \|\| touchscreenOrientation == ORIENTATION_270;
		const int32_t width = displayRotated ? DISPLAY_HEIGHT : DISPLAY_WIDTH;
		const int32_t height = displayRotated ? DISPLAY_WIDTH : DISPLAY_HEIGHT;
		const Rect physicalFrame{0, 0, width, height};
		configurePhysicalDisplay(ui::ROTATION_0, physicalFrame, width, height);

		const auto& expectedPoints = kMappedCorners.at(touchscreenOrientation);
		const float expectedPrecisionX = displayRotated ? 4 : 2;
		const float expectedPrecisionY = displayRotated ? 2 : 4;

		// Test all four corners.
		for (int i = 0; i < 4; i++) {
		const auto& raw = kRawCorners[i];
		processDown(mapper, raw.x, raw.y);
		processSync(mapper);
		const auto& expected = expectedPoints[i];
		ASSERT_NO_FATAL_FAILURE(mFakeListener->assertNotifyMotionWasCalled(
		AllOf(WithMotionAction(AMOTION_EVENT_ACTION_DOWN),
		WithCoords(expected.x, expected.y),
		WithPrecision(expectedPrecisionX, expectedPrecisionY))))
		<< "Failed to process raw point (" << raw.x << ", " << raw.y << ") "
		<< "with touchscreen orientation "
		<< ftl::enum_string(touchscreenOrientation).c_str() << ", expected point ("
		<< expected.x << ", " << expected.y << ").";
		processUp(mapper);
		processSync(mapper);
		ASSERT_NO_FATAL_FAILURE(mFakeListener->assertNotifyMotionWasCalled(
		AllOf(WithMotionAction(AMOTION_EVENT_ACTION_UP),
		WithCoords(expected.x, expected.y))));
		}
		}

		TEST_P(TouchscreenPrecisionTestsFixture, RotationPrecisionWhenOrientationAware) {
		static const std::map<ui::Rotation /rotation/, std::array<vec2, 4> /mappedCorners/>
		kMappedCorners = {
		{ui::ROTATION_0, {{{0, 0}, {479.5, 0}, {479.5, 799.75}, {0, 799.75}}}},
		{ui::ROTATION_90, {{{0.5, 0}, {480, 0}, {480, 799.75}, {0.5, 799.75}}}},
		{ui::ROTATION_180, {{{0.5, 0.25}, {480, 0.25}, {480, 800}, {0.5, 800}}}},
		{ui::ROTATION_270, {{{0, 0.25}, {479.5, 0.25}, {479.5, 800}, {0, 800}}}},
		};

		const ui::Rotation displayRotation = GetParam();

		addConfigurationProperty("touch.deviceType", "touchScreen");
		prepareDisplay(displayRotation);

		SingleTouchInputMapper& mapper = addMapperAndConfigure<SingleTouchInputMapper>();

		const auto& expectedPoints = kMappedCorners.at(displayRotation);

		// Test all four corners.
		for (int i = 0; i < 4; i++) {
		const auto& expected = expectedPoints[i];
		const auto& raw = kRawCorners[i];
		processDown(mapper, raw.x, raw.y);
		processSync(mapper);
		ASSERT_NO_FATAL_FAILURE(mFakeListener->assertNotifyMotionWasCalled(
		AllOf(WithMotionAction(AMOTION_EVENT_ACTION_DOWN),
		WithCoords(expected.x, expected.y), WithPrecision(2, 4))))
		<< "Failed to process raw point (" << raw.x << ", " << raw.y << ") "
		<< "with display rotation " << ui::toCString(displayRotation)
		<< ", expected point (" << expected.x << ", " << expected.y << ").";
		processUp(mapper);
		processSync(mapper);
		ASSERT_NO_FATAL_FAILURE(mFakeListener->assertNotifyMotionWasCalled(
		AllOf(WithMotionAction(AMOTION_EVENT_ACTION_UP),
		WithCoords(expected.x, expected.y))));
		}
		}

		TEST_P(TouchscreenPrecisionTestsFixture, RotationPrecisionOrientationAwareInOri270) {
		static const std::map<ui::Rotation /orientation/, std::array<vec2, 4> /mappedCorners/>
		kMappedCorners = {
		{ui::ROTATION_0, {{{799.75, 0}, {799.75, 479.5}, {0, 479.5}, {0, 0}}}},
		{ui::ROTATION_90, {{{800, 0}, {800, 479.5}, {0.25, 479.5}, {0.25, 0}}}},
		{ui::ROTATION_180, {{{800, 0.5}, {800, 480}, {0.25, 480}, {0.25, 0.5}}}},
		{ui::ROTATION_270, {{{799.75, 0.5}, {799.75, 480}, {0, 480}, {0, 0.5}}}},
		};

		const ui::Rotation displayRotation = GetParam();

		addConfigurationProperty("touch.deviceType", "touchScreen");
		addConfigurationProperty("touch.orientation", "ORIENTATION_270");

		SingleTouchInputMapper& mapper = addMapperAndConfigure<SingleTouchInputMapper>();

		// Ori 270, so width and height swapped
		const Rect physicalFrame{0, 0, DISPLAY_HEIGHT, DISPLAY_WIDTH};
		prepareDisplay(displayRotation);
		configurePhysicalDisplay(displayRotation, physicalFrame, DISPLAY_HEIGHT, DISPLAY_WIDTH);

		const auto& expectedPoints = kMappedCorners.at(displayRotation);

		// Test all four corners.
		for (int i = 0; i < 4; i++) {
		const auto& expected = expectedPoints[i];
		const auto& raw = kRawCorners[i];
		processDown(mapper, raw.x, raw.y);
		processSync(mapper);
		ASSERT_NO_FATAL_FAILURE(mFakeListener->assertNotifyMotionWasCalled(
		AllOf(WithMotionAction(AMOTION_EVENT_ACTION_DOWN),
		WithCoords(expected.x, expected.y), WithPrecision(4, 2))))
		<< "Failed to process raw point (" << raw.x << ", " << raw.y << ") "
		<< "with display rotation " << ui::toCString(displayRotation)
		<< ", expected point (" << expected.x << ", " << expected.y << ").";
		processUp(mapper);
		processSync(mapper);
		ASSERT_NO_FATAL_FAILURE(mFakeListener->assertNotifyMotionWasCalled(
		AllOf(WithMotionAction(AMOTION_EVENT_ACTION_UP),
		WithCoords(expected.x, expected.y))));
		}
		}

		// Run the precision tests for all rotations.
		INSTANTIATE_TEST_SUITE_P(TouchscreenPrecisionTests, TouchscreenPrecisionTestsFixture,
		::testing::Values(ui::ROTATION_0, ui::ROTATION_90, ui::ROTATION_180,
		ui::ROTATION_270),
		[](const testing::TestParamInfo<ui::Rotation>& testParamInfo) {
		return ftl::enum_string(testParamInfo.param);
		});

		// --- ExternalStylusFusionTest ---

		class ExternalStylusFusionTest : public SingleTouchInputMapperTest {

services/inputflinger/tests/TestInputListenerMatchers.h

+6 −0

Original line number	Diff line number	Diff line
		@@ -176,4 +176,10 @@ MATCHER_P(WithDownTime, downTime, "InputEvent with specified downTime") {
		return arg.downTime == downTime;
		}

		MATCHER_P2(WithPrecision, xPrecision, yPrecision, "MotionEvent with specified precision") {
		*result_listener << "expected x-precision " << xPrecision << " and y-precision " << yPrecision
		<< ", but got " << arg.xPrecision << " and " << arg.yPrecision;
		return arg.xPrecision == xPrecision && arg.yPrecision == yPrecision;
		}

		} // namespace android