Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 0d04e245 authored by Svetoslav Ganov's avatar Svetoslav Ganov
Browse files

Improving accessibility APIs used for UI automation. 

1. UiTestAutomationBridge was accessing the root node in the
   active window by tracking the accessibility event stream
   and keeping the last active window changing event. Now
   the bridge is stateless and the root node is fetched by
   passing special window and view id with the request to
   the system.

2. AccessibilityNodeInfos that are cached were not finished,
   i.e. not sealed, causing exception when trying to access
   their children or rpedecessors.

3. AccessibilityManagerService was not properly restoring its
   state after the UI automation bridge disconnects from it.
   I particular the devices was still in explore by touch mode
   event if no services are enabled and the sutomation bridge
   is disconnected.

4. ViewRootImpl for the focused window now fires accessibility
   events when accessibility is enabled to allow accessibility
   services to determine the current user location.

5. Several missing null checks in ViewRootImpl are fixed since
   there were scenraios in which a NPE can occur.

6. Update the internal window content querying tests.

7. ViewRootImpl was firing one extra focus event.
bug:6009813
bug:6026952

Change-Id: Ib2e058d64538ecc268f9ef7a8f36ead047868a05
parent b6ad5b14

core/java/android/accessibilityservice/IAccessibilityServiceConnection.aidl

+17 −17
Original line number Diff line number Diff line
@@ -30,14 +30,14 @@ interface IAccessibilityServiceConnection { 
    void setServiceInfo(in AccessibilityServiceInfo info);



    /**

     * Finds an {@link AccessibilityNodeInfo} by accessibility id.

     * Finds an {@link android.view.accessibility.AccessibilityNodeInfo} by accessibility id.

     *

     * @param accessibilityWindowId A unique window id. Use

     *     {@link com.android.server.accessibility.AccessibilityManagerService#ACTIVE_WINDOW_ID}

     *     {@link android.view.accessibility.AccessibilityNodeInfo#ACTIVE_WINDOW_ID}

     *     to query the currently active window.

     * @param accessibilityNodeId A unique view id or virtual descendant id from

     *     where to start the search. Use

     *     {@link com.android.server.accessibility.AccessibilityManagerService#ROOT_NODE_ID}

     *     {@link android.view.accessibility.AccessibilityNodeInfo#ROOT_NODE_ID}

     *     to start from the root.

     * @param interactionId The id of the interaction for matching with the callback result.

     * @param callback Callback which to receive the result.
@@ -49,17 +49,16 @@ interface IAccessibilityServiceConnection { 
        IAccessibilityInteractionConnectionCallback callback, long threadId);



    /**

     * Finds {@link AccessibilityNodeInfo}s by View text. The match is case

     * insensitive containment. The search is performed in the window whose

     * id is specified and starts from the node whose accessibility id is

     * specified.

     * Finds {@link android.view.accessibility.AccessibilityNodeInfo}s by View text.

     * The match is case insensitive containment. The search is performed in the window

     * whose id is specified and starts from the node whose accessibility id is specified.

     *

     * @param accessibilityWindowId A unique window id. Use

     *     {@link com.android.server.accessibility.AccessibilityManagerService#ACTIVE_WINDOW_ID}

     *     {@link android.view.accessibility.AccessibilityNodeInfo#ACTIVE_WINDOW_ID}

     *     to query the currently active window.

     * @param accessibilityNodeId A unique view id or virtual descendant id from

     *     where to start the search. Use

     *     {@link com.android.server.accessibility.AccessibilityManagerService#ROOT_NODE_ID}

     *     {@link android.view.accessibility.AccessibilityNodeInfo#ROOT_NODE_ID}

     *     to start from the root.

     * @param text The searched text.

     * @param interactionId The id of the interaction for matching with the callback result.
@@ -72,16 +71,16 @@ interface IAccessibilityServiceConnection { 
        long threadId);



    /**

     * Finds an {@link AccessibilityNodeInfo} by View id. The search is performed in

     * the window whose id is specified and starts from the node whose accessibility

     * id is specified.

     * Finds an {@link android.view.accessibility.AccessibilityNodeInfo} by View id. The search

     * is performed in the window whose id is specified and starts from the node whose

     * accessibility id is specified.

     *

     * @param accessibilityWindowId A unique window id. Use

     *     {@link com.android.server.accessibility.AccessibilityManagerService#ACTIVE_WINDOW_ID}

     *     {@link android.view.accessibility.AccessibilityNodeInfo#ACTIVE_WINDOW_ID}

     *     to query the currently active window.

     * @param accessibilityNodeId A unique view id or virtual descendant id from

     *     where to start the search. Use

     *     {@link com.android.server.accessibility.AccessibilityManagerService#ROOT_NODE_ID}

     *     {@link android.view.accessibility.AccessibilityNodeInfo#ROOT_NODE_ID}

     *     to start from the root.

     * @param id The id of the node.

     * @param interactionId The id of the interaction for matching with the callback result.
@@ -94,14 +93,15 @@ interface IAccessibilityServiceConnection { 
        long threadId);



    /**

     * Performs an accessibility action on an {@link AccessibilityNodeInfo}.

     * Performs an accessibility action on an

     * {@link android.view.accessibility.AccessibilityNodeInfo}.

     *

     * @param accessibilityWindowId A unique window id. Use

     *     {@link com.android.server.accessibility.AccessibilityManagerService#ACTIVE_WINDOW_ID}

     *     {@link android.view.accessibility.AccessibilityNodeInfo#ACTIVE_WINDOW_ID}

     *     to query the currently active window.

     * @param accessibilityNodeId A unique view id or virtual descendant id from

     *     where to start the search. Use

     *     {@link com.android.server.accessibility.AccessibilityManagerService#ROOT_NODE_ID}

     *     {@link android.view.accessibility.AccessibilityNodeInfo#ROOT_NODE_ID}

     *     to start from the root.

     * @param action The action to perform.

     * @param interactionId The id of the interaction for matching with the callback result.

core/java/android/accessibilityservice/UiTestAutomationBridge.java

+59 −31
Original line number Diff line number Diff line
@@ -49,11 +49,13 @@ public class UiTestAutomationBridge { 


    private static final String LOG_TAG = UiTestAutomationBridge.class.getSimpleName();



    public static final int ACTIVE_WINDOW_ID = -1;

    private static final int TIMEOUT_REGISTER_SERVICE = 5000;



    public static final long ROOT_NODE_ID = -1;

    public static final int ACTIVE_WINDOW_ID = AccessibilityNodeInfo.ACTIVE_WINDOW_ID;



    private static final int TIMEOUT_REGISTER_SERVICE = 5000;

    public static final long ROOT_NODE_ID = AccessibilityNodeInfo.ROOT_NODE_ID;



    public static final int UNDEFINED = -1;



    private final Object mLock = new Object();
@@ -63,8 +65,6 @@ public class UiTestAutomationBridge { 


    private AccessibilityEvent mLastEvent;



    private AccessibilityEvent mLastWindowStateChangeEvent;



    private volatile boolean mWaitingForEventDelivery;



    private volatile boolean mUnprocessedEventAvailable;
@@ -141,17 +141,8 @@ public class UiTestAutomationBridge { 
                synchronized (mLock) {

                    while (true) {

                        mLastEvent = AccessibilityEvent.obtain(event);



                        final int eventType = event.getEventType();

                        if (eventType == AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED

                                || eventType == AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED) {

                            if (mLastWindowStateChangeEvent != null) {

                                mLastWindowStateChangeEvent.recycle();

                            }

                            mLastWindowStateChangeEvent = mLastEvent;

                        }



                        if (!mWaitingForEventDelivery) {

                            mLock.notifyAll();

                            break;

                        }

                        if (!mUnprocessedEventAvailable) {
@@ -294,6 +285,43 @@ public class UiTestAutomationBridge { 
        }

    }



    /**

     * Waits for the accessibility event stream to become idle, which is not to

     * have received a new accessibility event within <code>idleTimeout</code>,

     * and do so within a maximal global timeout as specified by

     * <code>globalTimeout</code>.

     *

     * @param idleTimeout The timeout between two event to consider the device idle.

     * @param globalTimeout The maximal global timeout in which to wait for idle.

     */

    public void waitForIdle(long idleTimeout, long globalTimeout) {

        final long startTimeMillis = SystemClock.uptimeMillis();

        long lastEventTime = (mLastEvent != null)

                ? mLastEvent.getEventTime() : SystemClock.uptimeMillis();

        synchronized (mLock) {

            while (true) {

                final long currentTimeMillis = SystemClock.uptimeMillis();

                final long sinceLastEventTimeMillis = currentTimeMillis - lastEventTime;

                if (sinceLastEventTimeMillis > idleTimeout) {

                    return;

                }

                if (mLastEvent != null) {

                    lastEventTime = mLastEvent.getEventTime();

                }

                final long elapsedTimeMillis = SystemClock.uptimeMillis() - startTimeMillis;

                final long remainingTimeMillis = globalTimeout - elapsedTimeMillis;

                if (remainingTimeMillis <= 0) {

                    return;

                }

                try {

                     mLock.wait(idleTimeout);

                } catch (InterruptedException e) {

                     /* ignore */

                }

            }

        }

    }



    /**

     * Finds an {@link AccessibilityNodeInfo} by accessibility id in the active

     * window. The search is performed from the root node.
@@ -310,8 +338,8 @@ public class UiTestAutomationBridge { 
    /**

     * Finds an {@link AccessibilityNodeInfo} by accessibility id.

     *

     * @param accessibilityWindowId A unique window id. Use {@link #ACTIVE_WINDOW_ID}

     *     to query the currently active window.

     * @param accessibilityWindowId A unique window id. Use {@link #ACTIVE_WINDOW_ID} to query

     *     the currently active window.

     * @param accessibilityNodeId A unique view id or virtual descendant id for

     *     which to search.

     * @return The current window scale, where zero means a failure.
@@ -341,8 +369,8 @@ public class UiTestAutomationBridge { 
     * the window whose id is specified and starts from the node whose accessibility

     * id is specified.

     *

     * @param accessibilityWindowId A unique window id. Use {@link #ACTIVE_WINDOW_ID}

     *     to query the currently active window.

     * @param accessibilityWindowId A unique window id. Use

     *     {@link  #ACTIVE_WINDOW_ID} to query the currently active window.

     * @param accessibilityNodeId A unique view id or virtual descendant id from

     *     where to start the search. Use {@link  #ROOT_NODE_ID} to start from the root.

     * @return The current window scale, where zero means a failure.
@@ -374,8 +402,8 @@ public class UiTestAutomationBridge { 
     * id is specified and starts from the node whose accessibility id is

     * specified.

     *

     * @param accessibilityWindowId A unique window id. Use {@link #ACTIVE_WINDOW_ID}

     *     to query the currently active window.

     * @param accessibilityWindowId A unique window id. Use

     *     {@link #ACTIVE_WINDOW_ID} to query the currently active window.

     * @param accessibilityNodeId A unique view id or virtual descendant id from

     *     where to start the search. Use {@link #ROOT_NODE_ID} to start from the root.

     * @param text The searched text.
@@ -406,8 +434,8 @@ public class UiTestAutomationBridge { 
    /**

     * Performs an accessibility action on an {@link AccessibilityNodeInfo}.

     *

     * @param accessibilityWindowId A unique window id. Use {@link #ACTIVE_WINDOW_ID}

     *     to query the currently active window.

     * @param accessibilityWindowId A unique window id. Use

     *     {@link #ACTIVE_WINDOW_ID} to query the currently active window.

     * @param accessibilityNodeId A unique node id (accessibility and virtual descendant id).

     * @param action The action to perform.

     * @return Whether the action was performed.
@@ -427,16 +455,16 @@ public class UiTestAutomationBridge { 
     * @return The root info.

     */

    public AccessibilityNodeInfo getRootAccessibilityNodeInfoInActiveWindow() {

        synchronized (mLock) {

            if (mLastWindowStateChangeEvent != null) {

                return mLastWindowStateChangeEvent.getSource();

            }

        }

        return null;

        // Cache the id to avoid locking

        final int connectionId = mConnectionId;

        ensureValidConnection(connectionId);

        return AccessibilityInteractionClient.getInstance()

                .findAccessibilityNodeInfoByAccessibilityId(connectionId, ACTIVE_WINDOW_ID,

                        ROOT_NODE_ID);

    }



    private void ensureValidConnection(int connectionId) {

        if (connectionId == AccessibilityInteractionClient.NO_ID) {

        if (connectionId == UNDEFINED) {

            throw new IllegalStateException("UiAutomationService not connected."

                    + " Did you call #register()?");

        }

core/java/android/view/AccessibilityNodeInfoCache.java

+27 −2
Original line number Diff line number Diff line
@@ -16,6 +16,8 @@ 


package android.view;



import android.os.Process;

import android.util.Log;

import android.util.LongSparseArray;

import android.view.accessibility.AccessibilityEvent;

import android.view.accessibility.AccessibilityNodeInfo;
@@ -30,7 +32,11 @@ import android.view.accessibility.AccessibilityNodeInfo; 
 */

public class AccessibilityNodeInfoCache {



    private final boolean ENABLED = true;

    private static final String LOG_TAG = AccessibilityNodeInfoCache.class.getSimpleName();



    private static final boolean ENABLED = true;



    private static final boolean DEBUG = false;



    /**

     * @return A new <strong>not synchronized</strong> AccessibilityNodeInfoCache.
@@ -95,6 +101,7 @@ public class AccessibilityNodeInfoCache { 
    public void onAccessibilityEvent(AccessibilityEvent event) {

        final int eventType = event.getEventType();

        switch (eventType) {

            case AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED:

            case AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED:

            case AccessibilityEvent.TYPE_VIEW_SCROLLED:

                clear();
@@ -117,7 +124,14 @@ public class AccessibilityNodeInfoCache { 
     */

    public AccessibilityNodeInfo get(long accessibilityNodeId) {

        if (ENABLED) {

            if (DEBUG) {

                AccessibilityNodeInfo info = mCacheImpl.get(accessibilityNodeId);

                Log.i(LOG_TAG, "Process: " + Process.myPid() +

                        " get(" + accessibilityNodeId + ") = " + info);

                return info;

            } else {

                return mCacheImpl.get(accessibilityNodeId);

            }

        } else {

            return null;

        }
@@ -131,6 +145,10 @@ public class AccessibilityNodeInfoCache { 
     */

    public void put(long accessibilityNodeId, AccessibilityNodeInfo info) {

        if (ENABLED) {

            if (DEBUG) {

                Log.i(LOG_TAG, "Process: " + Process.myPid()

                        + " put(" + accessibilityNodeId + ", " + info + ")");

            }

            mCacheImpl.put(accessibilityNodeId, info);

        }

    }
@@ -156,6 +174,10 @@ public class AccessibilityNodeInfoCache { 
     */

    public void remove(long accessibilityNodeId) {

        if (ENABLED) {

            if (DEBUG) {

                Log.i(LOG_TAG,  "Process: " + Process.myPid()

                        + " remove(" + accessibilityNodeId + ")");

            }

            mCacheImpl.remove(accessibilityNodeId);

        }

    }
@@ -165,6 +187,9 @@ public class AccessibilityNodeInfoCache { 
     */

    public void clear() {

        if (ENABLED) {

            if (DEBUG) {

                Log.i(LOG_TAG,  "Process: " + Process.myPid() + "clear()");

            }

            mCacheImpl.clear();

        }

    }

core/java/android/view/ViewRootImpl.java

+40 −37
Original line number Diff line number Diff line
@@ -2647,8 +2647,8 @@ public final class ViewRootImpl implements ViewParent, 
                        mHasHadWindowFocus = true;

                    }



                    if (hasWindowFocus && mView != null) {

                        sendAccessibilityEvents();

                    if (hasWindowFocus && mView != null && mAccessibilityManager.isEnabled()) {

                        mView.sendAccessibilityEvent(AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED);

                    }

                }

            } break;
@@ -4062,21 +4062,6 @@ public final class ViewRootImpl implements ViewParent, 
        }

    }



    /**

     * The window is getting focus so if there is anything focused/selected

     * send an {@link AccessibilityEvent} to announce that.

     */

    private void sendAccessibilityEvents() {

        if (!mAccessibilityManager.isEnabled()) {

            return;

        }

        mView.sendAccessibilityEvent(AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED);

        View focusedView = mView.findFocus();

        if (focusedView != null && focusedView != mView) {

            focusedView.sendAccessibilityEvent(AccessibilityEvent.TYPE_VIEW_FOCUSED);

        }

    }



    /**

     * Post a callback to send a

     * {@link AccessibilityEvent#TYPE_WINDOW_CONTENT_CHANGED} event.
@@ -4646,24 +4631,35 @@ public final class ViewRootImpl implements ViewParent, 
        public void onAccessibilityStateChanged(boolean enabled) {

            if (enabled) {

                ensureConnection();

                if (mAttachInfo != null && mAttachInfo.mHasWindowFocus) {

                    mView.sendAccessibilityEvent(AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED);

                    View focusedView = mView.findFocus();

                    if (focusedView != null && focusedView != mView) {

                        focusedView.sendAccessibilityEvent(AccessibilityEvent.TYPE_VIEW_FOCUSED);

                    }

                }

            } else {

                ensureNoConnection();

            }

        }



        public void ensureConnection() {

            final boolean registered = mAttachInfo.mAccessibilityWindowId != View.NO_ID;

            if (mAttachInfo != null) {

                final boolean registered =

                    mAttachInfo.mAccessibilityWindowId != AccessibilityNodeInfo.UNDEFINED;

                if (!registered) {

                    mAttachInfo.mAccessibilityWindowId =

                        mAccessibilityManager.addAccessibilityInteractionConnection(mWindow,

                                new AccessibilityInteractionConnection(ViewRootImpl.this));

                }

            }

        }



        public void ensureNoConnection() {

            final boolean registered = mAttachInfo.mAccessibilityWindowId != View.NO_ID;

            final boolean registered =

                mAttachInfo.mAccessibilityWindowId != AccessibilityNodeInfo.UNDEFINED;

            if (registered) {

                mAttachInfo.mAccessibilityWindowId = View.NO_ID;

                mAttachInfo.mAccessibilityWindowId = AccessibilityNodeInfo.UNDEFINED;

                mAccessibilityManager.removeAccessibilityInteractionConnection(mWindow);

            }

        }
@@ -4860,16 +4856,23 @@ public final class ViewRootImpl implements ViewParent, 
            List<AccessibilityNodeInfo> infos = mTempAccessibilityNodeInfoList;

            infos.clear();

            try {

                if (accessibilityViewId == AccessibilityNodeInfo.UNDEFINED) {

                    View target = ViewRootImpl.this.mView;

                    if (target != null && target.getVisibility() == View.VISIBLE) {

                        infos.add(target.createAccessibilityNodeInfo());

                    }

                } else {

                    View target = findViewByAccessibilityId(accessibilityViewId);

                    if (target != null && target.getVisibility() == View.VISIBLE) {

                        AccessibilityNodeProvider provider = target.getAccessibilityNodeProvider();

                        if (provider != null) {

                            infos.add(provider.createAccessibilityNodeInfo(virtualDescendantId));

                    } else if (virtualDescendantId == View.NO_ID) {

                        } else if (virtualDescendantId == AccessibilityNodeInfo.UNDEFINED) {

                            getAccessibilityPrefetchStrategy().prefetchAccessibilityNodeInfos(

                                    interrogatingPid, target, infos);

                        }

                    }

                }

            } finally {

                try {

                    callback.setFindAccessibilityNodeInfosResult(infos, interactionId);
@@ -4915,7 +4918,7 @@ public final class ViewRootImpl implements ViewParent, 
            AccessibilityNodeInfo info = null;

            try {

                View root = null;

                if (accessibilityViewId != View.NO_ID) {

                if (accessibilityViewId != AccessibilityNodeInfo.UNDEFINED) {

                    root = findViewByAccessibilityId(accessibilityViewId);

                } else {

                    root = ViewRootImpl.this.mView;
@@ -4973,7 +4976,7 @@ public final class ViewRootImpl implements ViewParent, 
            List<AccessibilityNodeInfo> infos = null;

            try {

                View target;

                if (accessibilityViewId != View.NO_ID) {

                if (accessibilityViewId != AccessibilityNodeInfo.UNDEFINED) {

                    target = findViewByAccessibilityId(accessibilityViewId);

                } else {

                    target = ViewRootImpl.this.mView;
@@ -4983,7 +4986,7 @@ public final class ViewRootImpl implements ViewParent, 
                    if (provider != null) {

                        infos = provider.findAccessibilityNodeInfosByText(text,

                                virtualDescendantId);

                    } else if (virtualDescendantId == View.NO_ID) {

                    } else if (virtualDescendantId == AccessibilityNodeInfo.UNDEFINED) {

                        ArrayList<View> foundViews = mAttachInfo.mFocusablesTempList;

                        foundViews.clear();

                        target.findViewsWithText(foundViews, text, View.FIND_VIEWS_WITH_TEXT
@@ -5063,7 +5066,7 @@ public final class ViewRootImpl implements ViewParent, 
                    if (provider != null) {

                        succeeded = provider.performAccessibilityAction(action,

                                virtualDescendantId);

                    } else if (virtualDescendantId == View.NO_ID) {

                    } else if (virtualDescendantId == AccessibilityNodeInfo.UNDEFINED) {

                        switch (action) {

                            case AccessibilityNodeInfo.ACTION_FOCUS: {

                                if (!target.hasFocus()) {
@@ -5171,7 +5174,7 @@ public final class ViewRootImpl implements ViewParent, 
        private void addAndCacheNotCachedNodeInfo(long interrogatingPid,

                View view, List<AccessibilityNodeInfo> outInfos) {

            final long accessibilityNodeId = AccessibilityNodeInfo.makeNodeId(

                    view.getAccessibilityViewId(), View.NO_ID);

                    view.getAccessibilityViewId(), AccessibilityNodeInfo.UNDEFINED);

            AccessibilityNodeInfoCache cache = getCacheForInterrogatingPid(interrogatingPid);

            if (!cache.containsKey(accessibilityNodeId)) {

                // Account for the ids of the fetched infos. The infos will be

core/java/android/view/accessibility/AccessibilityInteractionClient.java

+28 −32

File changed.

Preview size limit exceeded, changes collapsed.