Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 24c90450 authored by Svetoslav Ganov's avatar Svetoslav Ganov
Browse files

Autofill compatibility mode. 

Autofill helps users fill credentials, addresses, payment methods,
emails, etc without manually typing. When focus lands on a fillable
element the platform captures a snapshot of the screen content and
sends it to an autofill service for analysis and suggestions. The
screen snapshot is a structured representation of the screen content.
If this content is composed of standard widgets, autofill works
out-of-the-box. However, some apps do their own rendering and
the content in this case looks like a single view to the platform
while it may have semantic structure. For example, a view may render
a login page with two input test fields.

The platform exposes APIs for apps to report virtual view structure
allowing autofill services to handle apps that have virtual content.
As opposed to apps using standard widgets, this case requires the app
developer to implement the new APIs which may require a fair amount
of code and could be seen as a processes that could take some time.
The most prominent typs of apps that fall into this category are
browsers.

Until most apps rendering virtual content and specifically browsers
don't implement the virutal APIs, autofill providers need to fall-
back to using the accessibliity APIs to provide autofill support
for these apps. This requires developers to work against two sets
of APIs - autofill and accessibility - which is incovenient and error
prone. Also, users need to enable two plugins - autofill and
accessibility which is confusing. Additionally, the privacy and
perfomance impact of using the accessibility APIs cannot be addressed
while autofill providers need to use thes APis.

This change adds an autofill compatibility mode that would allow
autofill services to work with apps that don't implement the
virtual structure autofill APIs. The key idea is to locally enable
accessibility for the target package and remap accessibility to
autofill APIs and vise versa. This way an autofill provider codes
against a single set of APIs, the users enable a single plugin,
the privacy/performance implications of using the accessibility
APIs are addressed, the target app only takes a performance hit
since accessibility is enabled locally which is still more efficient
compared to the performance hit it would incur if accessibility is
enabled globally.

To enable compatibility mode an autofill service declares in its
metadata which packages it is interested in and also what is
the max version code of the package for which to enable compat
mode. Targeted versioning allows targeting only older versions of
the package that are known to not support autofill while newer
versions that are known to support autofill would work in normal
mode.

Since compatibility mode should be used only as a fallback we
have a white list setting with the packages for which this mode
can be requested. This allows applying policy to target only
apps that are known to not support autofill.

Test:
     cts-tradefed run cts-dev -m CtsAutoFillServiceTestCases
     cts-tradefed run cts-dev -m CtsAccessibilityServiceTestCases

bug:72811034

Change-Id: I11f1580ced0f8b4300a10b3a5174a1758a5702a0
parent 94baed8b

api/current.txt

+11 −9
Original line number Diff line number Diff line
@@ -902,6 +902,7 @@ package android { 
    field public static final int maxLength = 16843104; // 0x1010160

    field public static final int maxLevel = 16843186; // 0x10101b2

    field public static final int maxLines = 16843091; // 0x1010153

    field public static final int maxLongVersionCode = 16844163; // 0x1010583

    field public static final int maxRecents = 16843846; // 0x1010446

    field public static final int maxRows = 16843059; // 0x1010133

    field public static final int maxSdkVersion = 16843377; // 0x1010271
@@ -11616,15 +11617,15 @@ package android.content.res { 














  public final class AssetManager implements java.lang.AutoCloseable {















    method public void close();













    method public final java.lang.String[] getLocales();













    method public final java.lang.String[] list(java.lang.String) throws java.io.IOException;













    method public final java.io.InputStream open(java.lang.String) throws java.io.IOException;













    method public final java.io.InputStream open(java.lang.String, int) throws java.io.IOException;













    method public final android.content.res.AssetFileDescriptor openFd(java.lang.String) throws java.io.IOException;













    method public final android.content.res.AssetFileDescriptor openNonAssetFd(java.lang.String) throws java.io.IOException;













    method public final android.content.res.AssetFileDescriptor openNonAssetFd(int, java.lang.String) throws java.io.IOException;













    method public final android.content.res.XmlResourceParser openXmlResourceParser(java.lang.String) throws java.io.IOException;













    method public final android.content.res.XmlResourceParser openXmlResourceParser(int, java.lang.String) throws java.io.IOException;













    method public java.lang.String[] getLocales();













    method public java.lang.String[] list(java.lang.String) throws java.io.IOException;













    method public java.io.InputStream open(java.lang.String) throws java.io.IOException;













    method public java.io.InputStream open(java.lang.String, int) throws java.io.IOException;













    method public android.content.res.AssetFileDescriptor openFd(java.lang.String) throws java.io.IOException;













    method public android.content.res.AssetFileDescriptor openNonAssetFd(java.lang.String) throws java.io.IOException;













    method public android.content.res.AssetFileDescriptor openNonAssetFd(int, java.lang.String) throws java.io.IOException;













    method public android.content.res.XmlResourceParser openXmlResourceParser(java.lang.String) throws java.io.IOException;













    method public android.content.res.XmlResourceParser openXmlResourceParser(int, java.lang.String) throws java.io.IOException;















    field public static final int ACCESS_BUFFER = 3; // 0x3















    field public static final int ACCESS_RANDOM = 1; // 0x1















    field public static final int ACCESS_STREAMING = 2; // 0x2










@@ -47429,6 +47430,7 @@ package android.view {













    method public boolean isTextDirectionResolved();















    method public boolean isVerticalFadingEdgeEnabled();















    method public boolean isVerticalScrollBarEnabled();













    method public boolean isVisibleToUserForAutofill(int);















    method public void jumpDrawablesToCurrentState();















    method public android.view.View keyboardNavigationClusterSearch(android.view.View, int);















    method public void layout(int, int, int, int);

























api/system-current.txt



+1
−0




























Original line number
Diff line number
Diff line








@@ -4119,6 +4119,7 @@ package android.provider {













  public static final class Settings.Global extends android.provider.Settings.NameValueTable {















    method public static boolean putString(android.content.ContentResolver, java.lang.String, java.lang.String, java.lang.String, boolean);















    method public static void resetToDefaults(android.content.ContentResolver, java.lang.String);













    field public static final java.lang.String AUTOFILL_COMPAT_ALLOWED_PACKAGES = "autofill_compat_allowed_packages";















    field public static final java.lang.String OTA_DISABLE_AUTOMATIC_UPDATE = "ota_disable_automatic_update";















    field public static final java.lang.String THEATER_MODE_ON = "theater_mode_on";















    field public static final java.lang.String WEBVIEW_MULTIPROCESS = "webview_multiprocess";

































core/java/android/accessibilityservice/AccessibilityServiceInfo.java



+19
−1




























Original line number
Diff line number
Diff line








@@ -16,6 +16,7 @@





























package android.accessibilityservice;





























import android.annotation.IntDef;















import android.content.ComponentName;















import android.content.Context;















import android.content.pm.PackageManager;










@@ -43,6 +44,8 @@ import org.xmlpull.v1.XmlPullParser;













import org.xmlpull.v1.XmlPullParserException;































import java.io.IOException;













import java.lang.annotation.Retention;













import java.lang.annotation.RetentionPolicy;















import java.util.ArrayList;















import java.util.Collections;















import java.util.List;










@@ -346,6 +349,19 @@ public class AccessibilityServiceInfo implements Parcelable {













     */















    public String[] packageNames;











































    /** @hide */













    @IntDef(flag = true, prefix = { "FEEDBACK_" }, value = {













            FEEDBACK_AUDIBLE,













            FEEDBACK_GENERIC,













            FEEDBACK_HAPTIC,













            FEEDBACK_SPOKEN,













            FEEDBACK_VISUAL,













            FEEDBACK_BRAILLE













    })













    @Retention(RetentionPolicy.SOURCE)













    public @interface FeedbackType {}





























    /**















     * The feedback type an {@link AccessibilityService} provides.















     * <p>









@@ -358,6 +374,7 @@ public class AccessibilityServiceInfo implements Parcelable {













     * @see #FEEDBACK_VISUAL















     * @see #FEEDBACK_BRAILLE















     */













    @FeedbackType















    public int feedbackType;































    /**










@@ -818,7 +835,8 @@ public class AccessibilityServiceInfo implements Parcelable {













        return stringBuilder.toString();















    }





























    private static void appendFeedbackTypes(StringBuilder stringBuilder, int feedbackTypes) {













    private static void appendFeedbackTypes(StringBuilder stringBuilder,













            @FeedbackType int feedbackTypes) {















        stringBuilder.append("feedbackTypes:");















        stringBuilder.append("[");















        while (feedbackTypes != 0) {

































core/java/android/app/Activity.java



+87
−44




























Original line number
Diff line number
Diff line








@@ -113,6 +113,7 @@ import android.view.Window.WindowControllerCallback;













import android.view.WindowManager;















import android.view.WindowManagerGlobal;















import android.view.accessibility.AccessibilityEvent;













import android.view.autofill.AutofillId;















import android.view.autofill.AutofillManager;















import android.view.autofill.AutofillManager.AutofillClient;















import android.view.autofill.AutofillPopupWindow;









@@ -125,7 +126,6 @@ import com.android.internal.annotations.GuardedBy;













import com.android.internal.app.IVoiceInteractor;















import com.android.internal.app.ToolbarActionBar;















import com.android.internal.app.WindowDecorActionBar;













import com.android.internal.policy.DecorView;















import com.android.internal.policy.PhoneWindow;































import dalvik.system.VMRuntime;










@@ -5961,12 +5961,16 @@ public class Activity extends ContextThemeWrapper













     *















     * @return Returns the complete component name for this activity















     */













    @Override













    public ComponentName getComponentName()













    {













    public ComponentName getComponentName() {















        return mComponent;















    }





























    /** @hide */













    @Override













    public final ComponentName autofillClientGetComponentName() {













        return getComponentName();













    }





























    /**















     * Retrieve a {@link SharedPreferences} object for accessing preferences















     * that are private to this activity.  This simply calls the underlying










@@ -6262,7 +6266,6 @@ public class Activity extends ContextThemeWrapper













     *















     * @param action the action to run on the UI thread















     */













    @Override















    public final void runOnUiThread(Runnable action) {















        if (Thread.currentThread() != mUiThread) {















            mHandler.post(action);









@@ -6271,6 +6274,12 @@ public class Activity extends ContextThemeWrapper













        }















    }





























    /** @hide */













    @Override













    public final void autofillClientRunOnUiThread(Runnable action) {













        runOnUiThread(action);













    }





























    /**















     * Standard implementation of















     * {@link android.view.LayoutInflater.Factory#onCreateView} used when










@@ -7076,6 +7085,18 @@ public class Activity extends ContextThemeWrapper













        mCurrentConfig = config;































        mWindow.setColorMode(info.colorMode);



























        setAutofillCompatibilityEnabled(application.isAutofillCompatibilityEnabled());













        enableAutofillCompatibilityIfNeeded();













    }



























    private void enableAutofillCompatibilityIfNeeded() {













        if (isAutofillCompatibilityEnabled()) {













            final AutofillManager afm = getSystemService(AutofillManager.class);













            if (afm != null) {













                afm.enableCompatibilityMode();













            }













        }















    }































    /** @hide */










@@ -7572,7 +7593,7 @@ public class Activity extends ContextThemeWrapper





























    /** @hide */















    @Override













    final public void autofillCallbackAuthenticate(int authenticationId, IntentSender intent,













    public final void autofillClientAuthenticate(int authenticationId, IntentSender intent,















            Intent fillInIntent) {















        try {















            startIntentSenderForResultInner(intent, AUTO_FILL_AUTH_WHO_PREFIX,









@@ -7584,13 +7605,13 @@ public class Activity extends ContextThemeWrapper





























    /** @hide */















    @Override













    final public void autofillCallbackResetableStateAvailable() {













    public final void autofillClientResetableStateAvailable() {















        mAutoFillResetNeeded = true;















    }































    /** @hide */















    @Override













    final public boolean autofillCallbackRequestShowFillUi(@NonNull View anchor, int width,













    public final boolean autofillClientRequestShowFillUi(@NonNull View anchor, int width,















            int height, @Nullable Rect anchorBounds, IAutofillWindowPresenter presenter) {















        final boolean wasShowing;

























@@ -7607,7 +7628,7 @@ public class Activity extends ContextThemeWrapper





























    /** @hide */















    @Override













    final public boolean autofillCallbackRequestHideFillUi() {













    public final boolean autofillClientRequestHideFillUi() {















        if (mAutofillPopupWindow == null) {















            return false;















        }









@@ -7618,8 +7639,16 @@ public class Activity extends ContextThemeWrapper





























    /** @hide */















    @Override













    @NonNull public View[] findViewsByAutofillIdTraversal(@NonNull int[] viewIds) {













        final View[] views = new View[viewIds.length];













    public final boolean autofillClientIsFillUiShowing() {













        return mAutofillPopupWindow != null && mAutofillPopupWindow.isShowing();













    }



























    /** @hide */













    @Override













    @NonNull













    public final View[] autofillClientFindViewsByAutofillIdTraversal(













            @NonNull AutofillId[] autofillId) {













        final View[] views = new View[autofillId.length];















        final ArrayList<ViewRootImpl> roots =















                WindowManagerGlobal.getInstance().getRootViews(getActivityToken());

























@@ -7627,10 +7656,11 @@ public class Activity extends ContextThemeWrapper













            final View rootView = roots.get(rootNum).getView();































            if (rootView != null) {













                for (int viewNum = 0; viewNum < viewIds.length; viewNum++) {













                final int viewCount = autofillId.length;













                for (int viewNum = 0; viewNum < viewCount; viewNum++) {















                    if (views[viewNum] == null) {















                        views[viewNum] = rootView.findViewByAutofillIdTraversal(













                                viewIds[viewNum]);













                                autofillId[viewNum].getViewId());















                    }















                }















            }









@@ -7641,14 +7671,15 @@ public class Activity extends ContextThemeWrapper





























    /** @hide */















    @Override













    @Nullable public View findViewByAutofillIdTraversal(int viewId) {













    @Nullable













    public final View autofillClientFindViewByAutofillIdTraversal(AutofillId autofillId) {















        final ArrayList<ViewRootImpl> roots =















                WindowManagerGlobal.getInstance().getRootViews(getActivityToken());















        for (int rootNum = 0; rootNum < roots.size(); rootNum++) {















            final View rootView = roots.get(rootNum).getView();































            if (rootView != null) {













                final View view = rootView.findViewByAutofillIdTraversal(viewId);













                final View view = rootView.findViewByAutofillIdTraversal(autofillId.getViewId());















                if (view != null) {















                    return view;















                }









@@ -7660,50 +7691,62 @@ public class Activity extends ContextThemeWrapper





























    /** @hide */















    @Override













    @NonNull public boolean[] getViewVisibility(@NonNull int[] viewIds) {













        final boolean[] isVisible = new boolean[viewIds.length];













        final View views[] = findViewsByAutofillIdTraversal(viewIds);



























        for (int i = 0; i < viewIds.length; i++) {













            View view = views[i];













            if (view == null) {













                isVisible[i] = false;













                continue;













    public final @NonNull boolean[] autofillClientGetViewVisibility(













            @NonNull AutofillId[] autofillIds) {













        final int autofillIdCount = autofillIds.length;













        final boolean[] visible = new boolean[autofillIdCount];













        for (int i = 0; i < autofillIdCount; i++) {













            final AutofillId autofillId = autofillIds[i];













            final View view = autofillClientFindViewByAutofillIdTraversal(autofillId);













            if (view != null) {













                if (!autofillId.isVirtual()) {













                    visible[i] = view.isVisibleToUser();













                } else {













                    visible[i] = view.isVisibleToUserForAutofill(autofillId.getVirtualChildId());















                }



























            isVisible[i] = true;



























            // Check if the view is visible by checking all parents













            while (true) {













                if (view instanceof DecorView && view.getViewRootImpl() == view.getParent()) {













                    break;















            }



























                if (view.getVisibility() != View.VISIBLE) {













                    isVisible[i] = false;













                    break;













        }













        return visible;















    }





























                if (view.getParent() instanceof View) {













                    view = (View) view.getParent();













                } else {













                    break;













    /** @hide */













    public final @Nullable View autofillClientFindViewByAccessibilityIdTraversal(int viewId,













            int windowId) {













        final ArrayList<ViewRootImpl> roots = WindowManagerGlobal.getInstance()













                .getRootViews(getActivityToken());













        for (int rootNum = 0; rootNum < roots.size(); rootNum++) {













            final View rootView = roots.get(rootNum).getView();













            if (rootView != null && rootView.getAccessibilityWindowId() == windowId) {













                final View view = rootView.findViewByAccessibilityIdTraversal(viewId);













                if (view != null) {













                    return view;















                }















            }















        }













        return null;













    }





























        return isVisible;













    /** @hide */













    @Override













    public final @Nullable IBinder autofillClientGetActivityToken() {













        return getActivityToken();















    }































    /** @hide */















    @Override













    public boolean isVisibleForAutofill() {













    public final boolean autofillClientIsVisibleForAutofill() {















        return !mStopped;















    }































    /** @hide */















    @Override













    public boolean isDisablingEnterExitEventForAutofill() {













    public final boolean autofillIsCompatibilityModeEnabled() {













        return isAutofillCompatibilityEnabled();













    }



























    /** @hide */













    @Override













    public final boolean isDisablingEnterExitEventForAutofill() {















        return mAutoFillIgnoreFirstResumePause || !mResumed;















    }

















































core/java/android/app/ActivityThread.java



+8
−1




























Original line number
Diff line number
Diff line








@@ -637,6 +637,8 @@ public final class ActivityThread extends ClientTransactionHandler {













        /** Initial values for {@link Profiler}. */















        ProfilerInfo initProfilerInfo;





























        boolean autofillCompatibilityEnabled;





























        public String toString() {















            return "AppBindData{appInfo=" + appInfo + "}";















        }










@@ -863,7 +865,7 @@ public final class ActivityThread extends ClientTransactionHandler {













                boolean enableBinderTracking, boolean trackAllocation,















                boolean isRestrictedBackupMode, boolean persistent, Configuration config,















                CompatibilityInfo compatInfo, Map services, Bundle coreSettings,













                String buildSerial) {













                String buildSerial, boolean autofillCompatibilityEnabled) {































            if (services != null) {















                // Setup the service cache in the ServiceManager









@@ -889,6 +891,7 @@ public final class ActivityThread extends ClientTransactionHandler {













            data.compatInfo = compatInfo;















            data.initProfilerInfo = profilerInfo;















            data.buildSerial = buildSerial;













            data.autofillCompatibilityEnabled = autofillCompatibilityEnabled;















            sendMessage(H.BIND_APPLICATION, data);















        }


























@@ -5840,6 +5843,10 @@ public final class ActivityThread extends ClientTransactionHandler {













            // If the app is being launched for full backup or restore, bring it up in















            // a restricted environment with the base application class.















            app = data.info.makeApplication(data.restrictedBackupMode, null);



























            // Propagate autofill compat state













            app.setAutofillCompatibilityEnabled(data.autofillCompatibilityEnabled);





























            mInitialApplication = app;































            // don't bring up providers in restricted mode; they may depend on the