Initial docs for Ravenwood.

# Ravenwood

Ravenwood is an officially-supported lightweight unit testing environment for Android platform code that runs on the host.

Ravenwood’s focus on Android platform use-cases, improved maintainability, and device consistency distinguishes it from Robolectric, which remains a popular choice for app testing.

## Background

Executing tests on a typical Android device has substantial overhead, such as flashing the build, waiting for the boot to complete, and retrying tests that fail due to general flakiness.

In contrast, defining a lightweight unit testing environment mitigates these issues by running directly from build artifacts (no flashing required), runs immediately (no booting required), and runs in an isolated environment (less flakiness).

## Guiding principles

Here’s a summary of the guiding principles for Ravenwood, aimed at addressing Robolectric design concerns and better supporting Android platform developers:

* **API support for Ravenwood is opt-in.**  Teams that own APIs decide exactly what, and how, they support their API functionality being available to tests.  When an API hasn’t opted-in, the API signatures remain available for tests to compile against and/or mock, but they throw when called under a Ravenwood environment.

    * _Contrasted with Robolectric which attempts to run API implementations as-is, causing maintenance pains as teams maintain or redesign their API internals._

* **API support and customizations for Ravenwood appear directly inline with relevant code.** This improves maintenance of APIs by providing awareness of what code runs under Ravenwood, including the ability to replace code at a per-method level when Ravenwood-specific customization is needed.

    * _Contrasted with Robolectric which maintains customized behavior in separate “Shadow” classes that are difficult for maintainers to be aware of._

* **APIs supported under Ravenwood are tested to remain consistent with physical devices.**  As teams progressively opt-in supporting APIs under Ravenwood, we’re requiring they bring along “bivalent” tests (such as the relevant CTS) to validate that Ravenwood behaves just like a physical device.

    * _Contrasted with Robolectric, which has limited (and forked) testing of their environment, increasing their risk of accidental divergence over time and misleading “passing” signals._

* **Ravenwood aims to support more “real” code.**  As API owners progressively opt-in their code, they have the freedom to provide either a limited “fake” that is a faithful emulation of how a device behaves, or they can bring more “real” code that runs on physical devices.

    * _Contrasted with Robolectric, where support for “real” code ends at the app process boundary, such as a call into `system_server`._

## More details

* [Ravenwood for Test Authors](test-authors.md)

* [Ravenwood for API Maintainers](api-maintainers.md)

# Ravenwood for API Maintainers

By default, Android APIs aren’t opted-in to Ravenwood, and they default to throwing when called under the Ravenwood environment.

To opt-in to supporting an API under Ravenwood, you can use the inline annotations documented below to customize your API behavior when running under Ravenwood.  Because these annotations are inline in the relevant platform source code, they serve as valuable reminders to future API maintainers of Ravenwood support expectations.

> **Note:** to ensure that API teams are well-supported during early Ravenwood onboarding, the Ravenwood team is manually maintaining an allow-list of classes that are able to use Ravenwood annotations.  Please reach out to ravenwood@ so we can offer design advice and allow-list your APIs.

These Ravenwood-specific annotations have no bearing on the status of an API being public, `@SystemApi`, `@TestApi`, `@hide`, etc.  Ravenwood annotations are an orthogonal concept that are only consumed by the internal `hoststubgen` tool during a post-processing step that generates the Ravenwood runtime environment.  Teams that own APIs can continue to refactor opted-in `@hide` implementation details, as long as the test-visible behavior continues passing.

As described in our Guiding Principles, when a team opts-in an API, we’re requiring that they bring along “bivalent” tests (such as the relevant CTS) to validate that Ravenwood behaves just like a physical device.  At the moment this means adding the bivalent tests to relevant `TEST_MAPPING` files to ensure they remain consistently passing over time.  These bivalent tests are important because they progressively provide the foundation on which higher-level unit tests place their trust.

## Opt-in to supporting a single method while other methods remained opt-out

```

@RavenwoodKeepPartialClass

public class MyManager {

    @RavenwoodKeep

    public static String modeToString(int mode) {

        // This method implementation runs as-is on both devices and Ravenwood

    }

    public static void doComplex() {

        // This method implementation runs as-is on devices, but because there

        // is no method-level annotation, and the class-level default is

        // “keep partial”, this method is not supported under Ravenwood and

        // will throw

    }

}

```

## Opt-in an entire class with opt-out of specific methods

```

@RavenwoodKeepWholeClass

public class MyStruct {

    public void doSimple() {

        // This method implementation runs as-is on both devices and Ravenwood,

        // implicitly inheriting the class-level annotation

    }

    @RavenwoodThrow

    public void doComplex() {

        // This method implementation runs as-is on devices, but the

        // method-level annotation overrides the class-level annotation, so

        // this method is not supported under Ravenwood and will throw

    }

}

```

## Replace a complex method when under Ravenwood

```

@RavenwoodKeepWholeClass

public class MyStruct {

    @RavenwoodReplace

    public void doComplex() {

        // This method implementation runs as-is on devices, but the

        // implementation is replaced/substituted by the

        // doComplex$ravenwood() method implementation under Ravenwood

    }

    public void doComplex$ravenwood() {

        // This method implementation only runs under Ravenwood

    }

}

```

## General strategies for side-stepping tricky dependencies

The “replace” strategy described above is quite powerful, and can be used in creative ways to sidestep tricky underlying dependencies that aren’t ready yet.

For example, consider a constructor or static initializer that relies on unsupported functionality from another team.  By factoring the unsupported logic into a dedicated method, that method can then be replaced under Ravenwood to offer baseline functionality.

# Ravenwood for Test Authors

The Ravenwood testing environment runs inside a single Java process on the host side, and provides a limited yet growing set of Android API functionality.

Ravenwood explicitly does not support “large” integration tests that expect a fully booted Android OS.  Instead, it’s more suited for “small” and “medium” tests where your code-under-test has been factored to remove dependencies on a fully booted device.

When writing tests under Ravenwood, all Android API symbols associated with your declared `sdk_version` are available to link against using, but unsupported APIs will throw an exception.  This design choice enables mocking of unsupported APIs, and supports sharing of test code to build “bivalent” test suites that run against either Ravenwood or a traditional device.

## Typical test structure

Below are the typical steps needed to add a straightforward “small” unit test:

* Define an `android_ravenwood_test` rule in your `Android.bp` file:

```

android_ravenwood_test {

    name: "MyTestsRavenwood",

    static_libs: [

        "androidx.annotation_annotation",

        "androidx.test.rules",

    ],

    srcs: [

        "src/com/example/MyCode.java",

        "tests/src/com/example/MyCodeTest.java",

    ],

    sdk_version: "test_current",

    auto_gen_config: true,

}

```

* Write your unit test just like you would for an Android device:

```

@RunWith(AndroidJUnit4.class)

public class MyCodeTest {

    @Test

    public void testSimple() {

        // ...

    }

}

```

* APIs available under Ravenwood are stateless by default.  If your test requires explicit states (such as defining the UID you’re running under, or requiring a main `Looper` thread), add a `RavenwoodRule` to declare that:

```

@RunWith(AndroidJUnit4.class)

public class MyCodeTest {

    @Rule

    public final RavenwoodRule mRavenwood = new RavenwoodRule.Builder()

            .setProcessApp()

            .setProvideMainThread(true)

            .build();

```

Once you’ve defined your test, you can use typical commands to execute it locally:

```

$ atest MyTestsRavenwood

```

> **Note:** There's a known bug where `atest` currently requires a connected device to run Ravenwood tests, but that device isn't used for testing.

You can also run your new tests automatically via `TEST_MAPPING` rules like this:

```

{

  "ravenwood-presubmit": [

    {

      "name": "MyTestsRavenwood",

      "host": true

    }

  ]

}

```

## Strategies for migration/bivalent tests

Ravenwood aims to support tests that are written in a “bivalent” way, where the same test code can run on both a real Android device and under a Ravenwood environment.

In situations where a test method depends on API functionality not yet available under Ravenwood, we provide an annotation to quietly “ignore” that test under Ravenwood, while continuing to validate that test on real devices.  Please note that your test must declare a `RavenwoodRule` for the annotation to take effect.

Test authors are encouraged to provide a `blockedBy` or `reason` argument to help future maintainers understand why a test is being ignored, and under what conditions it might be supported in the future.

```

@RunWith(AndroidJUnit4.class)

public class MyCodeTest {

    @Rule

    public final RavenwoodRule mRavenwood = new RavenwoodRule();

    @Test

    public void testSimple() {

        // Simple test that runs on both devices and Ravenwood

    }

    @Test

    @IgnoreUnderRavenwood(blockedBy = PackageManager.class)

    public void testComplex() {

        // Complex test that runs on devices, but is ignored under Ravenwood

    }

}

```

## Strategies for unsupported APIs

As you write tests against Ravenwood, you’ll likely discover API dependencies that aren’t supported yet.  Here’s a few strategies that can help you make progress:

* Your code-under-test may benefit from subtle dependency refactoring to reduce coupling.  (For example, providing a specific `File` argument instead of deriving it internally from a `Context`.)

* Although mocking code that your team doesn’t own is a generally discouraged testing practice, it can be a valuable pressure relief valve when a dependency isn’t yet supported.

## Strategies for debugging test development

When writing tests you may encounter odd or hard to debug behaviors.  One good place to start is at the beginning of the logs stored by atest:

```

$ atest MyTestsRavenwood

...

Test Logs have saved in /tmp/atest_result/20231128_094010_0e90t8v8/log

Run 'atest --history' to review test result history.

```

The most useful logs are in the `isolated-java-logs` text file, which can typically be tab-completed by copy-pasting the logs path mentioned in the atest output:

```

$ less /tmp/atest_result/20231128_133105_h9al__79/log/i*/i*/isolated-java-logs*

```

Here are some common known issues and recommended workarounds:

* Some code may unconditionally interact with unsupported APIs, such as via static initializers.  One strategy is to shift the logic into `@Before` methods and make it conditional by testing `RavenwoodRule.isUnderRavenwood()`.

* Some code may reference API symbols not yet present in the Ravenwood runtime, such as ART or ICU internals, or APIs from Mainline modules.  One strategy is to refactor to avoid these internal dependencies, but Ravenwood aims to better support them soon.

    * This may also manifest as very odd behavior, such as test not being executed at all, tracked by bug #312517322

    * This may also manifest as an obscure Mockito error claiming “Mockito can only mock non-private & non-final classes”