@@ -4,25 +4,4 @@ Ravenwood is an officially-supported lightweight unit testing environment for An
Ravenwood’s focus on Android platform use-cases, improved maintainability, and device consistency distinguishes it from Robolectric, which remains a popular choice for app testing.
## Background
Executing tests on a typical Android device has substantial overhead, such as flashing the build, waiting for the boot to complete, and retrying tests that fail due to general flakiness.
In contrast, defining a lightweight unit testing environment mitigates these issues by running directly from build artifacts (no flashing required), runs immediately (no booting required), and runs in an isolated environment (less flakiness).
## Guiding principles
Here’s a summary of the guiding principles for Ravenwood, aimed at addressing Robolectric design concerns and better supporting Android platform developers:
***API support for Ravenwood is opt-in.** Teams that own APIs decide exactly what, and how, they support their API functionality being available to tests. When an API hasn’t opted-in, the API signatures remain available for tests to compile against and/or mock, but they throw when called under a Ravenwood environment.
* _Contrasted with Robolectric which attempts to run API implementations as-is, causing maintenance pains as teams maintain or redesign their API internals._
***API support and customizations for Ravenwood appear directly inline with relevant code.** This improves maintenance of APIs by providing awareness of what code runs under Ravenwood, including the ability to replace code at a per-method level when Ravenwood-specific customization is needed.
* _Contrasted with Robolectric which maintains customized behavior in separate “Shadow” classes that are difficult for maintainers to be aware of._
***APIs supported under Ravenwood are tested to remain consistent with physical devices.** As teams progressively opt-in supporting APIs under Ravenwood, we’re requiring they bring along “bivalent” tests (such as the relevant CTS) to validate that Ravenwood behaves just like a physical device.
* _Contrasted with Robolectric, which has limited (and forked) testing of their environment, increasing their risk of accidental divergence over time and misleading “passing” signals._
***Ravenwood aims to support more “real” code.** As API owners progressively opt-in their code, they have the freedom to provide either a limited “fake” that is a faithful emulation of how a device behaves, or they can bring more “real” code that runs on physical devices.
* _Contrasted with Robolectric, where support for “real” code ends at the app process boundary, such as a call into `system_server`._
## More details
*[Ravenwood for Test Authors](test-authors.md)
*[Ravenwood for API Maintainers](api-maintainers.md)
By default, Android APIs aren’t opted-in to Ravenwood, and they default to throwing when called under the Ravenwood environment.
To opt-in to supporting an API under Ravenwood, you can use the inline annotations documented below to customize your API behavior when running under Ravenwood. Because these annotations are inline in the relevant platform source code, they serve as valuable reminders to future API maintainers of Ravenwood support expectations.
> **Note:** to ensure that API teams are well-supported during early Ravenwood onboarding, the Ravenwood team is manually maintaining an allow-list of classes that are able to use Ravenwood annotations. Please reach out to ravenwood@ so we can offer design advice and allow-list your APIs.
These Ravenwood-specific annotations have no bearing on the status of an API being public, `@SystemApi`, `@TestApi`, `@hide`, etc. Ravenwood annotations are an orthogonal concept that are only consumed by the internal `hoststubgen` tool during a post-processing step that generates the Ravenwood runtime environment. Teams that own APIs can continue to refactor opted-in `@hide` implementation details, as long as the test-visible behavior continues passing.
As described in our Guiding Principles, when a team opts-in an API, we’re requiring that they bring along “bivalent” tests (such as the relevant CTS) to validate that Ravenwood behaves just like a physical device. At the moment this means adding the bivalent tests to relevant `TEST_MAPPING` files to ensure they remain consistently passing over time. These bivalent tests are important because they progressively provide the foundation on which higher-level unit tests place their trust.
## Opt-in to supporting a single method while other methods remained opt-out
```
@RavenwoodKeepPartialClass
public class MyManager {
@RavenwoodKeep
public static String modeToString(int mode) {
// This method implementation runs as-is on both devices and Ravenwood
}
public static void doComplex() {
// This method implementation runs as-is on devices, but because there
// is no method-level annotation, and the class-level default is
// “keep partial”, this method is not supported under Ravenwood and
// will throw
}
}
```
## Opt-in an entire class with opt-out of specific methods
```
@RavenwoodKeepWholeClass
public class MyStruct {
public void doSimple() {
// This method implementation runs as-is on both devices and Ravenwood,
// implicitly inheriting the class-level annotation
}
@RavenwoodThrow
public void doComplex() {
// This method implementation runs as-is on devices, but the
// method-level annotation overrides the class-level annotation, so
// this method is not supported under Ravenwood and will throw
}
}
```
## Replace a complex method when under Ravenwood
```
@RavenwoodKeepWholeClass
public class MyStruct {
@RavenwoodReplace
public void doComplex() {
// This method implementation runs as-is on devices, but the
// implementation is replaced/substituted by the
// doComplex$ravenwood() method implementation under Ravenwood
}
private void doComplex$ravenwood() {
// This method implementation only runs under Ravenwood.
// The visibility of this replacement method does not need to match
// the original method, so it's recommmended to always use
// private methods so that these methods won't be accidentally used
// by unexpected users.
}
}
```
## Redirect a complex method to a separate class when under Ravenwood
// This method implementation runs as-is on devices, but calls to this
// method is redirected to MyComplexClass_ravenwood#doComplex()
// under Ravenwood.
}
@RavenwoodRedirect
public static void staticDoComplex(int i, int j, int k) {
// This method implementation runs as-is on devices, but calls to this
// method is redirected to MyComplexClass_ravenwood#staticDoComplex()
// under Ravenwood.
}
/**
* This class is only available in the Ravenwood environment.
*/
class MyComplexClass_ravenwood {
public static void doComplex(MyComplexClass obj, int i, int j, int k) {
// Because the original method is a non-static method, the current
// object instance of the original method (the "this" reference)
// will be passed over as the first argument of the redirection method,
// with the remaining arguments to follow.
}
public static void staticDoComplex(int i, int j, int k) {
// Because the original method is a static method, there is no current
// object instance, so the parameter list of the redirection method
// is the same as the original method.
}
}
```
## General strategies for side-stepping tricky dependencies
The “replace” strategy described above is quite powerful, and can be used in creative ways to sidestep tricky underlying dependencies that aren’t ready yet.
For example, consider a constructor or static initializer that relies on unsupported functionality from another team. By factoring the unsupported logic into a dedicated method, that method can then be replaced under Ravenwood to offer baseline functionality.
## Strategies for JNI
At the moment, JNI support is considered "experimental". To ensure that teams are well-supported, for any JNI usage, please reach out to ravenwood@ so we can offer design advice and help you onboard native methods. If a native method can be trivially re-implemented in pure-Java, using the replacement or redirection mechanisms described above is also a viable option.
The main caveat regarding JNI support on Ravenwood is due to how native code is built for Ravenwood. Unlike Java/Kotlin code where Ravenwood directly uses the same intermediate artifacts when building the real target for Android, Ravenwood uses the "host" variant of native libraries. The "host" variant of native libraries usually either don't exist, are extremely limited, or behave drastically different compared to the Android variant.
The Ravenwood testing environment runs inside a single Java process on the host side, and provides a limited yet growing set of Android API functionality.
Ravenwood explicitly does not support “large” integration tests that expect a fully booted Android OS. Instead, it’s more suited for “small” and “medium” tests where your code-under-test has been factored to remove dependencies on a fully booted device.
When writing tests under Ravenwood, all Android API symbols associated with your declared `sdk_version` are available to link against using, but unsupported APIs will throw an exception. This design choice enables mocking of unsupported APIs, and supports sharing of test code to build “bivalent” test suites that run against either Ravenwood or a traditional device.
Once you’ve defined your test, you can use typical commands to execute it locally:
```
$ atest MyTestsRavenwood
```
You can also run your new tests automatically via `TEST_MAPPING` rules like this:
```
{
"ravenwood-presubmit": [
{
"name": "MyTestsRavenwood",
"host": true
}
]
}
```
## Using resources
At the moment, the `android_ravenwood_test` module type cannot directly build resources yet. In order to use resources in Ravenwood tests, you have to build the resource APK in a separate `android_app` module and "borrow" the resources and R classes:
```
android_app {
name: "MyTests-res",
resource_dirs: ["res"],
// This has to be set to false, or ".aapt.srcjar" will not be generated
use_resource_processor: false,
}
android_ravenwood_test {
name: "MyTestsRavenwood",
srcs: [
"src/**/*.java",
// ...
// Include R.java from the resource APK
":MyTests-res{.aapt.srcjar}",
],
// Set the resource APK
resource_apk: "MyTests-res",
// ...
}
```
## Strategies for migration/bivalent tests
Ravenwood aims to support tests that are written in a “bivalent” way, where the same test code can be dual-compiled to run on both a real Android device and under a Ravenwood environment.
In situations where a test method depends on API functionality not yet available under Ravenwood, we provide an annotation to quietly skip that test under Ravenwood, while continuing to validate that test on real devices. The annotation can be applied to either individual methods or to an entire test class.
Test authors are encouraged to provide a `blockedBy` or `reason` argument to help future maintainers understand why a test is being ignored, and under what conditions it might be supported in the future.
```
@RunWith(AndroidJUnit4.class)
public class MyCodeTest {
@Test
public void testSimple() {
// Simple test that runs on both devices and Ravenwood
// Complex test that runs on devices, but is disabled under Ravenwood
}
}
```
## Strategies for unsupported APIs
As you write tests against Ravenwood, you’ll likely discover API dependencies that aren’t supported yet. Here’s a few strategies that can help you make progress:
* Your code-under-test may benefit from subtle dependency refactoring to reduce coupling. (For example, providing a specific `File` argument instead of deriving paths internally from a `Context` or `Environment`.)
* One common use-case is providing a directory for your test to store temporary files, which can easily be accomplished using the `Files.createTempDirectory()` API which works on both physical devices and under Ravenwood:
* Although mocking code that your team doesn’t own is a generally discouraged testing practice, it can be a valuable pressure relief valve when a dependency isn’t yet supported.
## Strategies for debugging test development
When writing tests you may encounter odd or hard to debug behaviors. One good place to start is at the beginning of the logs stored by atest:
```
$ atest MyTestsRavenwood
...
Test Logs have saved in /tmp/atest_result/20231128_094010_0e90t8v8/log
Run 'atest --history' to review test result history.
```
The most useful logs are in the `isolated-java-logs` text file, which can typically be tab-completed by copy-pasting the logs path mentioned in the atest output:
```
$ less /tmp/atest_result/20231128_133105_h9al__79/log/i*/i*/isolated-java-logs*
```
Here are some common known issues and recommended workarounds:
* Some code may unconditionally interact with unsupported APIs, such as via static initializers. One strategy is to shift the logic into `@Before` methods and make it conditional by testing `RavenwoodRule.isOnRavenwood()`.
