Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 08bd163e authored by Eino-Ville Talvala's avatar Eino-Ville Talvala
Browse files

Camera: Documentation updates for calibration and distortion correction 

- Add more notes on coordinate axes
- Add more text on metadata when distortion correction is active
- Note that poseTranslation needs to be negated in many use cases
- Fix coordinate system references for OIS reporting, add more information
- Note that pixel centers at half-integers for the camera API metadata
  such as lens intrinsics

Bug: 79371566
Bug: 74434422
Bug: 109742048
Bug: 109834325
Bug: 109817371
Bug: 112107924
Test: Manual reading of added text
Change-Id: I450e80b79ef66ce8d82a4dee835db6abd1e598a3
parent ae3f6561

core/java/android/hardware/camera2/CameraCharacteristics.java

+18 −6
Original line number Diff line number Diff line
@@ -1249,7 +1249,9 @@ public final class CameraCharacteristics extends CameraMetadata<CameraCharacteri 
     * <p>If this device is the largest or only camera device with a given facing, then this

     * position will be <code>(0, 0, 0)</code>; a camera device with a lens optical center located 3 cm

     * from the main sensor along the +X axis (to the right from the user's perspective) will

     * report <code>(0.03, 0, 0)</code>.</p>

     * report <code>(0.03, 0, 0)</code>.  Note that this means that, for many computer vision

     * applications, the position needs to be negated to convert it to a translation from the

     * camera to the origin.</p>

     * <p>To transform a pixel coordinates between two cameras facing the same direction, first

     * the source camera {@link CameraCharacteristics#LENS_DISTORTION android.lens.distortion} must be corrected for.  Then the source

     * camera {@link CameraCharacteristics#LENS_INTRINSIC_CALIBRATION android.lens.intrinsicCalibration} needs to be applied, followed by the
@@ -1261,7 +1263,8 @@ public final class CameraCharacteristics extends CameraMetadata<CameraCharacteri 
     * <p>To compare this against a real image from the destination camera, the destination camera

     * image then needs to be corrected for radial distortion before comparison or sampling.</p>

     * <p>When {@link CameraCharacteristics#LENS_POSE_REFERENCE android.lens.poseReference} is GYROSCOPE, then this position is relative to

     * the center of the primary gyroscope on the device.</p>

     * the center of the primary gyroscope on the device. The axis definitions are the same as

     * with PRIMARY_CAMERA.</p>

     * <p><b>Units</b>: Meters</p>

     * <p><b>Optional</b> - This value may be {@code null} on some devices.</p>

     *
@@ -1293,13 +1296,15 @@ public final class CameraCharacteristics extends CameraMetadata<CameraCharacteri 
     * </code></pre>

     * <p>which can then be combined with the camera pose rotation

     * <code>R</code> and translation <code>t</code> ({@link CameraCharacteristics#LENS_POSE_ROTATION android.lens.poseRotation} and

     * {@link CameraCharacteristics#LENS_POSE_TRANSLATION android.lens.poseTranslation}, respective) to calculate the

     * {@link CameraCharacteristics#LENS_POSE_TRANSLATION android.lens.poseTranslation}, respectively) to calculate the

     * complete transform from world coordinates to pixel

     * coordinates:</p>

     * <pre><code>P = [ K 0   * [ R t

     * <pre><code>P = [ K 0   * [ R -Rt

     *      0 1 ]      0 1 ]

     * </code></pre>

     * <p>and with <code>p_w</code> being a point in the world coordinate system

     * <p>(Note the negation of poseTranslation when mapping from camera

     * to world coordinates, and multiplication by the rotation).</p>

     * <p>With <code>p_w</code> being a point in the world coordinate system

     * and <code>p_s</code> being a point in the camera active pixel array

     * coordinate system, and with the mapping including the

     * homogeneous division by z:</p>
@@ -1321,6 +1326,13 @@ public final class CameraCharacteristics extends CameraMetadata<CameraCharacteri 
     * activeArraySize rectangle), to determine the final pixel

     * coordinate of the world point for processed (non-RAW)

     * output buffers.</p>

     * <p>For camera devices, the center of pixel <code>(x,y)</code> is located at

     * coordinate <code>(x + 0.5, y + 0.5)</code>.  So on a device with a

     * precorrection active array of size <code>(10,10)</code>, the valid pixel

     * indices go from <code>(0,0)-(9,9)</code>, and an perfectly-built camera would

     * have an optical center at the exact center of the pixel grid, at

     * coordinates <code>(5.0, 5.0)</code>, which is the top-left corner of pixel

     * <code>(5,5)</code>.</p>

     * <p><b>Units</b>:

     * Pixels in the

     * {@link CameraCharacteristics#SENSOR_INFO_PRE_CORRECTION_ACTIVE_ARRAY_SIZE android.sensor.info.preCorrectionActiveArraySize}

core/java/android/hardware/camera2/CaptureRequest.java

+31 −6
Original line number Diff line number Diff line
@@ -2522,7 +2522,7 @@ public final class CaptureRequest extends CameraMetadata<CaptureRequest.Key<?>> 
     * outputs will crop horizontally (pillarbox), and 16:9

     * streams will match exactly. These additional crops will

     * be centered within the crop region.</p>

     * <p>If the coordinate system is android.sensor.info.activeArraysSize, the width and height

     * <p>If the coordinate system is {@link CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE android.sensor.info.activeArraySize}, the width and height

     * of the crop region cannot be set to be smaller than

     * <code>floor( activeArraySize.width / {@link CameraCharacteristics#SCALER_AVAILABLE_MAX_DIGITAL_ZOOM android.scaler.availableMaxDigitalZoom} )</code> and

     * <code>floor( activeArraySize.height / {@link CameraCharacteristics#SCALER_AVAILABLE_MAX_DIGITAL_ZOOM android.scaler.availableMaxDigitalZoom} )</code>, respectively.</p>
@@ -2863,8 +2863,14 @@ public final class CaptureRequest extends CameraMetadata<CaptureRequest.Key<?>> 
            new Key<Integer>("android.statistics.lensShadingMapMode", int.class);



    /**

     * <p>A control for selecting whether OIS position information is included in output

     * result metadata.</p>

     * <p>A control for selecting whether optical stabilization (OIS) position

     * information is included in output result metadata.</p>

     * <p>Since optical image stabilization generally involves motion much faster than the duration

     * of individualq image exposure, multiple OIS samples can be included for a single capture

     * result. For example, if the OIS reporting operates at 200 Hz, a typical camera operating

     * at 30fps may have 6-7 OIS samples per capture result. This information can be combined

     * with the rolling shutter skew to account for lens motion during image exposure in

     * post-processing algorithms.</p>

     * <p><b>Possible values:</b>

     * <ul>

     *   <li>{@link #STATISTICS_OIS_DATA_MODE_OFF OFF}</li>
@@ -3264,14 +3270,28 @@ public final class CaptureRequest extends CameraMetadata<CaptureRequest.Key<?>> 
     * any correction at all would slow down capture rate.  Every output stream will have a

     * similar amount of enhancement applied.</p>

     * <p>The correction only applies to processed outputs such as YUV, JPEG, or DEPTH16; it is not

     * applied to any RAW output. Metadata coordinates such as face rectangles or metering

     * regions are also not affected by correction.</p>

     * applied to any RAW output.</p>

     * <p>This control will be on by default on devices that support this control. Applications

     * disabling distortion correction need to pay extra attention with the coordinate system of

     * metering regions, crop region, and face rectangles. When distortion correction is OFF,

     * metadata coordinates follow the coordinate system of

     * {@link CameraCharacteristics#SENSOR_INFO_PRE_CORRECTION_ACTIVE_ARRAY_SIZE android.sensor.info.preCorrectionActiveArraySize}. When distortion is not OFF, metadata

     * coordinates follow the coordinate system of {@link CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE android.sensor.info.activeArraySize}.</p>

     * coordinates follow the coordinate system of {@link CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE android.sensor.info.activeArraySize}.  The

     * camera device will map these metadata fields to match the corrected image produced by the

     * camera device, for both capture requests and results.  However, this mapping is not very

     * precise, since rectangles do not generally map to rectangles when corrected.  Only linear

     * scaling between the active array and precorrection active array coordinates is

     * performed. Applications that require precise correction of metadata need to undo that

     * linear scaling, and apply a more complete correction that takes into the account the app's

     * own requirements.</p>

     * <p>The full list of metadata that is affected in this way by distortion correction is:</p>

     * <ul>

     * <li>{@link CaptureRequest#CONTROL_AF_REGIONS android.control.afRegions}</li>

     * <li>{@link CaptureRequest#CONTROL_AE_REGIONS android.control.aeRegions}</li>

     * <li>{@link CaptureRequest#CONTROL_AWB_REGIONS android.control.awbRegions}</li>

     * <li>{@link CaptureRequest#SCALER_CROP_REGION android.scaler.cropRegion}</li>

     * <li>{@link CaptureResult#STATISTICS_FACES android.statistics.faces}</li>

     * </ul>

     * <p><b>Possible values:</b>

     * <ul>

     *   <li>{@link #DISTORTION_CORRECTION_MODE_OFF OFF}</li>
@@ -3282,10 +3302,15 @@ public final class CaptureRequest extends CameraMetadata<CaptureRequest.Key<?>> 
     * {@link CameraCharacteristics#DISTORTION_CORRECTION_AVAILABLE_MODES android.distortionCorrection.availableModes}</p>

     * <p><b>Optional</b> - This value may be {@code null} on some devices.</p>

     *

     * @see CaptureRequest#CONTROL_AE_REGIONS

     * @see CaptureRequest#CONTROL_AF_REGIONS

     * @see CaptureRequest#CONTROL_AWB_REGIONS

     * @see CameraCharacteristics#DISTORTION_CORRECTION_AVAILABLE_MODES

     * @see CameraCharacteristics#LENS_DISTORTION

     * @see CaptureRequest#SCALER_CROP_REGION

     * @see CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE

     * @see CameraCharacteristics#SENSOR_INFO_PRE_CORRECTION_ACTIVE_ARRAY_SIZE

     * @see CaptureResult#STATISTICS_FACES

     * @see #DISTORTION_CORRECTION_MODE_OFF

     * @see #DISTORTION_CORRECTION_MODE_FAST

     * @see #DISTORTION_CORRECTION_MODE_HIGH_QUALITY

core/java/android/hardware/camera2/CaptureResult.java

+76 −25
Original line number Diff line number Diff line
@@ -2852,7 +2852,9 @@ public class CaptureResult extends CameraMetadata<CaptureResult.Key<?>> { 
     * <p>If this device is the largest or only camera device with a given facing, then this

     * position will be <code>(0, 0, 0)</code>; a camera device with a lens optical center located 3 cm

     * from the main sensor along the +X axis (to the right from the user's perspective) will

     * report <code>(0.03, 0, 0)</code>.</p>

     * report <code>(0.03, 0, 0)</code>.  Note that this means that, for many computer vision

     * applications, the position needs to be negated to convert it to a translation from the

     * camera to the origin.</p>

     * <p>To transform a pixel coordinates between two cameras facing the same direction, first

     * the source camera {@link CameraCharacteristics#LENS_DISTORTION android.lens.distortion} must be corrected for.  Then the source

     * camera {@link CameraCharacteristics#LENS_INTRINSIC_CALIBRATION android.lens.intrinsicCalibration} needs to be applied, followed by the
@@ -2864,7 +2866,8 @@ public class CaptureResult extends CameraMetadata<CaptureResult.Key<?>> { 
     * <p>To compare this against a real image from the destination camera, the destination camera

     * image then needs to be corrected for radial distortion before comparison or sampling.</p>

     * <p>When {@link CameraCharacteristics#LENS_POSE_REFERENCE android.lens.poseReference} is GYROSCOPE, then this position is relative to

     * the center of the primary gyroscope on the device.</p>

     * the center of the primary gyroscope on the device. The axis definitions are the same as

     * with PRIMARY_CAMERA.</p>

     * <p><b>Units</b>: Meters</p>

     * <p><b>Optional</b> - This value may be {@code null} on some devices.</p>

     *
@@ -2896,13 +2899,15 @@ public class CaptureResult extends CameraMetadata<CaptureResult.Key<?>> { 
     * </code></pre>

     * <p>which can then be combined with the camera pose rotation

     * <code>R</code> and translation <code>t</code> ({@link CameraCharacteristics#LENS_POSE_ROTATION android.lens.poseRotation} and

     * {@link CameraCharacteristics#LENS_POSE_TRANSLATION android.lens.poseTranslation}, respective) to calculate the

     * {@link CameraCharacteristics#LENS_POSE_TRANSLATION android.lens.poseTranslation}, respectively) to calculate the

     * complete transform from world coordinates to pixel

     * coordinates:</p>

     * <pre><code>P = [ K 0   * [ R t

     * <pre><code>P = [ K 0   * [ R -Rt

     *      0 1 ]      0 1 ]

     * </code></pre>

     * <p>and with <code>p_w</code> being a point in the world coordinate system

     * <p>(Note the negation of poseTranslation when mapping from camera

     * to world coordinates, and multiplication by the rotation).</p>

     * <p>With <code>p_w</code> being a point in the world coordinate system

     * and <code>p_s</code> being a point in the camera active pixel array

     * coordinate system, and with the mapping including the

     * homogeneous division by z:</p>
@@ -2924,6 +2929,13 @@ public class CaptureResult extends CameraMetadata<CaptureResult.Key<?>> { 
     * activeArraySize rectangle), to determine the final pixel

     * coordinate of the world point for processed (non-RAW)

     * output buffers.</p>

     * <p>For camera devices, the center of pixel <code>(x,y)</code> is located at

     * coordinate <code>(x + 0.5, y + 0.5)</code>.  So on a device with a

     * precorrection active array of size <code>(10,10)</code>, the valid pixel

     * indices go from <code>(0,0)-(9,9)</code>, and an perfectly-built camera would

     * have an optical center at the exact center of the pixel grid, at

     * coordinates <code>(5.0, 5.0)</code>, which is the top-left corner of pixel

     * <code>(5,5)</code>.</p>

     * <p><b>Units</b>:

     * Pixels in the

     * {@link CameraCharacteristics#SENSOR_INFO_PRE_CORRECTION_ACTIVE_ARRAY_SIZE android.sensor.info.preCorrectionActiveArraySize}
@@ -3188,7 +3200,7 @@ public class CaptureResult extends CameraMetadata<CaptureResult.Key<?>> { 
     * outputs will crop horizontally (pillarbox), and 16:9

     * streams will match exactly. These additional crops will

     * be centered within the crop region.</p>

     * <p>If the coordinate system is android.sensor.info.activeArraysSize, the width and height

     * <p>If the coordinate system is {@link CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE android.sensor.info.activeArraySize}, the width and height

     * of the crop region cannot be set to be smaller than

     * <code>floor( activeArraySize.width / {@link CameraCharacteristics#SCALER_AVAILABLE_MAX_DIGITAL_ZOOM android.scaler.availableMaxDigitalZoom} )</code> and

     * <code>floor( activeArraySize.height / {@link CameraCharacteristics#SCALER_AVAILABLE_MAX_DIGITAL_ZOOM android.scaler.availableMaxDigitalZoom} )</code>, respectively.</p>
@@ -4077,8 +4089,14 @@ public class CaptureResult extends CameraMetadata<CaptureResult.Key<?>> { 
            new Key<Integer>("android.statistics.lensShadingMapMode", int.class);



    /**

     * <p>A control for selecting whether OIS position information is included in output

     * result metadata.</p>

     * <p>A control for selecting whether optical stabilization (OIS) position

     * information is included in output result metadata.</p>

     * <p>Since optical image stabilization generally involves motion much faster than the duration

     * of individualq image exposure, multiple OIS samples can be included for a single capture

     * result. For example, if the OIS reporting operates at 200 Hz, a typical camera operating

     * at 30fps may have 6-7 OIS samples per capture result. This information can be combined

     * with the rolling shutter skew to account for lens motion during image exposure in

     * post-processing algorithms.</p>

     * <p><b>Possible values:</b>

     * <ul>

     *   <li>{@link #STATISTICS_OIS_DATA_MODE_OFF OFF}</li>
@@ -4112,11 +4130,15 @@ public class CaptureResult extends CameraMetadata<CaptureResult.Key<?>> { 
    /**

     * <p>An array of shifts of OIS samples, in x direction.</p>

     * <p>The array contains the amount of shifts in x direction, in pixels, based on OIS samples.

     * A positive value is a shift from left to right in active array coordinate system. For

     * example, if the optical center is (1000, 500) in active array coordinates, a shift of

     * (3, 0) puts the new optical center at (1003, 500).</p>

     * A positive value is a shift from left to right in the pre-correction active array

     * coordinate system. For example, if the optical center is (1000, 500) in pre-correction

     * active array coordinates, a shift of (3, 0) puts the new optical center at (1003, 500).</p>

     * <p>The number of shifts must match the number of timestamps in

     * android.statistics.oisTimestamps.</p>

     * <p>The OIS samples are not affected by whether lens distortion correction is enabled (on

     * supporting devices). They are always reported in pre-correction active array coordinates,

     * since the scaling of OIS shifts would depend on the specific spot on the sensor the shift

     * is needed.</p>

     * <p><b>Units</b>: Pixels in active array.</p>

     * <p><b>Optional</b> - This value may be {@code null} on some devices.</p>

     * @hide
@@ -4127,11 +4149,15 @@ public class CaptureResult extends CameraMetadata<CaptureResult.Key<?>> { 
    /**

     * <p>An array of shifts of OIS samples, in y direction.</p>

     * <p>The array contains the amount of shifts in y direction, in pixels, based on OIS samples.

     * A positive value is a shift from top to bottom in active array coordinate system. For

     * example, if the optical center is (1000, 500) in active array coordinates, a shift of

     * (0, 5) puts the new optical center at (1000, 505).</p>

     * A positive value is a shift from top to bottom in pre-correction active array coordinate

     * system. For example, if the optical center is (1000, 500) in active array coordinates, a

     * shift of (0, 5) puts the new optical center at (1000, 505).</p>

     * <p>The number of shifts must match the number of timestamps in

     * android.statistics.oisTimestamps.</p>

     * <p>The OIS samples are not affected by whether lens distortion correction is enabled (on

     * supporting devices). They are always reported in pre-correction active array coordinates,

     * since the scaling of OIS shifts would depend on the specific spot on the sensor the shift

     * is needed.</p>

     * <p><b>Units</b>: Pixels in active array.</p>

     * <p><b>Optional</b> - This value may be {@code null} on some devices.</p>

     * @hide
@@ -4140,15 +4166,21 @@ public class CaptureResult extends CameraMetadata<CaptureResult.Key<?>> { 
            new Key<float[]>("android.statistics.oisYShifts", float[].class);



    /**

     * <p>An array of OIS samples.</p>

     * <p>An array of optical stabilization (OIS) position samples.</p>

     * <p>Each OIS sample contains the timestamp and the amount of shifts in x and y direction,

     * in pixels, of the OIS sample.</p>

     * <p>A positive value for a shift in x direction is a shift from left to right in active array

     * coordinate system. For example, if the optical center is (1000, 500) in active array

     * coordinates, a shift of (3, 0) puts the new optical center at (1003, 500).</p>

     * <p>A positive value for a shift in y direction is a shift from top to bottom in active array

     * coordinate system. For example, if the optical center is (1000, 500) in active array

     * coordinates, a shift of (0, 5) puts the new optical center at (1000, 505).</p>

     * <p>A positive value for a shift in x direction is a shift from left to right in the

     * pre-correction active array coordinate system. For example, if the optical center is

     * (1000, 500) in pre-correction active array coordinates, a shift of (3, 0) puts the new

     * optical center at (1003, 500).</p>

     * <p>A positive value for a shift in y direction is a shift from top to bottom in

     * pre-correction active array coordinate system. For example, if the optical center is

     * (1000, 500) in active array coordinates, a shift of (0, 5) puts the new optical center at

     * (1000, 505).</p>

     * <p>The OIS samples are not affected by whether lens distortion correction is enabled (on

     * supporting devices). They are always reported in pre-correction active array coordinates,

     * since the scaling of OIS shifts would depend on the specific spot on the sensor the shift

     * is needed.</p>

     * <p><b>Optional</b> - This value may be {@code null} on some devices.</p>

     */

    @PublicKey
@@ -4578,14 +4610,28 @@ public class CaptureResult extends CameraMetadata<CaptureResult.Key<?>> { 
     * any correction at all would slow down capture rate.  Every output stream will have a

     * similar amount of enhancement applied.</p>

     * <p>The correction only applies to processed outputs such as YUV, JPEG, or DEPTH16; it is not

     * applied to any RAW output. Metadata coordinates such as face rectangles or metering

     * regions are also not affected by correction.</p>

     * applied to any RAW output.</p>

     * <p>This control will be on by default on devices that support this control. Applications

     * disabling distortion correction need to pay extra attention with the coordinate system of

     * metering regions, crop region, and face rectangles. When distortion correction is OFF,

     * metadata coordinates follow the coordinate system of

     * {@link CameraCharacteristics#SENSOR_INFO_PRE_CORRECTION_ACTIVE_ARRAY_SIZE android.sensor.info.preCorrectionActiveArraySize}. When distortion is not OFF, metadata

     * coordinates follow the coordinate system of {@link CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE android.sensor.info.activeArraySize}.</p>

     * coordinates follow the coordinate system of {@link CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE android.sensor.info.activeArraySize}.  The

     * camera device will map these metadata fields to match the corrected image produced by the

     * camera device, for both capture requests and results.  However, this mapping is not very

     * precise, since rectangles do not generally map to rectangles when corrected.  Only linear

     * scaling between the active array and precorrection active array coordinates is

     * performed. Applications that require precise correction of metadata need to undo that

     * linear scaling, and apply a more complete correction that takes into the account the app's

     * own requirements.</p>

     * <p>The full list of metadata that is affected in this way by distortion correction is:</p>

     * <ul>

     * <li>{@link CaptureRequest#CONTROL_AF_REGIONS android.control.afRegions}</li>

     * <li>{@link CaptureRequest#CONTROL_AE_REGIONS android.control.aeRegions}</li>

     * <li>{@link CaptureRequest#CONTROL_AWB_REGIONS android.control.awbRegions}</li>

     * <li>{@link CaptureRequest#SCALER_CROP_REGION android.scaler.cropRegion}</li>

     * <li>{@link CaptureResult#STATISTICS_FACES android.statistics.faces}</li>

     * </ul>

     * <p><b>Possible values:</b>

     * <ul>

     *   <li>{@link #DISTORTION_CORRECTION_MODE_OFF OFF}</li>
@@ -4596,10 +4642,15 @@ public class CaptureResult extends CameraMetadata<CaptureResult.Key<?>> { 
     * {@link CameraCharacteristics#DISTORTION_CORRECTION_AVAILABLE_MODES android.distortionCorrection.availableModes}</p>

     * <p><b>Optional</b> - This value may be {@code null} on some devices.</p>

     *

     * @see CaptureRequest#CONTROL_AE_REGIONS

     * @see CaptureRequest#CONTROL_AF_REGIONS

     * @see CaptureRequest#CONTROL_AWB_REGIONS

     * @see CameraCharacteristics#DISTORTION_CORRECTION_AVAILABLE_MODES

     * @see CameraCharacteristics#LENS_DISTORTION

     * @see CaptureRequest#SCALER_CROP_REGION

     * @see CameraCharacteristics#SENSOR_INFO_ACTIVE_ARRAY_SIZE

     * @see CameraCharacteristics#SENSOR_INFO_PRE_CORRECTION_ACTIVE_ARRAY_SIZE

     * @see CaptureResult#STATISTICS_FACES

     * @see #DISTORTION_CORRECTION_MODE_OFF

     * @see #DISTORTION_CORRECTION_MODE_FAST

     * @see #DISTORTION_CORRECTION_MODE_HIGH_QUALITY