Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit e57f9e4d authored by Alec Mouri's avatar Alec Mouri
Browse files

Rework how SF generates gainmapped screenshots 

Previously, SF generated an HDR screenshot by:

* Asking CompositionEngine to composite layers offscreen into SDR
* Asking CompositionEngine _again_ to composite layers offscreen into
  HDR
* Asking RenderEngine to build a gainmap
* Sending the SDR buffer and the gainmap to the client

But, this had some problems:
* This path bit-rot, so the HDR layer list was totally empty :)
* LayerFE now has an assumption that only one *thing* will be consuming
  the layer and emitting a release fence back. So, this fixing the above
  caused ghost images, and undoing that assumption just for screenshots
  is a bit unappetizing.
* If the above assumption *were* undone, then the release fence for the
  layer would be delayed due to rendering to an additional render
  target, which sucks.
* We talk to RenderEngine three times, which involves thread hops, which
  sucks.

So, we solve these problems by:

* Only asking CompositionEngine to composite layers into HDR
* Then, ask RenderEngine _once_ to tonemap the HDR buffer into SDR, and
  generate a gainmap.
* Send the SDR buffer and the gainmap to the client.

And now most things are solved: LayerFE is only used to render to one
render target, so it only gets one release fence back, as expected.
Moreover, compositing into HDR is expected to be very cheap, as the
local tonemapper only has to run when tonemapping from a very large HDR
range (like a PQ video) -- when the application is already adapting to
display conditions then we don't have to do anything. And, we only have
to talk to RenderEngine twice now.

We could optimize this further by moving the gainmap generation into
CompositionEngine, but that seems like adding complexity for the moment.

Note that in theory, screenshot quality for the SDR rendition could be a
little worse, since MouriMap introduces shadowing artifacts that may now
bleed across layers. But it doesn't seem to be noticeable in practice,
and it was a pre-existing issue for application-tonemapped content.

Also, we remove CaptureArgs::attachGainmap, since we don't need it at
all to safely trigger this path.

Bug: 329465218
Flag: com.android.graphics.surfaceflinger.flags.true_hdr_screenshots
Test: Trigger screencapture
Change-Id: I5a274886acb135b21096300a034aadca20eceeb5
parent 17e8bc7f

libs/gui/aidl/android/gui/CaptureArgs.aidl

+0 −5
Original line number Diff line number Diff line
@@ -69,10 +69,5 @@ parcelable CaptureArgs { 
    // exact colorspace is not an appropriate intermediate result.

    // Note that if the caller is requesting a specific dataspace, this hint does nothing.

    boolean hintForSeamlessTransition = false;



    // Allows the screenshot to attach a gainmap, which allows for a per-pixel

    // transformation of the screenshot to another luminance range, typically

    // mapping an SDR base image into HDR.

    boolean attachGainmap = false;

}

libs/renderengine/RenderEngine.cpp

+4 −5
Original line number Diff line number Diff line
@@ -107,16 +107,15 @@ ftl::Future<FenceResult> RenderEngine::drawLayers(const DisplaySettings& display 
    return resultFuture;

}



ftl::Future<FenceResult> RenderEngine::drawGainmap(

        const std::shared_ptr<ExternalTexture>& sdr, base::borrowed_fd&& sdrFence,

ftl::Future<FenceResult> RenderEngine::tonemapAndDrawGainmap(

        const std::shared_ptr<ExternalTexture>& hdr, base::borrowed_fd&& hdrFence,

        float hdrSdrRatio, ui::Dataspace dataspace,

        float hdrSdrRatio, ui::Dataspace dataspace, const std::shared_ptr<ExternalTexture>& sdr,

        const std::shared_ptr<ExternalTexture>& gainmap) {

    const auto resultPromise = std::make_shared<std::promise<FenceResult>>();

    std::future<FenceResult> resultFuture = resultPromise->get_future();

    updateProtectedContext({}, {sdr.get(), hdr.get(), gainmap.get()});

    drawGainmapInternal(std::move(resultPromise), sdr, std::move(sdrFence), hdr,

                        std::move(hdrFence), hdrSdrRatio, dataspace, gainmap);

    tonemapAndDrawGainmapInternal(std::move(resultPromise), hdr, std::move(hdrFence), hdrSdrRatio,

                                  dataspace, sdr, gainmap);

    return resultFuture;

}

libs/renderengine/include/renderengine/RenderEngine.h

+13 −9
Original line number Diff line number Diff line
@@ -217,11 +217,16 @@ public: 
                                                const std::shared_ptr<ExternalTexture>& buffer,

                                                base::unique_fd&& bufferFence);



    virtual ftl::Future<FenceResult> drawGainmap(const std::shared_ptr<ExternalTexture>& sdr,

                                                 base::borrowed_fd&& sdrFence,

                                                 const std::shared_ptr<ExternalTexture>& hdr,

                                                 base::borrowed_fd&& hdrFence, float hdrSdrRatio,

                                                 ui::Dataspace dataspace,

    // Tonemaps an HDR input image and draws an SDR rendition, plus a gainmap

    // describing how to recover the HDR image.

    //

    // The HDR input image is ALWAYS encoded with an sRGB transfer function and

    // is a floating point format. Accordingly, the hdrSdrRatio describes the

    // max luminance in the HDR input image above SDR, and the dataspace

    // describes the input primaries.

    virtual ftl::Future<FenceResult> tonemapAndDrawGainmap(

            const std::shared_ptr<ExternalTexture>& hdr, base::borrowed_fd&& hdrFence,

            float hdrSdrRatio, ui::Dataspace dataspace, const std::shared_ptr<ExternalTexture>& sdr,

            const std::shared_ptr<ExternalTexture>& gainmap);



    // Clean-up method that should be called on the main thread after the
@@ -310,11 +315,10 @@ protected: 
            const DisplaySettings& display, const std::vector<LayerSettings>& layers,

            const std::shared_ptr<ExternalTexture>& buffer, base::unique_fd&& bufferFence) = 0;



    virtual void drawGainmapInternal(

    virtual void tonemapAndDrawGainmapInternal(

            const std::shared_ptr<std::promise<FenceResult>>&& resultPromise,

            const std::shared_ptr<ExternalTexture>& sdr, base::borrowed_fd&& sdrFence,

            const std::shared_ptr<ExternalTexture>& hdr, base::borrowed_fd&& hdrFence,

            float hdrSdrRatio, ui::Dataspace dataspace,

            float hdrSdrRatio, ui::Dataspace dataspace, const std::shared_ptr<ExternalTexture>& sdr,

            const std::shared_ptr<ExternalTexture>& gainmap) = 0;

};

libs/renderengine/include/renderengine/mock/RenderEngine.h

+5 −6
Original line number Diff line number Diff line
@@ -46,17 +46,16 @@ public: 
                 ftl::Future<FenceResult>(const DisplaySettings&, const std::vector<LayerSettings>&,

                                          const std::shared_ptr<ExternalTexture>&,

                                          base::unique_fd&&));

    MOCK_METHOD7(drawGainmap,

    MOCK_METHOD6(tonemapAndDrawGainmap,

                 ftl::Future<FenceResult>(const std::shared_ptr<ExternalTexture>&,

                                          base::borrowed_fd&&,

                                          const std::shared_ptr<ExternalTexture>&,

                                          base::borrowed_fd&&, float, ui::Dataspace,

                                          const std::shared_ptr<ExternalTexture>&,

                                          const std::shared_ptr<ExternalTexture>&));

    MOCK_METHOD8(drawGainmapInternal,

    MOCK_METHOD7(tonemapAndDrawGainmapInternal,

                 void(const std::shared_ptr<std::promise<FenceResult>>&&,

                      const std::shared_ptr<ExternalTexture>&, base::borrowed_fd&&,

                      const std::shared_ptr<ExternalTexture>&, base::borrowed_fd&&, float,

                      ui::Dataspace, const std::shared_ptr<ExternalTexture>&));

                      ui::Dataspace, const std::shared_ptr<ExternalTexture>&,

                      const std::shared_ptr<ExternalTexture>&));

    MOCK_METHOD5(drawLayersInternal,

                 void(const std::shared_ptr<std::promise<FenceResult>>&&, const DisplaySettings&,

                      const std::vector<LayerSettings>&, const std::shared_ptr<ExternalTexture>&,

libs/renderengine/skia/SkiaRenderEngine.cpp

+44 −26
Original line number Diff line number Diff line
@@ -567,9 +567,7 @@ sk_sp<SkShader> SkiaRenderEngine::createRuntimeEffectShader( 
        if (usingLocalTonemap) {

            const float inputRatio =

                    hdrType == HdrRenderType::GENERIC_HDR ? 1.0f : parameters.layerDimmingRatio;

            static MouriMap kMapper;

            shader = kMapper.mouriMap(getActiveContext(), shader, inputRatio,

                                      parameters.display.targetHdrSdrRatio);

            shader = localTonemap(shader, inputRatio, parameters.display.targetHdrSdrRatio);

        }



        // disable tonemapping if we already locally tonemapped
@@ -610,6 +608,12 @@ sk_sp<SkShader> SkiaRenderEngine::createRuntimeEffectShader( 
    return shader;

}



sk_sp<SkShader> SkiaRenderEngine::localTonemap(sk_sp<SkShader> shader, float inputMultiplier,

                                               float targetHdrSdrRatio) {

    static MouriMap kMapper;

    return kMapper.mouriMap(getActiveContext(), shader, inputMultiplier, targetHdrSdrRatio);

}



void SkiaRenderEngine::initCanvas(SkCanvas* canvas, const DisplaySettings& display) {

    if (CC_UNLIKELY(mCapture->isCaptureRunning())) {

        // Record display settings when capture is running.
@@ -1212,44 +1216,58 @@ void SkiaRenderEngine::drawLayersInternal( 
    resultPromise->set_value(std::move(drawFence));

}



void SkiaRenderEngine::drawGainmapInternal(

void SkiaRenderEngine::tonemapAndDrawGainmapInternal(

        const std::shared_ptr<std::promise<FenceResult>>&& resultPromise,

        const std::shared_ptr<ExternalTexture>& sdr, base::borrowed_fd&& sdrFence,

        const std::shared_ptr<ExternalTexture>& hdr, base::borrowed_fd&& hdrFence,

        float hdrSdrRatio, ui::Dataspace dataspace,

        float hdrSdrRatio, ui::Dataspace dataspace, const std::shared_ptr<ExternalTexture>& sdr,

        const std::shared_ptr<ExternalTexture>& gainmap) {

    std::lock_guard<std::mutex> lock(mRenderingMutex);

    auto context = getActiveContext();

    auto surfaceTextureRef = getOrCreateBackendTexture(gainmap->getBuffer(), true);

    sk_sp<SkSurface> dstSurface =

            surfaceTextureRef->getOrCreateSurface(ui::Dataspace::V0_SRGB_LINEAR);



    waitFence(context, sdrFence);

    const auto sdrTextureRef = getOrCreateBackendTexture(sdr->getBuffer(), false);

    const auto sdrImage = sdrTextureRef->makeImage(dataspace, kPremul_SkAlphaType);

    const auto sdrShader =

            sdrImage->makeShader(SkTileMode::kClamp, SkTileMode::kClamp,

                                 SkSamplingOptions({SkFilterMode::kLinear, SkMipmapMode::kNone}),

                                 nullptr);

    auto gainmapTextureRef = getOrCreateBackendTexture(gainmap->getBuffer(), true);

    sk_sp<SkSurface> gainmapSurface =

            gainmapTextureRef->getOrCreateSurface(ui::Dataspace::V0_SRGB_LINEAR);



    auto sdrTextureRef = getOrCreateBackendTexture(sdr->getBuffer(), true);

    sk_sp<SkSurface> sdrSurface = sdrTextureRef->getOrCreateSurface(dataspace);



    waitFence(context, hdrFence);

    const auto hdrTextureRef = getOrCreateBackendTexture(hdr->getBuffer(), false);

    const auto hdrImage = hdrTextureRef->makeImage(dataspace, kPremul_SkAlphaType);

    const auto hdrShader =

            hdrImage->makeShader(SkTileMode::kClamp, SkTileMode::kClamp,

                                 SkSamplingOptions({SkFilterMode::kLinear, SkMipmapMode::kNone}),

                                 SkSamplingOptions({SkFilterMode::kNearest, SkMipmapMode::kNone}),

                                 nullptr);



    const auto tonemappedShader = localTonemap(hdrShader, 1.0f, 1.0f);



    static GainmapFactory kGainmapFactory;

    const auto gainmapShader = kGainmapFactory.createSkShader(sdrShader, hdrShader, hdrSdrRatio);

    const auto gainmapShader =

            kGainmapFactory.createSkShader(tonemappedShader, hdrShader, hdrSdrRatio);



    sp<Fence> drawFence;



    const auto canvas = dstSurface->getCanvas();

    {

        const auto canvas = sdrSurface->getCanvas();

        SkPaint paint;

    paint.setShader(gainmapShader);

        paint.setShader(tonemappedShader);

        paint.setBlendMode(SkBlendMode::kSrc);

        canvas->drawPaint(paint);



    auto drawFence = sp<Fence>::make(flushAndSubmit(context, dstSurface));

        drawFence = sp<Fence>::make(flushAndSubmit(context, sdrSurface));

        trace(drawFence);

    }



    {

        const auto canvas = gainmapSurface->getCanvas();

        SkPaint paint;

        paint.setShader(gainmapShader);

        paint.setBlendMode(SkBlendMode::kSrc);

        canvas->drawPaint(paint);



        auto gmFence = sp<Fence>::make(flushAndSubmit(context, gainmapSurface));

        trace(gmFence);

        drawFence = Fence::merge("gm-ss", drawFence, gmFence);

    }

    resultPromise->set_value(std::move(drawFence));

}