Revert "Revert "Fix a11y cache correctness bug""

This reverts commit 4d70022b.
This is so certain tests start passing again. The revert was a low risk
measure for the security update.
We still need to address the crash that this introduces

Test: It builds