Fix crash when unfolding a device.

Fix a crash when unfolding a device with reference implementation.
There is a crash because we get a callback before changes propagate.

Bug: 218980376
Test: manual - open the fold samples with config changing app.
 fold and unfold the device.
Change-Id: I3c99058e47e29b18a30921bc23b14182e970ac65