Use legacy layer name for screenshot shell display rotation

Just to avoid divergence of flicker test which assumes there will
be a "RotationLayer" when running display rotation animation.

Also make the condition to calculate luma more accurate.

Bug: 242056267
Test: setprop persist.wm.debug.shell_transit_blast 0;
      setprop persist.wm.debug.shell_transit 1; reboot
      atest FlickerTests:ChangeAppRotationTest
Change-Id: Icedc18192249fbba27a6429e66d0e81c42cefc22