Optimize loading permission usages.

Make a few minor optimizations involved in loading permission usages.
Together, these reduce loading time (without UI elements) by about 15%
on my dev device.

Bug: 124052170
Test: Time loading permission usages with and without this change.
Change-Id: Iea0b07d47b719c0b631925751692058d17aaf5dd