Prevent ConcurrentModificationExceptions

Use sets backed by ConcurrentHashMaps instead of HashSets to prevent
concurrent exceptions if listeners try to remove themselves in callbacks
while iterating over the listeners.

Bug:16325026
Change-Id: I5089eac83154f8d71b759a891c0fbf3d3a9a4e6a