Revert "Use ArraySet for the list of observer WeakReference."

This reverts commit 3488f1af.

Reason for revert: The change to ArraySet increase system_server startup time by 20ms as reported in b/111658334. It is because add() operation into ArraySet takes much longer than ArrayList. 

Change-Id: I1523332a8c9fc74cfef29b119a3c4c54d55596ec