Revert "Revert "Use std::set instead of SortedVector""

This reverts commit d688ab1b.

Reason for revert: Likely build was having issues, and the original CLs were fine. From looking into the issues more closely, it appears that the topic was not merged atomically.

Change-Id: I26bd8104a6d397f56dde305b952467f9ef81ee4f