Revert "Revert "Use std::set instead of SortedVector""

This reverts commit 2d8cc894.

Reason for revert: Likely build was having issues, and the original CLs were fine. From looking into the issues more closely, it appears that the topic was not merged atomically.

Change-Id: I42393e615066b4cf10aa953757a3f43be5676a2f