Fix SummaryLoader lifecycle second and for all

Mostly hypothetical fix for weird circumstance where listening
lifecycle gets broken.  To fix this, track the current state
of listening in the SummaryLoader and force it into the non-listening
state when released.

Bug: 26027137
Bug: 26731143
Change-Id: I7299749230924eafa3e6e7d5b0de6e48ff014a38