Revert "leaudio: Fix removing group descriptors"

This reverts commit d430d357.

Reason for revert: This cl didn't pass TH before merging, which brings the build failure and blocking other fixes, see b/202679185 for details. Try to revert this change first.

Bug: 150670922
Tag: #feature
Sponsor: jpawlowski@
Test: atest BluetoothInstrumentationTests

Change-Id: I4067a20f4892993184cfa56c4771c333db1bed48