le_audio: Don't verify periodic audio data time in test

This could lead to issues with no real time scheduling systems.

Tag: #feature
Bug: 266830004
Test: atest bluetooth_le_audio_test
Change-Id: I81066bbb67a7aeef5e79752122b63fafd1f9726c