BTM: RNR: Validate page scan repetition mode before send out a request

An invalid parameter would cause bta_dm_device_search get stuck in the
BTA_DM_SEARCH_CANCELLING state.

Bug: 339945771
Bug: 344520866
Tag: #floss
Test: mmm packages/modules/Bluetooth
Test: bluetooth_AdapterQuickHealth.AVL.all_floss
Change-Id: I2978ea591bb7db791174a15dd79c44b3de1532fd