mtd: tests: fix more potential integer overflows

Caught by Coverity (CID #200625 and others)

Signed-off-by: Brian Norris <computersforpeace@gmail.com>
Cc: Akinobu Mita <akinobu.mita@gmail.com>