toolbox: ioctl: fix signedness on ioctl_nr

ioctl_nr is passed signed to the kernel, but many users refer
to ioctl's in unsigned 32 bit notation.

Let's use stroll to set it, to let people specify either
negative numbers or large positive numbers.

Also improve error reporting.

Reported-By: Alex Piliev <ktnr74@motorola.com>
Change-Id: I789434895b70b051d4d5c1a5db566a1925478ef0