Keep track of remaining fd when devices are removed

Sometimes the wrong fd was accessed when the device was addressed
by device id.

The earlier implementation assumed that two arrays were in sync
but one of them was compacted when devices were removed. Instead
of that dependency the device now keeps track of it's file descriptor.

Change-Id: Ib0f320603aafb07ded354bc3687de9759c9068f2