staging: drm/omap: Fix usage of IS_ERR_OR_NULL and PTR_ERR

Return -ENOMEM if dmm_txn_init cannot allocate a refill engine.

v2: Fix typing issue seen with newer compilers

Signed-off-by: Andy Gross <andy.gross@ti.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>