Don't crash when making a layer larger than supported dimensions

Bug #8437401

A misplaced ref count decrement was causing a crash when attempting to
resize a layer to dimensions larger than the max texture size supported
by the GPU.

This change fixes the crash and clarifies the warnings to make it more
obvious what's happening.

Change-Id: I632dc1b90aaa2605969e10523491a81c4922d3dc