drm/nouveau/kms/nv04-nv40: improve overlay error detection, fix pitch setting

We were previously setting the pitch based on a perfectly packed buffer.
This does not necessarily happen. Either modetest started generating
such buffers recently, or earlier testing only happened with well-picked
overlay sizes.

While we're at it, beef up and refactor the error state detection.

Signed-off-by: Ilia Mirkin <imirkin@alum.mit.edu>
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>