spi: fsl-(e)spi: simplify cleanup code

Now that most cleanup is done automatically the remove functions
can be significantly simplified.

Signed-off-by: Heiner Kallweit <hkallweit1@gmail.com>
Signed-off-by: Mark Brown <broonie@kernel.org>