ASoC: rcar: call missing of_clk_del_provider() when remove

adg is calling of_clk_add_provider() when probe time,
thus, remove should call of_clk_del_provider(), it doesn't now.
This patch fix this issue.

Signed-off-by: Kuninori Morimoto <kuninori.morimoto.gx@renesas.com>
Signed-off-by: Mark Brown <broonie@kernel.org>