Revert "ARM: tegra: Fix ULPI regression on Tegra20"

This reverts commit b39ac542.

The issue was fixed by upstream commit 5d797111afe1 ("clk:
tegra: Add quirk for getting CDEV1/2 clocks on Tegra20").

Signed-off-by: Sasha Levin <sashal@kernel.org>