[Blackfin] arch: fix bug - Remove module will not free L1 memory used

Remove module will not free L1 memory used which caused by
memory access after free.  This patch fixes it.

Signed-off-by: Meihui Fan <mhfan@hhcn.com>
Signed-off-by: Bryan Wu <cooloney@kernel.org>