Fix comparison between signed and unsigned error on darwin

mode_t is a uint16_t on darwin, which causes
sb.st_mode & ~S_IFMT
to produce an int when the uint16_t is promoted for the operator.
Cast to unsigned int before comparing against 0660U.

Change-Id: Ib1439c08d9e2b297eeeba701891508d269c19a3d
(cherry-pick from commit 56b37345)