gitlab-ci: Only trigger builds if build directory has changed

Doesn't makes sense to trigger builds unless our patches stored inside
build/ directory has changed

Signed-off-by: Aayush Gupta <theimpulson@e.email>