Commit a052ddbb authored Sep 25, 2023 by Colin Cross

Shard srcjars when sharding javac compilation

java_library rules with javac_shard_size set split the sources into
shards to invoke javac multiple times, but were using a single javac
invocation for all srcjars.  For fraemwork-minus-apex, this srcjar
shard was the long pole at 15.7 seconds, containing 266 srcjars with
1542 java files with a total of 614593 lines.

Use a rough approximation of 5 sources per srcjar to determine the
number of shards to split the srcjars into based on javac_shard_size.
This results in splitting the srcjars for frameworks-minus-apex into
8 shards, with the longest taking 10.5 seconds to compile.

The longest shard contains most of the aidl srcjars, which have been
generated by sharded groups of 50 aidl files and have a much higher
average number of sources per srcjar (a mean and median of 27).  A
future improvement could be to shard those separately assuming a
higher number of sources per srcjar.

Bug: 302033097
Test: USE_RBE=false m frameworks-minus-apex
Change-Id: I85e740c7fcf5651cf18c0cdc90ab8c6ee39cb47b

parent ed27322a

java/base.go

+10 −3

Original line number	Diff line number	Diff line
		@@ -1343,11 +1343,18 @@ func (j *Module) compile(ctx android.ModuleContext, extraSrcJars, extraClasspath
		jars = append(jars, classes)
		}
		}
		// Assume approximately 5 sources per srcjar.
		// For framework-minus-apex in AOSP at the time this was written, there are 266 srcjars, with a mean
		// of 5.8 sources per srcjar, but a median of 1, a standard deviation of 10, and a max of 48 source files.
		if len(srcJars) > 0 {
		classes := j.compileJavaClasses(ctx, jarName, len(shardSrcs),
		nil, srcJars, flags, extraJarDeps)
		startIdx := len(shardSrcs)
		shardSrcJarsList := android.ShardPaths(srcJars, shardSize/5)
		for idx, shardSrcJars := range shardSrcJarsList {
		classes := j.compileJavaClasses(ctx, jarName, startIdx+idx,
		nil, shardSrcJars, flags, extraJarDeps)
		jars = append(jars, classes)
		}
		}
		} else {
		classes := j.compileJavaClasses(ctx, jarName, -1, uniqueJavaFiles, srcJars, flags, extraJarDeps)
		jars = append(jars, classes)