diff options
| author |  Jeff King <peff@peff.net> | 2019-05-06 19:43:34 -0400 |
|---|---|---|
| committer |  Junio C Hamano <gitster@pobox.com> | 2019-05-07 11:37:17 +0900 |
| commit | 960154b9c17afb276e12d0bec83513f3e46de565 (patch) | |
| tree | e9ad87474bd99a807348e42c5f10eb1cfedbd5ee /Makefile | |
| parent | The seventh batch (diff) | |
| download | tgif-960154b9c17afb276e12d0bec83513f3e46de565.tar.xz | |
coccicheck: optionally batch spatch invocations
In our "make coccicheck" rule, we currently feed each source file to its
own individual invocation of spatch. This has a few downsides:
- it repeats any overhead spatch has for starting up and reading the
patch file
- any included header files may get processed from multiple
invocations. This is slow (we see the same header files multiple
times) and may produce a resulting patch with repeated hunks (which
cannot be applied without further cleanup)
Ideally we'd just invoke a single instance of spatch per rule-file and
feed it all source files. But spatch can be rather memory hungry when
run in this way. I measured the peak RSS going from ~90MB for a single
file to ~1900MB for all files. Multiplied by multiple rule files being
processed at the same time (for "make -j"), this can make things slower
or even cause them to fail (e.g., this is reported to happen on our
Travis builds).
Instead, let's provide a tunable knob. We'll leave the default at "1",
but it can be cranked up to "999" for maximum CPU/memory tradeoff, or
people can find points in between that serve their particular machines.
Here are a few numbers running a single rule via:
SIZES='1 4 16 999'
RULE=contrib/coccinelle/object_id.cocci
for i in $SIZES; do
make clean
/usr/bin/time -o $i.out --format='%e | %U | %S | %M' \
make $RULE.patch SPATCH_BATCH_SIZE=$i
done
for i in $SIZES; do
printf '%4d | %s\n' $i "$(cat $i.out)"
done
which yields:
1 | 97.73 | 93.38 | 4.33 | 100128
4 | 52.80 | 51.14 | 1.69 | 135204
16 | 35.82 | 35.09 | 0.76 | 284124
999 | 23.30 | 23.13 | 0.20 | 1903852
The implementation is done with xargs, which should be widely available;
it's in POSIX, we rely on it already in the test suite. And "coccicheck"
is really a developer-only tool anyway, so it's not a big deal if
obscure systems can't run it.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Diffstat (limited to 'Makefile')
| -rw-r--r-- | Makefile | 13 |
1 files changed, 6 insertions, 7 deletions
@@ -1174,8 +1174,10 @@ PTHREAD_CFLAGS = SPARSE_FLAGS ?= SP_EXTRA_FLAGS = -# For the 'coccicheck' target +# For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will +# usually result in less CPU usage at the cost of higher peak memory. SPATCH_FLAGS = --all-includes --patch . +SPATCH_BATCH_SIZE = 1 include config.mak.uname -include config.mak.autogen @@ -2790,12 +2792,9 @@ endif %.cocci.patch: %.cocci $(COCCI_SOURCES) @echo ' ' SPATCH $<; \ - ret=0; \ - for f in $(COCCI_SOURCES); do \ - $(SPATCH) --sp-file $< $$f $(SPATCH_FLAGS) || \ - { ret=$$?; break; }; \ - done >$@+ 2>$@.log; \ - if test $$ret != 0; \ + if ! echo $(COCCI_SOURCES) | xargs -n $(SPATCH_BATCH_SIZE) \ + $(SPATCH) --sp-file $< $(SPATCH_FLAGS) \ + >$@+ 2>$@.log; \ then \ cat $@.log; \ exit 1; \ |