From ec48540fe8c387cf7424d5387ddbd53e89bb9d51 Mon Sep 17 00:00:00 2001 From: Colin Stolley Date: Wed, 27 Nov 2019 16:24:53 -0600 Subject: packfile.c: speed up loading lots of packfiles When loading packfiles on start-up, we traverse the internal packfile list once per file to avoid reloading packfiles that have already been loaded. This check runs in quadratic time, so for poorly maintained repos with a large number of packfiles, it can be pretty slow. Add a hashmap containing the packfile names as we load them so that the average runtime cost of checking for already-loaded packs becomes constant. Add a perf test to p5303 to show speed-up. The existing p5303 test runtimes are dominated by other factors and do not show an appreciable speed-up. The new test in p5303 clearly exposes a speed-up in bad cases. In this test we create 10,000 packfiles and measure the start-up time of git rev-parse, which does little else besides load in the packs. Here are the numbers for the new p5303 test: Test HEAD^ HEAD --------------------------------------------------------------------- 5303.12: load 10,000 packs 1.03(0.92+0.10) 0.12(0.02+0.09) -88.3% Signed-off-by: Colin Stolley Helped-by: Jeff King [jc: squashed the change to call hashmap in install_packed_git() by peff] Signed-off-by: Junio C Hamano --- t/perf/p5303-many-packs.sh | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) (limited to 't/perf') diff --git a/t/perf/p5303-many-packs.sh b/t/perf/p5303-many-packs.sh index 3779851941..ede78e19e2 100755 --- a/t/perf/p5303-many-packs.sh +++ b/t/perf/p5303-many-packs.sh @@ -84,4 +84,22 @@ do ' done +# Measure pack loading with 10,000 packs. +test_expect_success 'generate lots of packs' ' + for i in $(test_seq 10000); do + echo "blob" + echo "data <