From 1d89d88d37d18cd3af37ef4742aa39282441ec14 Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 28 Sep 2021 21:55:18 -0400 Subject: builtin/repack.c: support writing a MIDX while repacking Teach `git repack` a new `--write-midx` option for callers that wish to persist a multi-pack index in their repository while repacking. There are two existing alternatives to this new flag, but they don't cover our particular use-case. These alternatives are: - Call 'git multi-pack-index write' after running 'git repack', or - Set 'GIT_TEST_MULTI_PACK_INDEX=1' in your environment when running 'git repack'. The former works, but introduces a gap in bitmap coverage between repacking and writing a new MIDX (since the repack may have deleted a pack included in the existing MIDX, invalidating it altogether). Setting the 'GIT_TEST_' environment variable is obviously unsupported. In fact, even if it were supported officially, it still wouldn't work, because it generates the MIDX *after* redundant packs have been dropped, leading to the same issue as above. Introduce a new option which eliminates this race by teaching `git repack` to generate the MIDX at the critical point: after the new packs have been written and moved into place, but before the redundant packs have been removed. This option is compatible with `git repack`'s '--bitmap' option (it changes the interpretation to be: "write a bitmap corresponding to the MIDX after one has been generated"). There is a little bit of additional noise in the patch below to avoid repeating ourselves when selecting which packs to delete. Instead of a single loop as before (where we iterate over 'existing_packs', decide if a pack is worth deleting, and if so, delete it), we have two loops (the first where we decide which ones are worth deleting, and the second where we actually do the deleting). This makes it so we have a single check we can make consistently when (1) telling the MIDX which packs we want to exclude, and (2) actually unlinking the redundant packs. There is also a tiny change to short-circuit the body of write_midx_included_packs() when no packs remain in the case of an empty repository. The MIDX code does not handle this, so avoid trying to generate a MIDX covering zero packs in the first place. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- t/lib-midx.sh | 8 ++++ t/t7700-repack.sh | 96 +++++++++++++++++++++++++++++++++++++++++++++ t/t7703-repack-geometric.sh | 2 +- 3 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 t/lib-midx.sh (limited to 't') diff --git a/t/lib-midx.sh b/t/lib-midx.sh new file mode 100644 index 0000000000..1261994744 --- /dev/null +++ b/t/lib-midx.sh @@ -0,0 +1,8 @@ +# test_midx_consistent +test_midx_consistent () { + ls $1/pack/pack-*.idx | xargs -n 1 basename | sort >expect && + test-tool read-midx $1 | grep ^pack-.*\.idx$ | sort >actual && + + test_cmp expect actual && + git multi-pack-index --object-dir=$1 verify +} diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh index 98eda3bfeb..6792531dfd 100755 --- a/t/t7700-repack.sh +++ b/t/t7700-repack.sh @@ -3,6 +3,8 @@ test_description='git repack works correctly' . ./test-lib.sh +. "${TEST_DIRECTORY}/lib-bitmap.sh" +. "${TEST_DIRECTORY}/lib-midx.sh" commit_and_pack () { test_commit "$@" 1>&2 && @@ -234,4 +236,98 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' ' test_must_be_empty actual ' +objdir=.git/objects +midx=$objdir/pack/multi-pack-index + +test_expect_success 'setup for --write-midx tests' ' + git init midx && + ( + cd midx && + git config core.multiPackIndex true && + + test_commit base + ) +' + +test_expect_success '--write-midx unchanged' ' + ( + cd midx && + GIT_TEST_MULTI_PACK_INDEX=0 git repack && + test_path_is_missing $midx && + test_path_is_missing $midx-*.bitmap && + + GIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx && + + test_path_is_file $midx && + test_path_is_missing $midx-*.bitmap && + test_midx_consistent $objdir + ) +' + +test_expect_success '--write-midx with a new pack' ' + ( + cd midx && + test_commit loose && + + GIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx && + + test_path_is_file $midx && + test_path_is_missing $midx-*.bitmap && + test_midx_consistent $objdir + ) +' + +test_expect_success '--write-midx with -b' ' + ( + cd midx && + GIT_TEST_MULTI_PACK_INDEX=0 git repack -mb && + + test_path_is_file $midx && + test_path_is_file $midx-*.bitmap && + test_midx_consistent $objdir + ) +' + +test_expect_success '--write-midx with -d' ' + ( + cd midx && + test_commit repack && + + GIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad --write-midx && + + test_path_is_file $midx && + test_path_is_missing $midx-*.bitmap && + test_midx_consistent $objdir + ) +' + +test_expect_success 'cleans up MIDX when appropriate' ' + ( + cd midx && + + test_commit repack-2 && + GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx && + + checksum=$(midx_checksum $objdir) && + test_path_is_file $midx && + test_path_is_file $midx-$checksum.bitmap && + test_path_is_file $midx-$checksum.rev && + + test_commit repack-3 && + GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx && + + test_path_is_file $midx && + test_path_is_missing $midx-$checksum.bitmap && + test_path_is_missing $midx-$checksum.rev && + test_path_is_file $midx-$(midx_checksum $objdir).bitmap && + test_path_is_file $midx-$(midx_checksum $objdir).rev && + + test_commit repack-4 && + GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb && + + find $objdir/pack -type f -name "multi-pack-index*" >files && + test_must_be_empty files + ) +' + test_done diff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh index 5ccaa440e0..67049f7637 100755 --- a/t/t7703-repack-geometric.sh +++ b/t/t7703-repack-geometric.sh @@ -15,7 +15,7 @@ test_expect_success '--geometric with no packs' ' ( cd geometric && - git repack --geometric 2 >out && + git repack --write-midx --geometric 2 >out && test_i18ngrep "Nothing new to pack" out ) ' -- cgit v1.2.3