summaryrefslogtreecommitdiff
path: root/t/t1700-split-index.sh
AgeCommit message (Collapse)AuthorFilesLines
2021-09-07read-cache: fix GIT_TEST_SPLIT_INDEXLibravatar SZEDER Gábor1-0/+11
Running tests with GIT_TEST_SPLIT_INDEX=1 is supposed to turn on the split index feature and trigger index splitting (mostly) randomly. Alas, this has been broken since 6e37c8ed3c (read-cache.c: fix writing "link" index ext with null base oid, 2019-02-13), and GIT_TEST_SPLIT_INDEX=1 hasn't triggered any index splitting since then. This patch makes GIT_TEST_SPLIT_INDEX work again, though it doesn't restore the pre-6e37c8ed3c behavior. To understand the bug, the fix, and the behavior change we first have to look at how GIT_TEST_SPLIT_INDEX used to work before 6e37c8ed3c: There are two places where we check the value of GIT_TEST_SPLIT_INDEX, and before 6e37c8ed3c they worked like this: 1) In the lower-level do_write_index(), where, if GIT_TEST_SPLIT_INDEX is enabled, we call init_split_index(). This call merely allocates and zero-initializes 'istate->split_index', but does nothing else (i.e. doesn't fill the base/shared index with cache entries, doesn't actually write a shared index file, etc.). Pertinent to this issue, the hash of the base index remains all zeroed out. 2) In the higher-level write_locked_index(), but only when 'istate->split_index' has already been initialized. Then, if GIT_TEST_SPLIT_INDEX is enabled, it randomly sets the flag that triggers index splitting later in this function. This randomness comes from the first byte of the hash of the base index via an 'if ((first_byte & 15) < 6)' condition. However, if 'istate->split_index' hasn't been initialized (i.e. it is still NULL), then write_locked_index() just calls do_write_locked_index(), which internally calls the above mentioned do_write_index(). This means that while GIT_TEST_SPLIT_INDEX=1 usually triggered index splitting randomly, the first two index writes were always deterministic (though I suspect this was unintentional): - The initial index write never splits the index. During the first index write write_locked_index() is called with 'istate->split_index' still uninitialized, so the check in 2) is not executed. It still calls do_write_index(), though, which then executes the check in 1). The resulting all zero base index hash then leads to the 'link' extension being written to '.git/index', though a shared index file is not written: $ rm .git/index $ GIT_TEST_SPLIT_INDEX=1 git update-index --add file $ test-tool dump-split-index .git/index own c6ef71168597caec8553c83d9d0048f1ef416170 base 0000000000000000000000000000000000000000 100644 d00491fd7e5bb6fa28c517a0bb32b8b506539d4d 0 file replacements: deletions: $ ls -l .git/sharedindex.* ls: cannot access '.git/sharedindex.*': No such file or directory - The second index write always splits the index. When the index written in the previous point is read, 'istate->split_index' is initialized because of the presence of the 'link' extension. So during the second write write_locked_index() does run the check in 2), and the first byte of the all zero base index hash always fulfills the randomness condition, which in turn always triggers the index splitting. - Subsequent index writes will find the 'link' extension with a real non-zero base index hash, so from then on the check in 2) is executed and the first byte of the base index hash is as random as it gets (coming from the SHA-1 of index data including timestamps and inodes...). All this worked until 6e37c8ed3c came along, and stopped writing the 'link' extension if the hash of the base index was all zero: $ rm .git/index $ GIT_TEST_SPLIT_INDEX=1 git update-index --add file $ test-tool dump-split-index .git/index own abbd6f6458d5dee73ae8e210ca15a68a390c6fd7 not a split index $ ls -l .git/sharedindex.* ls: cannot access '.git/sharedindex.*': No such file or directory So, since the first index write with GIT_TEST_SPLIT_INDEX=1 doesn't write a 'link' extension, in the second index write 'istate->split_index' remains uninitialized, and the check in 2) is not executed, and ultimately the index is never split. Fix this by modifying write_locked_index() to make sure to check GIT_TEST_SPLIT_INDEX even if 'istate->split_index' is still uninitialized, and initialize it if necessary. The check for GIT_TEST_SPLIT_INDEX and separate init_split_index() call in do_write_index() thus becomes unnecessary, so remove it. Furthermore, add a test to 't1700-split-index.sh' to make sure that GIT_TEST_SPLIT_INDEX=1 will keep working (though only check the index splitting on the first index write, because after that it will be random). Note that this change does not restore the pre-6e37c8ed3c behaviour, as it will deterministically split the index already on the first index write. Since GIT_TEST_SPLIT_INDEX is purely a developer aid, there is no backwards compatibility issue here. The new behaviour did trigger test failures in 't0003-attributes.sh' and 't1600-index.sh', though, which have been fixed in preparatory patches in this series. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Acked-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-07read-cache: look for shared index files next to the index, tooLibravatar SZEDER Gábor1-0/+23
When reading a split index git always looks for its referenced shared base index in the gitdir of the current repository, even when reading an alternate index specified via GIT_INDEX_FILE, and even when that alternate index file is the "main" '.git/index' file of an other repository. However, if that split index and its referenced shared index files were written by a git command running entirely in that other repository, then, naturally, the shared index file is written to that other repository's gitdir. Consequently, a git command attempting to read that shared index file while running in a different repository won't be able find it and will error out. I'm not sure in what use case it is necessary to read the index of one repository by a git command running in a different repository, but it is certainly possible to do so, and in fact the test 'bare repository: check that --cached honors index' in 't0003-attributes.sh' does exactly that. If GIT_TEST_SPLIT_INDEX=1 were to split the index in just the right moment [1], then this test would indeed fail, because the referenced shared index file could not be found. Let's look for the referenced shared index file not only in the gitdir of the current directory, but, if the shared index is not there, right next to the split index as well. [1] We haven't seen this issue trigger a failure in t0003 yet, because: - While GIT_TEST_SPLIT_INDEX=1 is supposed to trigger index splitting randomly, the first index write has always been deterministic and it has never split the index. - That alternate index file in the other repository is written only once in the entire test script, so it's never split. However, the next patch will fix GIT_TEST_SPLIT_INDEX, and while doing so it will slightly change its behavior to always split the index already on the first index write, and t0003 would always fail without this patch. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Acked-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-11-19t[01]*: adjust the references to the default branch name "main"Libravatar Johannes Schindelin1-2/+2
Carefully excluding t1309, which sees independent development elsewhere at the time of writing, we transition above-mentioned tests to the default branch name `main`. This trick was performed via $ (cd t && sed -i -e 's/master/main/g' -e 's/MASTER/MAIN/g' \ -e 's/Master/Main/g' -e 's/naster/nain/g' -- t[01]*.sh && git checkout HEAD -- t1309\*) Note that t5533 contains a variation of the name `master` (`naster`) that we rename here, too. This allows us to define `GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main` for those tests. Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-11-19tests: mark tests relying on the current default for `init.defaultBranch`Libravatar Johannes Schindelin1-0/+3
In addition to the manual adjustment to let the `linux-gcc` CI job run the test suite with `master` and then with `main`, this patch makes sure that GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME is set in all test scripts that currently rely on the initial branch name being `master by default. To determine which test scripts to mark up, the first step was to force-set the default branch name to `master` in - all test scripts that contain the keyword `master`, - t4211, which expects `t/t4211/history.export` with a hard-coded ref to initialize the default branch, - t5560 because it sources `t/t556x_common` which uses `master`, - t8002 and t8012 because both source `t/annotate-tests.sh` which also uses `master`) This trick was performed by this command: $ sed -i '/^ *\. \.\/\(test-lib\|lib-\(bash\|cvs\|git-svn\)\|gitweb-lib\)\.sh$/i\ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\ ' $(git grep -l master t/t[0-9]*.sh) \ t/t4211*.sh t/t5560*.sh t/t8002*.sh t/t8012*.sh After that, careful, manual inspection revealed that some of the test scripts containing the needle `master` do not actually rely on a specific default branch name: either they mention `master` only in a comment, or they initialize that branch specificially, or they do not actually refer to the current default branch. Therefore, the aforementioned modification was undone in those test scripts thusly: $ git checkout HEAD -- \ t/t0027-auto-crlf.sh t/t0060-path-utils.sh \ t/t1011-read-tree-sparse-checkout.sh \ t/t1305-config-include.sh t/t1309-early-config.sh \ t/t1402-check-ref-format.sh t/t1450-fsck.sh \ t/t2024-checkout-dwim.sh \ t/t2106-update-index-assume-unchanged.sh \ t/t3040-subprojects-basic.sh t/t3301-notes.sh \ t/t3308-notes-merge.sh t/t3423-rebase-reword.sh \ t/t3436-rebase-more-options.sh \ t/t4015-diff-whitespace.sh t/t4257-am-interactive.sh \ t/t5323-pack-redundant.sh t/t5401-update-hooks.sh \ t/t5511-refspec.sh t/t5526-fetch-submodules.sh \ t/t5529-push-errors.sh t/t5530-upload-pack-error.sh \ t/t5548-push-porcelain.sh \ t/t5552-skipping-fetch-negotiator.sh \ t/t5572-pull-submodule.sh t/t5608-clone-2gb.sh \ t/t5614-clone-submodules-shallow.sh \ t/t7508-status.sh t/t7606-merge-custom.sh \ t/t9302-fast-import-unpack-limit.sh We excluded one set of test scripts in these commands, though: the range of `git p4` tests. The reason? `git p4` stores the (foreign) remote branch in the branch called `p4/master`, which is obviously not the default branch. Manual analysis revealed that only five of these tests actually require a specific default branch name to pass; They were modified thusly: $ sed -i '/^ *\. \.\/lib-git-p4\.sh$/i\ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\ ' t/t980[0167]*.sh t/t9811*.sh Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-07-01t1710: make hash independentLibravatar brian m. carlson1-17/+34
This test uses several index hashes, which necessarily depend on the version of the index and the hash algorithm in use. Use test_oid_cache to provide values for these for both SHA-1 and SHA-256. Also, compute an object ID and use $EMPTY_BLOB to make the remainder of the tests independent of the hash algorithm in use. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-07Merge branch 'nd/split-index-null-base-fix'Libravatar Junio C Hamano1-0/+18
Split-index fix. * nd/split-index-null-base-fix: read-cache.c: fix writing "link" index ext with null base oid
2019-02-13read-cache.c: fix writing "link" index ext with null base oidLibravatar Nguyễn Thái Ngọc Duy1-0/+18
Since commit 7db118303a (unpack_trees: fix breakage when o->src_index != o->dst_index - 2018-04-23) and changes in merge code to use separate index_state for source and destination, when doing a merge with split index activated, we may run into this line in unpack_trees(): o->result.split_index = init_split_index(&o->result); This is by itself not wrong. But this split index information is not fully populated (and it's only so when move_cache_to_base_index() is called, aka force splitting the index, or loading index_state from a file). Both "base_oid" and "base" in this case remain null. So when writing the main index down, we link to this index with null oid (default value after init_split_index()), which also means "no split index" internally. This triggers an incorrect base index refresh: warning: could not freshen shared index '.../sharedindex.0{40}' This patch makes sure we will not refresh null base_oid (because the file is never there). It also makes sure not to write "link" extension with null base_oid in the first place (no point having it at all). Read code already has protection against null base_oid. There is also another side fix in remove_split_index() that causes a crash when doing "git update-index --no-split-index" when base_oid in the index file is null. In this case we will not load istate->split_index->base but we dereference it anyway and are rewarded with a segfault. This should not happen anymore, but it's still wrong to dereference a potential NULL pointer, especially when we do check for NULL pointer in the next code. Reported-by: Luke Diamand <luke@diamand.org> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-21Merge branch 'sg/split-index-test' into maintLibravatar Junio C Hamano1-1/+1
Test updates. * sg/split-index-test: t0090: disable GIT_TEST_SPLIT_INDEX for the test checking split index t1700-split-index: drop unnecessary 'grep'
2018-11-21Merge branch 'cc/shared-index-permbits'Libravatar Junio C Hamano1-0/+20
The way .git/index and .git/sharedindex* files were initially created gave these files different perm bits until they were adjusted for shared repository settings. This was made consistent. * cc/shared-index-permbits: read-cache: make the split index obey umask settings
2018-11-21eoie: default to not writing EOIE sectionLibravatar Jonathan Nieder1-4/+7
Since 3b1d9e04 (eoie: add End of Index Entry (EOIE) extension, 2018-10-10) Git defaults to writing the new EOIE section when writing out an index file. Usually that is a good thing because it improves threaded performance, but when a Git repository is shared with older versions of Git, it produces a confusing warning: $ git status ignoring EOIE extension HEAD detached at 371ed0defa nothing to commit, working tree clean Let's introduce the new index extension more gently. First we'll roll out the new version of Git that understands it, and then once sufficiently many users are using such a version, we can flip the default to writing it by default. Introduce a '[index] recordEndOfIndexEntries' configuration variable to allow interested users to benefit from this index extension early. Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-19read-cache: make the split index obey umask settingsLibravatar Ævar Arnfjörð Bjarmason1-0/+20
Make the split index write out its .git/sharedindex_* files with the same permissions as .git/index. This only changes the behavior when core.sharedRepository isn't set, i.e. the user's umask settings will be respected. This hasn't been the case ever since the split index was originally implemented in c18b80a0e8 ("update-index: new options to enable/disable split index mode", 2014-06-13). A mkstemp()-like function has always been used to create it. First mkstemp() itself, and then later our own mkstemp()-like in f6ecc62dbf ("write_shared_index(): use tempfile module", 2015-08-10) A related bug was fixed in df801f3f9f ("read-cache: use shared perms when writing shared index", 2017-06-25). Since then the split index has respected core.sharedRepository. However, using that setting should not be required simply to make git obey the user's umask setting. It's intended for the use-case of overriding whatever that umask is set to. This fixes cases where the user has e.g. set his umask to 022 on a shared server in anticipation of other user's needing to run "status", "log" etc. in his repository. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-26Merge branch 'sg/split-index-racefix'Libravatar Junio C Hamano1-21/+28
The codepath to support the experimental split-index mode had remaining "racily clean" issues fixed. * sg/split-index-racefix: split-index: BUG() when cache entry refers to non-existing shared entry split-index: smudge and add racily clean cache entries to split index split-index: don't compare cached data of entries already marked for split index split-index: count the number of deleted entries t1700-split-index: date back files to avoid racy situations split-index: add tests to demonstrate the racy split index problem t1700-split-index: document why FSMONITOR is disabled in this test script
2018-10-19Merge branch 'bp/read-cache-parallel'Libravatar Junio C Hamano1-4/+9
A new extension to the index file has been introduced, which allows the file to be read in parallel. * bp/read-cache-parallel: read-cache: load cache entries on worker threads ieot: add Index Entry Offset Table (IEOT) extension read-cache: load cache extensions on a worker thread config: add new index.threads config setting eoie: add End of Index Entry (EOIE) extension read-cache: clean up casting and byte decoding read-cache.c: optimize reading index format v4
2018-10-19Merge branch 'bp/rename-test-env-var'Libravatar Junio C Hamano1-1/+1
Some environment variables that control the runtime options of Git used during tests are getting renamed for consistency. * bp/rename-test-env-var: t0000: do not get self-test disrupted by environment warnings preload-index: update GIT_FORCE_PRELOAD_TEST support read-cache: update TEST_GIT_INDEX_VERSION support fsmonitor: update GIT_TEST_FSMONITOR support preload-index: use git_env_bool() not getenv() for customization t/README: correct spelling of "uncommon"
2018-10-12t1700-split-index: date back files to avoid racy situationsLibravatar SZEDER Gábor1-21/+28
't1700-split-index.sh' checks that the index was split correctly under various circumstances and that all the different ways to turn the split index feature on and off work correctly. To do so, most of its tests use 'test-tool dump-split-index' to see which files have their cache entries in the split index. All these tests assume that all cache entries are written to the shared index (called "base" throughout these tests) when a new shared index is created. This is an implementation detail: most git commands (basically all except 'git update-index') don't care or know at all about split index or whether a cache entry is stored in the split or shared index. As demonstrated in the previous patch, refreshing a split index is prone to a variant of the classic racy git issue. The next patch will fix this issue, but while doing so it will also slightly change this behaviour: only cache entries with mtime in the past will be written only to the newly created shared index, but racily clean cache entries will be written to the new split index (with smudged stat data). While this upcoming change won't at all affect any git commands, it will violate the above mentioned assumption of 't1700's tests. Since these tests create or modify files and create or refresh the split index in rapid succession, there are plenty of racily clean cache entries to be dealt with, which will then be written to the new split indexes, and, ultimately, will cause several tests in 't1700' to fail. Let's prepare 't1700-split-index.sh' for this upcoming change and modify its tests to avoid racily clean files by backdating the mtime of any file modifications (and since a lot of tests create or modify files, encapsulate it into a helper function). Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-11config: add new index.threads config settingLibravatar Ben Peart1-0/+5
Add support for a new index.threads config setting which will be used to control the threading code in do_read_index(). A value of 0 will tell the index code to automatically determine the correct number of threads to use. A value of 1 will make the code single threaded. A value greater than 1 will set the maximum number of threads to use. For testing purposes, this setting can be overwritten by setting the GIT_TEST_INDEX_THREADS=<n> environment variable to a value greater than 0. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-11eoie: add End of Index Entry (EOIE) extensionLibravatar Ben Peart1-4/+4
The End of Index Entry (EOIE) is used to locate the end of the variable length index entries and the beginning of the extensions. Code can take advantage of this to quickly locate the index extensions without having to parse through all of the index entries. The EOIE extension is always written out to the index file including to the shared index when using the split index feature. Because it is always written out, the SHA checksums in t/t1700-split-index.sh were updated to reflect its inclusion. It is written as an optional extension to ensure compatibility with other git implementations that do not yet support it. It is always written out to ensure it is available as often as possible to speed up index operations. Because it must be able to be loaded before the variable length cache entries and other index extensions, this extension must be written last. The signature for this extension is { 'E', 'O', 'I', 'E' }. The extension consists of: - 32-bit offset to the end of the index entries - 160-bit SHA-1 over the extension types and their sizes (but not their contents). E.g. if we have "TREE" extension that is N-bytes long, "REUC" extension that is M-bytes long, followed by "EOIE", then the hash would be: SHA-1("TREE" + <binary representation of N> + "REUC" + <binary representation of M>) Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-28fsmonitor: update GIT_TEST_FSMONITOR supportLibravatar Ben Peart1-1/+1
Rename GIT_FSMONITOR_TEST to GIT_TEST_FSMONITOR for consistency with the other GIT_TEST_ special setups and properly document its use. Add logic in t/test-lib.sh to give a warning when the old variable is set to let people know they need to update their environment to use the new variable. Remove the outdated instructions on how to run the test suite utilizing fsmonitor now that it is properly documented in t/README. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-28t1700-split-index: document why FSMONITOR is disabled in this test scriptLibravatar SZEDER Gábor1-0/+3
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-12t1700-split-index: drop unnecessary 'grep'Libravatar SZEDER Gábor1-1/+1
The test 'disable split index' in 't1700-split-index.sh' runs the following pipeline: cmd | grep <pattern> | sed s/// Drop that 'grep' from the pipeline, and let 'sed' take over its duties. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-08-27Merge branch 'sg/test-must-be-empty'Libravatar Junio C Hamano1-3/+1
Test fixes. * sg/test-must-be-empty: tests: use 'test_must_be_empty' instead of 'test_cmp <empty> <out>' tests: use 'test_must_be_empty' instead of 'test_cmp /dev/null <out>' tests: use 'test_must_be_empty' instead of 'test ! -s' tests: use 'test_must_be_empty' instead of '! test -s'
2018-08-21tests: use 'test_must_be_empty' instead of 'test_cmp <empty> <out>'Libravatar SZEDER Gábor1-3/+1
Using 'test_must_be_empty' is shorter and more idiomatic than >empty && test_cmp empty out as it saves the creation of an empty file. Furthermore, sometimes the expected empty file doesn't have such a descriptive name like 'empty', and its creation is far away from the place where it's finally used for comparison (e.g. in 't7600-merge.sh', where two expected empty files are created in the 'setup' test, but are used only about 500 lines later). These cases were found by instrumenting 'test_cmp' to error out the test script when it's used to compare empty files, and then converted manually. Note that even after this patch there still remain a lot of cases where we use 'test_cmp' to check empty files: - Sometimes the expected output is not hard-coded in the test, but 'test_cmp' is used to ensure that two similar git commands produce the same output, and that output happens to be empty, e.g. the test 'submodule update --merge - ignores --merge for new submodules' in 't7406-submodule-update.sh'. - Repetitive common tasks, including preparing the expected results and running 'test_cmp', are often extracted into a helper function, and some of this helper's callsites expect no output. - For the same reason as above, the whole 'test_expect_success' block is within a helper function, e.g. in 't3070-wildmatch.sh'. - Or 'test_cmp' is invoked in a loop, e.g. the test 'cvs update (-p)' in 't9400-git-cvsserver-server.sh'. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-03t: use test_might_fail() instead of manipulating exit code manuallyLibravatar Eric Sunshine1-1/+1
These tests manually coerce the exit code of invoked commands to "success" when they don't care if the command succeeds or fails since failure of those commands should not cause the test to fail overall. In doing so, they intentionally break the &&-chain. Modernize by replacing manual exit code management with test_might_fail() and a normal &&-chain. Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-05-14t: switch $_z40 to $ZERO_OIDLibravatar brian m. carlson1-1/+1
Switch all uses of $_z40 to $ZERO_OID so that they work correctly with larger hashes. This commit was created by using the following sed command to modify all files in the t directory except t/test-lib.sh: sed -i 's/\$_z40/$ZERO_OID/g' Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-27t/helper: merge test-index-version into test-toolLibravatar Nguyễn Thái Ngọc Duy1-1/+1
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-27t/helper: merge test-dump-split-index into test-toolLibravatar Nguyễn Thái Ngọc Duy1-24/+24
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-27t/helper: merge test-dump-cache-tree into test-toolLibravatar Nguyễn Thái Ngọc Duy1-1/+1
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-27t/helper: merge test-chmtime into test-toolLibravatar Nguyễn Thái Ngọc Duy1-6/+6
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-02-13Merge branch 'nd/shared-index-fix'Libravatar Junio C Hamano1-0/+19
Code clean-up. * nd/shared-index-fix: read-cache: don't write index twice if we can't write shared index read-cache.c: move tempfile creation/cleanup out of write_shared_index read-cache.c: change type of "temp" in write_shared_index()
2018-02-13Merge branch 'tg/split-index-fixes'Libravatar Junio C Hamano1-0/+19
The split-index mode had a few corner case bugs fixed. * tg/split-index-fixes: travis: run tests with GIT_TEST_SPLIT_INDEX split-index: don't write cache tree with null oid entries read-cache: fix reading the shared index for other repos
2018-01-24read-cache: don't write index twice if we can't write shared indexLibravatar Nguyễn Thái Ngọc Duy1-0/+19
In a0a967568e ("update-index --split-index: do not split if $GIT_DIR is read only", 2014-06-13), we tried to make sure we can still write an index, even if the shared index can not be written. We did so by just calling 'do_write_locked_index()' just before 'write_shared_index()'. 'do_write_locked_index()' always at least closes the tempfile nowadays, and used to close or commit the lockfile if COMMIT_LOCK or CLOSE_LOCK were given at the time this feature was introduced. COMMIT_LOCK or CLOSE_LOCK is passed in by most callers of 'write_locked_index()'. After calling 'write_shared_index()', we call 'write_split_index()', which calls 'do_write_locked_index()' again, which then tries to use the closed lockfile again, but in fact fails to do so as it's already closed. This eventually leads to a segfault. Make sure to write the main index only once. [nd: most of the commit message and investigation done by Thomas, I only tweaked the solution a bit] Helped-by: Thomas Gummerer <t.gummerer@gmail.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-01-19split-index: don't write cache tree with null oid entriesLibravatar Thomas Gummerer1-0/+19
In a96d3cc3f6 ("cache-tree: reject entries with null sha1", 2017-04-21) we made sure that broken cache entries do not get propagated to new trees. Part of that was making sure not to re-use an existing cache tree that includes a null oid. It did so by dropping the cache tree in 'do_write_index()' if one of the entries contains a null oid. In split index mode however, there are two invocations to 'do_write_index()', one for the shared index and one for the split index. The cache tree is only written once, to the split index. As we only loop through the elements that are effectively being written by the current invocation, that may not include the entry with a null oid in the split index (when it is already written to the shared index), where we write the cache tree. Therefore in split index mode we may still end up writing the cache tree, even though there is an entry with a null oid in the index. Fix this by checking for null oids in prepare_to_write_split_index, where we loop the entries of the shared index as well as the entries for the split index. This fixes t7009 with GIT_TEST_SPLIT_INDEX. Also add a new test that's more specifically showing the problem. Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-10-01split-index: disable the fsmonitor extension when running the split index testLibravatar Ben Peart1-0/+1
The split index test t1700-split-index.sh has hard coded SHA values for the index. Currently it supports index V4 and V3 but assumes there are no index extensions loaded. When manually forcing the fsmonitor extension to be turned on when running the test suite, the SHA values no longer match which causes the test to fail. The potential matrix of index extensions and index versions can is quite large so instead temporarily disable the extension before attempting to run the test until the underlying problem of hard coded SHA values is fixed. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-06-25t1700: make sure split-index respects core.sharedrepositoryLibravatar Christian Couder1-0/+30
Add a few tests to check that both the split-index file and the shared-index file are created using the right permissions when core.sharedrepository is set. Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-17Merge branch 'cc/split-index-config'Libravatar Junio C Hamano1-85/+239
The experimental "split index" feature has gained a few configuration variables to make it easier to use. * cc/split-index-config: (22 commits) Documentation/git-update-index: explain splitIndex.* Documentation/config: add splitIndex.sharedIndexExpire read-cache: use freshen_shared_index() in read_index_from() read-cache: refactor read_index_from() t1700: test shared index file expiration read-cache: unlink old sharedindex files config: add git_config_get_expiry() from gc.c read-cache: touch shared index files when used sha1_file: make check_and_freshen_file() non static Documentation/config: add splitIndex.maxPercentChange t1700: add tests for splitIndex.maxPercentChange read-cache: regenerate shared index if necessary config: add git_config_get_max_percent_split_change() Documentation/git-update-index: talk about core.splitIndex config var Documentation/config: add information for core.splitIndex t1700: add tests for core.splitIndex update-index: warn in case of split-index incoherency read-cache: add and then use tweak_split_index() split-index: add {add,remove}_split_index() functions config: add git_config_get_split_index() ...
2017-03-06read-cache: use freshen_shared_index() in read_index_from()Libravatar Christian Couder1-7/+7
This way a share index file will not be garbage collected if we still read from an index it is based from. As we need to read the current index before creating a new one, the tests have to be adjusted, so that we don't expect an old shared index file to be deleted right away when we create a new one. Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-06t1700: test shared index file expirationLibravatar Christian Couder1-0/+44
Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-01t1700: add tests for splitIndex.maxPercentChangeLibravatar Christian Couder1-0/+72
Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-01read-cache: regenerate shared index if necessaryLibravatar Christian Couder1-0/+1
When writing a new split-index and there is a big number of cache entries in the split-index compared to the shared index, it is a good idea to regenerate the shared index. By default when the ratio reaches 20%, we will push back all the entries from the split-index into a new shared index file instead of just creating a new split-index file. The threshold can be configured using the "splitIndex.maxPercentChange" config variable. We need to adjust the existing tests in t1700 by setting "splitIndex.maxPercentChange" to 100 at the beginning of t1700, as the existing tests are assuming that the shared index is regenerated only when `git update-index --split-index` is used. Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-01t1700: add tests for core.splitIndexLibravatar Christian Couder1-0/+37
Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-01t1700: change here document styleLibravatar Christian Couder1-85/+85
This improves test indentation by getting rid of the outdated here document style. Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-17rev-parse: fix several options when running in a subdirectoryLibravatar Johannes Schindelin1-1/+1
In addition to making git_path() aware of certain file names that need to be handled differently e.g. when running in worktrees, the commit 557bd833bb (git_path(): be aware of file relocation in $GIT_DIR, 2014-11-30) also snuck in a new option for `git rev-parse`: `--git-path`. On the face of it, there is no obvious bug in that commit's diff: it faithfully calls git_path() on the argument and prints it out, i.e. `git rev-parse --git-path <filename>` has the same precise behavior as calling `git_path("<filename>")` in C. The problem lies deeper, much deeper. In hindsight (which is always unfair), implementing the .git/ directory discovery in `setup_git_directory()` by changing the working directory may have allowed us to avoid passing around a struct that contains information about the current repository, but it bought us many, many problems. In this case, when being called in a subdirectory, `git rev-parse` changes the working directory to the top-level directory before calling `git_path()`. In the new working directory, the result is correct. But in the working directory of the calling script, it is incorrect. Example: when calling `git rev-parse --git-path HEAD` in, say, the Documentation/ subdirectory of Git's own source code, the string `.git/HEAD` is printed. Side note: that bug is hidden when running in a subdirectory of a worktree that was added by the `git worktree` command: in that case, the (correct) absolute path of the `HEAD` file is printed. In the interest of time, this patch does not go the "correct" route to introduce a struct with repository information (and removing global state in the process), instead this patch chooses to detect when the command was called in a subdirectory and forces the result to be an absolute path. While at it, we are also fixing the output of --git-common-dir and --shared-index-path. Lastly, please note that we reuse the same strbuf for all of the relative_path() calls; this avoids frequent allocation (and duplicated code), and it does not risk memory leaks, for two reasons: 1) the cmd_rev_parse() function does not return anywhere between the use of the new strbuf instance and its final release, and 2) git-rev-parse is one of these "one-shot" programs in Git, i.e. it exits after running for a very short time, meaning that all allocated memory is released with the exit() call anyway. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-17rev-parse tests: add tests executed from a subdirectoryLibravatar Michael Rappazzo1-0/+16
t2027-worktree-list has an incorrect expectation for --git-common-dir which has been adjusted and marked to expect failure. Some of the tests added have been marked to expect failure. These demonstrate a problem with the way that some options to git rev-parse behave when executed from a subdirectory of the main worktree. [jes: fixed incorrect assumption that objects/ lives in the worktree-specific git-dir (it lives in the common dir instead). Also adjusted t1700 so that the test case does not *need* to be the last one in that script.] Signed-off-by: Michael Rappazzo <rappazzo@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-18test-lib.sh: introduce and use $EMPTY_BLOBLibravatar Nguyễn Thái Ngọc Duy1-12/+12
Similar to $EMPTY_TREE this makes it easier to recognize this special SHA-1 and change hash later. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-27t/t1700-split-index.sh: use the $( ... ) construct for command substitutionLibravatar Elia Pinto1-1/+1
The Git CodingGuidelines prefer the $(...) construct for command substitution instead of using the backquotes `...`. The backquoted form is the traditional method for command substitution, and is supported by POSIX. However, all but the simplest uses become complicated quickly. In particular, embedded command substitutions and/or the use of double quotes require careful escaping with the backslash character. The patch was generated by: for _f in $(find . -name "*.sh") do perl -i -pe 'BEGIN{undef $/;} s/`(.+?)`/\$(\1)/smg' "${_f}" done and then carefully proof-read. Signed-off-by: Elia Pinto <gitter.spiros@gmail.com> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-26Merge branch 'jk/test-chain-lint'Libravatar Junio C Hamano1-1/+1
People often forget to chain the commands in their test together with &&, leaving a failure from an earlier command in the test go unnoticed. The new GIT_TEST_CHAIN_LINT mechanism allows you to catch such a mistake more easily. * jk/test-chain-lint: (36 commits) t9001: drop save_confirm helper t0020: use test_* helpers instead of hand-rolled messages t: simplify loop exit-code status variables t: fix some trivial cases of ignored exit codes in loops t7701: fix ignored exit code inside loop t3305: fix ignored exit code inside loop t0020: fix ignored exit code inside loops perf-lib: fix ignored exit code inside loop t6039: fix broken && chain t9158, t9161: fix broken &&-chain in git-svn tests t9104: fix test for following larger parents t4104: drop hand-rolled error reporting t0005: fix broken &&-chains t7004: fix embedded single-quotes t0050: appease --chain-lint t9001: use test_when_finished t4117: use modern test_* helpers t6034: use modern test_* helpers t1301: use modern test_* helpers t0020: use modern test_* helpers ...
2015-03-20t1700: make test pass with index-v4Libravatar Thomas Gummerer1-3/+12
The different index versions have different sha-1 checksums. Those checksums are checked in t1700, which makes it fail when the test suite is run with TEST_GIT_INDEX_VERSION=4. Fix it. Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-20t: fix severe &&-chain breakageLibravatar Jeff King1-1/+1
These are tests which are missing a link in their &&-chain, in a location which causes a significant portion of the test to be missed (e.g., the test effectively does nothing, or consists of a long string of actions and output comparisons, and we throw away the exit code of at least one part of the string). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-13t1700: new tests for split-index modeLibravatar Nguyễn Thái Ngọc Duy1-0/+194
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>