summaryrefslogtreecommitdiff
path: root/t/t5510-fetch.sh
AgeCommit message (Collapse)AuthorFilesLines
2021-01-25Merge branch 'jx/bundle'Libravatar Junio C Hamano1-22/+4
"git bundle" learns "--stdin" option to read its refs from the standard input. Also, it now does not lose refs whey they point at the same object. * jx/bundle: bundle: arguments can be read from stdin bundle: lost objects when removing duplicate pendings test: add helper functions for git-bundle
2021-01-25Merge branch 'ps/fetch-atomic'Libravatar Junio C Hamano1-0/+168
"git fetch" learns to treat ref updates atomically in all-or-none fashion, just like "git push" does, with the new "--atomic" option. * ps/fetch-atomic: fetch: implement support for atomic reference updates fetch: allow passing a transaction to `s_update_ref()` fetch: refactor `s_update_ref` to use common exit path fetch: use strbuf to format FETCH_HEAD updates fetch: extract writing to FETCH_HEAD
2021-01-25Merge branch 'js/default-branch-name-tests-final-stretch'Libravatar Junio C Hamano1-0/+3
Prepare tests not to be affected by the name of the default branch "git init" creates. * js/default-branch-name-tests-final-stretch: (28 commits) tests: drop prereq `PREPARE_FOR_MAIN_BRANCH` where no longer needed t99*: adjust the references to the default branch name "main" tests(git-p4): transition to the default branch name `main` t9[5-7]*: adjust the references to the default branch name "main" t9[0-4]*: adjust the references to the default branch name "main" t8*: adjust the references to the default branch name "main" t7[5-9]*: adjust the references to the default branch name "main" t7[0-4]*: adjust the references to the default branch name "main" t6[4-9]*: adjust the references to the default branch name "main" t64*: preemptively adjust alignment to prepare for `master` -> `main` t6[0-3]*: adjust the references to the default branch name "main" t5[6-9]*: adjust the references to the default branch name "main" t55[4-9]*: adjust the references to the default branch name "main" t55[23]*: adjust the references to the default branch name "main" t551*: adjust the references to the default branch name "main" t550*: adjust the references to the default branch name "main" t5503: prepare aligned comment for replacing `master` with `main` t5[0-4]*: adjust the references to the default branch name "main" t5323: prepare centered comment for `master` -> `main` t4*: adjust the references to the default branch name "main" ...
2021-01-12fetch: implement support for atomic reference updatesLibravatar Patrick Steinhardt1-0/+168
When executing a fetch, then git will currently allocate one reference transaction per reference update and directly commit it. This means that fetches are non-atomic: even if some of the reference updates fail, others may still succeed and modify local references. This is fine in many scenarios, but this strategy has its downsides. - The view of remote references may be inconsistent and may show a bastardized state of the remote repository. - Batching together updates may improve performance in certain scenarios. While the impact probably isn't as pronounced with loose references, the upcoming reftable backend may benefit as it needs to write less files in case the update is batched. - The reference-update hook is currently being executed twice per updated reference. While this doesn't matter when there is no such hook, we have seen severe performance regressions when doing a git-fetch(1) with reference-transaction hook when the remote repository has hundreds of thousands of references. Similar to `git push --atomic`, this commit thus introduces atomic fetches. Instead of allocating one reference transaction per updated reference, it causes us to only allocate a single transaction and commit it as soon as all updates were received. If locking of any reference fails, then we abort the complete transaction and don't update any reference, which gives us an all-or-nothing fetch. Note that this may not completely fix the first of above downsides, as the consistent view also depends on the server-side. If the server doesn't have a consistent view of its own references during the reference negotiation phase, then the client would get the same inconsistent view the server has. This is a separate problem though and, if it actually exists, can be fixed at a later point. This commit also changes the way we write FETCH_HEAD in case `--atomic` is passed. Instead of writing changes as we go, we need to accumulate all changes first and only commit them at the end when we know that all reference updates succeeded. Ideally, we'd just do so via a temporary file so that we don't need to carry all updates in-memory. This isn't trivially doable though considering the `--append` mode, where we do not truncate the file but simply append to it. And given that we support concurrent processes appending to FETCH_HEAD at the same time without any loss of data, seeding the temporary file with current contents of FETCH_HEAD initially and then doing a rename wouldn't work either. So this commit implements the simple strategy of buffering all changes and appending them to the file on commit. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-01-11test: add helper functions for git-bundleLibravatar Jiang Xin1-22/+4
Move git-bundle related functions from t5510 to a library, and this lib will be shared with a new testcase t6020 which finds a known breakage of "git-bundle". Signed-off-by: Jiang Xin <zhiyou.jx@alibaba-inc.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-12-16t5510: use `main` as initial branch nameLibravatar Johannes Schindelin1-83/+85
In 66713e84e71 (tests: prepare aligned mentions of the default branch name, 2020-10-23), we prepared this test script for a time when the default initial branch name would be `main`. However, there is no need to wait for that: let's adjust the test script to stop relying on a specific initial branch name by setting it explicitly. This allows us to drop the `PREPARE_FOR_MAIN_BRANCH` prereq from two test cases. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-11-19tests: drop prereq `PREPARE_FOR_MAIN_BRANCH` where no longer neededLibravatar Johannes Schindelin1-2/+2
We introduced the `PREPARE_FOR_MAIN_BRANCH` prereq for the sole purpose of allowing us to perform the non-trivial adjustments regarding the `master` -> `main` rename before the automatable ones. Now that the transition is almost complete, we can stop using it in most instances. The only two exceptions are t5526 and t9902: at the time of writing, there are other patches in flight that touch these test scripts, therefore their transition to `main` is postponed to a later date. This patch is the result of this command: sed -i 's/PREPARE_FOR_MAIN_BRANCH[ ,]//' t/t[0-9]*.sh && git checkout HEAD -- t/t5526\* t/t9902\* Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-11-19t551*: adjust the references to the default branch name "main"Libravatar Johannes Schindelin1-80/+80
This trick was performed via $ (cd t && sed -i -e 's/master/main/g' -e 's/MASTER/MAIN/g' \ -e 's/Master/Main/g' -- t551*.sh) This allows us to define `GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main` for those tests. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-11-19tests: mark tests relying on the current default for `init.defaultBranch`Libravatar Johannes Schindelin1-0/+3
In addition to the manual adjustment to let the `linux-gcc` CI job run the test suite with `master` and then with `main`, this patch makes sure that GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME is set in all test scripts that currently rely on the initial branch name being `master by default. To determine which test scripts to mark up, the first step was to force-set the default branch name to `master` in - all test scripts that contain the keyword `master`, - t4211, which expects `t/t4211/history.export` with a hard-coded ref to initialize the default branch, - t5560 because it sources `t/t556x_common` which uses `master`, - t8002 and t8012 because both source `t/annotate-tests.sh` which also uses `master`) This trick was performed by this command: $ sed -i '/^ *\. \.\/\(test-lib\|lib-\(bash\|cvs\|git-svn\)\|gitweb-lib\)\.sh$/i\ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\ ' $(git grep -l master t/t[0-9]*.sh) \ t/t4211*.sh t/t5560*.sh t/t8002*.sh t/t8012*.sh After that, careful, manual inspection revealed that some of the test scripts containing the needle `master` do not actually rely on a specific default branch name: either they mention `master` only in a comment, or they initialize that branch specificially, or they do not actually refer to the current default branch. Therefore, the aforementioned modification was undone in those test scripts thusly: $ git checkout HEAD -- \ t/t0027-auto-crlf.sh t/t0060-path-utils.sh \ t/t1011-read-tree-sparse-checkout.sh \ t/t1305-config-include.sh t/t1309-early-config.sh \ t/t1402-check-ref-format.sh t/t1450-fsck.sh \ t/t2024-checkout-dwim.sh \ t/t2106-update-index-assume-unchanged.sh \ t/t3040-subprojects-basic.sh t/t3301-notes.sh \ t/t3308-notes-merge.sh t/t3423-rebase-reword.sh \ t/t3436-rebase-more-options.sh \ t/t4015-diff-whitespace.sh t/t4257-am-interactive.sh \ t/t5323-pack-redundant.sh t/t5401-update-hooks.sh \ t/t5511-refspec.sh t/t5526-fetch-submodules.sh \ t/t5529-push-errors.sh t/t5530-upload-pack-error.sh \ t/t5548-push-porcelain.sh \ t/t5552-skipping-fetch-negotiator.sh \ t/t5572-pull-submodule.sh t/t5608-clone-2gb.sh \ t/t5614-clone-submodules-shallow.sh \ t/t7508-status.sh t/t7606-merge-custom.sh \ t/t9302-fast-import-unpack-limit.sh We excluded one set of test scripts in these commands, though: the range of `git p4` tests. The reason? `git p4` stores the (foreign) remote branch in the branch called `p4/master`, which is obviously not the default branch. Manual analysis revealed that only five of these tests actually require a specific default branch name to pass; They were modified thusly: $ sed -i '/^ *\. \.\/lib-git-p4\.sh$/i\ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\ ' t/t980[0167]*.sh t/t9811*.sh Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-10-23tests: prepare aligned mentions of the default branch nameLibravatar Johannes Schindelin1-4/+4
In some tests, the default branch name is part of aligned output. As we want to change the default branch name to `main`, which is two characters shorter than the old default branch name, we will have to adjust those tests. Since we use the original default branch name until the entire test suite has been adjusted accordingly, the touched test cases need to be guarded by a prereq (that is so far disabled so that they are skipped for now). The test cases that depend on those test cases that are newly guarded by that prereq naturally have to be guarded, too. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-25Merge branch 'ds/maintenance-part-1'Libravatar Junio C Hamano1-1/+1
A "git gc"'s big brother has been introduced to take care of more repository maintenance tasks, not limited to the object database cleaning. * ds/maintenance-part-1: maintenance: add trace2 regions for task execution maintenance: add auto condition for commit-graph task maintenance: use pointers to check --auto maintenance: create maintenance.<task>.enabled config maintenance: take a lock on the objects directory maintenance: add --task option maintenance: add commit-graph task maintenance: initialize task array maintenance: replace run_auto_gc() maintenance: add --quiet option maintenance: create basic maintenance runner
2020-09-17maintenance: replace run_auto_gc()Libravatar Derrick Stolee1-1/+1
The run_auto_gc() method is used in several places to trigger a check for repo maintenance after some Git commands, such as 'git commit' or 'git fetch'. To allow for extra customization of this maintenance activity, replace the 'git gc --auto [--quiet]' call with one to 'git maintenance run --auto [--quiet]'. As we extend the maintenance builtin with other steps, users will be able to select different maintenance activities. Rename run_auto_gc() to run_auto_maintenance() to be clearer what is happening on this call, and to expose all callers in the current diff. Rewrite the method to use a struct child_process to simplify the calls slightly. Since 'git fetch' already allows disabling the 'git gc --auto' subprocess, add an equivalent option with a different name to be more descriptive of the new behavior: '--[no-]maintenance'. Update the documentation to include these options at the same time. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-02fetch: no FETCH_HEAD display if --no-write-fetch-headLibravatar Jonathan Tan1-8/+10
887952b8c6 ("fetch: optionally allow disabling FETCH_HEAD update", 2020-08-18) introduced the ability to disable writing to FETCH_HEAD during fetch, but did not suppress the "<source> -> FETCH_HEAD" message when this ability is used. This message is misleading in this case, because FETCH_HEAD is not written. Also, because "fetch" is used to lazy-fetch missing objects in a partial clone, this significantly clutters up the output in that case since the objects to be fetched are potentially numerous. Therefore, suppress this message when --no-write-fetch-head is passed (but not when --dry-run is set). Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-08-18fetch: optionally allow disabling FETCH_HEAD updateLibravatar Junio C Hamano1-2/+13
If you run fetch but record the result in remote-tracking branches, and either if you do nothing with the fetched refs (e.g. you are merely mirroring) or if you always work from the remote-tracking refs (e.g. you fetch and then merge origin/branchname separately), you can get away with having no FETCH_HEAD at all. Teach "git fetch" a command line option "--[no-]write-fetch-head". The default is to write FETCH_HEAD, and the option is primarily meant to be used with the "--no-" prefix to override this default, because there is no matching fetch.writeFetchHEAD configuration variable to flip the default to off (in which case, the positive form may become necessary to defeat it). Note that under "--dry-run" mode, FETCH_HEAD is never written; otherwise you'd see list of objects in the file that you do not actually have. Passing `--write-fetch-head` does not force `git fetch` to write the file. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-08-13Merge branch 'ma/test-quote-cleanup'Libravatar Junio C Hamano1-2/+2
Test cleanup. * ma/test-quote-cleanup: t4104: modernize and simplify quoting t: don't spuriously close and reopen quotes
2020-08-11Merge branch 'bc/sha-256-part-3'Libravatar Junio C Hamano1-6/+10
The final leg of SHA-256 transition. * bc/sha-256-part-3: (39 commits) t: remove test_oid_init in tests docs: add documentation for extensions.objectFormat ci: run tests with SHA-256 t: make SHA1 prerequisite depend on default hash t: allow testing different hash algorithms via environment t: add test_oid option to select hash algorithm repository: enable SHA-256 support by default setup: add support for reading extensions.objectformat bundle: add new version for use with SHA-256 builtin/verify-pack: implement an --object-format option http-fetch: set up git directory before parsing pack hashes t0410: mark test with SHA1 prerequisite t5308: make test work with SHA-256 t9700: make hash size independent t9500: ensure that algorithm info is preserved in config t9350: make hash size independent t9301: make hash size independent t9300: use $ZERO_OID instead of hard-coded object ID t9300: abstract away SHA-1-specific constants t8011: make hash size independent ...
2020-08-06t: don't spuriously close and reopen quotesLibravatar Martin Ågren1-2/+2
In the test scripts, the recommended style is, e.g.: test_expect_success 'name' ' do-something somehow && do-some-more testing ' When using this style, any single quote in the multi-line test section is actually closing the lone single quotes that surround it. It can be a non-issue in practice: test_expect_success 'sed a little' ' sed -e 's/hi/lo/' in >out # "ok": no whitespace in s/hi/lo/ ' Or it can be a bug in the test, e.g., because variable interpolation happens before the test even begins executing: v=abc test_expect_success 'variable interpolation' ' v=def && echo '"$v"' # abc ' Change several such in-test single quotes to use double quotes instead or, in a few cases, drop them altogether. These were identified using some crude grepping. We're not fixing any test bugs here, but we're hopefully making these tests slightly easier to grok and to maintain. There are legitimate use cases for closing a quote and opening a new one, e.g., both '\'' and '"'"' can be used to produce a literal single quote. I'm not touching any of those here. In t9401, tuck the redirecting ">" to the filename while we're touching those lines. Signed-off-by: Martin Ågren <martin.agren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-30bundle: add new version for use with SHA-256Libravatar brian m. carlson1-6/+10
Currently we detect the hash algorithm in use by the length of the object ID. This is inelegant and prevents us from using a different hash algorithm that is also 256 bits in length. Since we cannot extend the v2 format in a backward-compatible way, let's add a v3 format, which is identical, except for the addition of capabilities, which are prefixed by an at sign. We add "object-format" as the only capability and reject unknown capabilities, since we do not have a network connection and therefore cannot negotiate with the other side. For compatibility, default to the v2 format for SHA-1 and require v3 for SHA-256. In t5510, always use format v3 so we can be sure we produce consistent results across hash algorithms. Since head -n N lists the top N lines instead of the Nth line, let's run our output through sed to normalize it and compare it against a fixed value, which will make sure we get exactly what we're expecting. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Reviewed-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-28Remove doubled words in various commentsLibravatar Elijah Newren1-1/+1
Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-03-02Merge branch 'ma/test-cleanup'Libravatar Junio C Hamano1-1/+0
Code cleanup. * ma/test-cleanup: t: drop debug `cat` calls t9810: drop debug `cat` call t4117: check for files using `test_path_is_file`
2020-02-24t: drop debug `cat` callsLibravatar Martin Ågren1-1/+0
We `cat` files, but don't inspect or grab the contents in any way. Unlike in an earlier commit, there is no reason to suspect that these files could be missing, so `cat`-ing them is just wasted effort. Signed-off-by: Martin Ågren <martin.agren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-05Merge branch 'bc/hash-independent-tests-part-7'Libravatar Junio C Hamano1-4/+5
Preparation of test scripts for the day when the object names will use SHA-256 continues. * bc/hash-independent-tests-part-7: t5604: make hash independent t5601: switch into repository to hash object t5562: use $ZERO_OID t5540: make hash size independent t5537: make hash size independent t5530: compute results based on object length t5512: abstract away SHA-1-specific constants t5510: make hash size independent t5504: make hash algorithm independent t5324: make hash size independent t5319: make test work with SHA-256 t5319: change invalid offset for SHA-256 compatibility t5318: update for SHA-256 t4300: abstract away SHA-1-specific constants t4204: make hash size independent t4202: abstract away SHA-1-specific constants t4200: make hash size independent t4134: compute appropriate length constant t4066: compute index line in diffs t4054: make hash-size independent
2020-01-21fetch: document and test --refmap=""Libravatar Derrick Stolee1-0/+24
To prevent long blocking time during a 'git fetch' call, a user may want to set up a schedule for background 'git fetch' processes. However, these runs will update the refs/remotes branches due to the default refspec set in the config when Git adds a remote. Hence the user will not notice when remote refs are updated during their foreground fetches. In fact, they may _want_ those refs to stay put so they can work with the refs from their last foreground fetch call. This can be accomplished by overriding the configured refspec using '--refmap=' along with a custom refspec: git fetch --refmap='' <remote> +refs/heads/*:refs/hidden/<remote>/* to populate a custom ref space and download a pack of the new reachable objects. This kind of call allows a few things to happen: 1. We download a new pack if refs have updated. 2. Since the refs/hidden branches exist, GC will not remove the newly-downloaded data. 3. With fetch.writeCommitGraph enabled, the refs/hidden refs are used to update the commit-graph file. To avoid the refs/hidden directory from filling without bound, the --prune option can be included. When providing a refspec like this, the --prune option does not delete remote refs and instead only deletes refs in the target refspace. Update the documentation to clarify how '--refmap=""' works and create tests to guarantee this behavior remains in the future. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-15t5510: make hash size independentLibravatar brian m. carlson1-4/+5
Use $OID_REGEX instead of hard-coding 40-based regular expressions. Change invocations of cut with a hard-coded constant to split using a delimiter instead. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-04Merge branch 'ds/commit-graph-on-fetch'Libravatar Junio C Hamano1-0/+16
Regression fix. * ds/commit-graph-on-fetch: commit-graph: fix writing first commit-graph during fetch t5510-fetch.sh: demonstrate fetch.writeCommitGraph bug
2019-10-25commit-graph: fix writing first commit-graph during fetchLibravatar Derrick Stolee1-1/+1
The previous commit includes a failing test for an issue around fetch.writeCommitGraph and fetching in a repo with a submodule. Here, we fix that bug and set the test to "test_expect_success". The problem arises with this set of commands when the remote repo at <url> has a submodule. Note that --recurse-submodules is not needed to demonstrate the bug. $ git clone <url> test $ cd test $ git -c fetch.writeCommitGraph=true fetch origin Computing commit graph generation numbers: 100% (12/12), done. BUG: commit-graph.c:886: missing parent <hash1> for commit <hash2> Aborted (core dumped) As an initial fix, I converted the code in builtin/fetch.c that calls write_commit_graph_reachable() to instead launch a "git commit-graph write --reachable --split" process. That code worked, but is not how we want the feature to work long-term. That test did demonstrate that the issue must be something to do with internal state of the 'git fetch' process. The write_commit_graph() method in commit-graph.c ensures the commits we plan to write are "closed under reachability" using close_reachable(). This method walks from the input commits, and uses the UNINTERESTING flag to mark which commits have already been visited. This allows the walk to take O(N) time, where N is the number of commits, instead of O(P) time, where P is the number of paths. (The number of paths can be exponential in the number of commits.) However, the UNINTERESTING flag is used in lots of places in the codebase. This flag usually means some barrier to stop a commit walk, such as in revision-walking to compare histories. It is not often cleared after the walk completes because the starting points of those walks do not have the UNINTERESTING flag, and clear_commit_marks() would stop immediately. This is happening during a 'git fetch' call with a remote. The fetch negotiation is comparing the remote refs with the local refs and marking some commits as UNINTERESTING. I tested running clear_commit_marks_many() to clear the UNINTERESTING flag inside close_reachable(), but the tips did not have the flag, so that did nothing. It turns out that the calculate_changed_submodule_paths() method is at fault. Thanks, Peff, for pointing out this detail! More specifically, for each submodule, the collect_changed_submodules() runs a revision walk to essentially do file-history on the list of submodules. That revision walk marks commits UNININTERESTING if they are simplified away by not changing the submodule. Instead, I finally arrived on the conclusion that I should use a flag that is not used in any other part of the code. In commit-reach.c, a number of flags were defined for commit walk algorithms. The REACHABLE flag seemed like it made the most sense, and it seems it was not actually used in the file. The REACHABLE flag was used in early versions of commit-reach.c, but was removed by 4fbcca4 (commit-reach: make can_all_from_reach... linear, 2018-07-20). Add the REACHABLE flag to commit-graph.c and use it instead of UNINTERESTING in close_reachable(). This fixes the bug in manual testing. Reported-by: Johannes Schindelin <johannes.schindelin@gmx.de> Helped-by: Jeff King <peff@peff.net> Helped-by: Szeder Gábor <szeder.dev@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-25t5510-fetch.sh: demonstrate fetch.writeCommitGraph bugLibravatar Derrick Stolee1-0/+16
While dogfooding, Johannes found a bug in the fetch.writeCommitGraph config behavior. His example initially happened during a clone with --recurse-submodules, we found that this happens with the first fetch after cloning a repository that contains a submodule: $ git clone <url> test $ cd test $ git -c fetch.writeCommitGraph=true fetch origin Computing commit graph generation numbers: 100% (12/12), done. BUG: commit-graph.c:886: missing parent <hash1> for commit <hash2> Aborted (core dumped) In the repo I had cloned, there were really 60 commits to scan, but only 12 were in the list to write when calling compute_generation_numbers(). A commit in the list expects to see a parent, but that parent is not in the list. A follow-up will fix the bug, but first we create a test that demonstrates the problem. This test must be careful about an existing commit-graph file, since GIT_TEST_COMMIT_GRAPH=1 will cause the repo we are cloning to already have one. This then prevents the incremtnal commit-graph write during the first 'git fetch'. Helped-by: Jeff King <peff@peff.net> Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de> Helped-by: Szeder Gábor <szeder.dev@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-09-30Merge branch 'ds/commit-graph-on-fetch'Libravatar Junio C Hamano1-0/+13
A configuration variable tells "git fetch" to write the commit graph after finishing. * ds/commit-graph-on-fetch: fetch: add fetch.writeCommitGraph config setting
2019-09-03fetch: add fetch.writeCommitGraph config settingLibravatar Derrick Stolee1-0/+13
The commit-graph feature is now on by default, and is being written during 'git gc' by default. Typically, Git only writes a commit-graph when a 'git gc --auto' command passes the gc.auto setting to actualy do work. This means that a commit-graph will typically fall behind the commits that are being used every day. To stay updated with the latest commits, add a step to 'git fetch' to write a commit-graph after fetching new objects. The fetch.writeCommitGraph config setting enables writing a split commit-graph, so on average the cost of writing this file is very small. Occasionally, the commit-graph chain will collapse to a single level, and this could be slow for very large repos. For additional use, adjust the default to be true when feature.experimental is enabled. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-08-01t5510-fetch: run non-httpd-specific test before sourcing 'lib-httpd.sh'Libravatar SZEDER Gábor1-22/+25
't5510-fetch.sh' sources 'lib-httpd.sh' near the end to run a httpd-specific test, but 'lib-httpd.sh' skips all the rest of the test script if the dependencies for running httpd tests are not fulfilled. Alas, recently cdbd70c437 (fetch: add --[no-]show-forced-updates argument, 2019-06-18) appended a non-httpd-specific test at the end, and this test is then skipped as well when httpd tests can't be run. Move this new test earlier in the test script, before 'lib-httpd.sh' is sourced, so it will be run even when httpd tests aren't. Also add a comment at the end of this test script to warn against adding non-httpd-specific tests at the end, in the hope that it will help prevent similar issues in the future. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-07-31t5510-fetch: fix negated 'test_i18ngrep' invocationLibravatar SZEDER Gábor1-1/+1
The test '--no-show-forced-updates' in 't5510-fetch.sh' added in cdbd70c437 (fetch: add --[no-]show-forced-updates argument, 2019-06-18) runs '! test_i18ngrep ...'. This is wrong, because when running the test with GIT_TEST_GETTEXT_POISON=true, then 'test_i18ngrep' is basically a noop and always returns with success, the leading ! turns that into a failure, which then fails the test. Use 'test_i18ngrep ! ...' instead. This went unnoticed by our GETTEXT_POISON CI builds, because those builds don't run this test case: in those builds we don't install Apache, and this test comes after 't5510' sources 'lib-httpd.sh', which, consequently, skips all the remaining tests, including this one. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-21fetch: add --[no-]show-forced-updates argumentLibravatar Derrick Stolee1-0/+23
After updating a set of remove refs during a 'git fetch', we walk the commits in the new ref value and not in the old ref value to discover if the update was a forced update. This results in two things happening during the command: 1. The line including the ref update has an additional "(forced-update)" marker at the end. 2. The ref log for that remote branch includes a bit saying that update is a forced update. For many situations, this forced-update message happens infrequently, or is a small bit of information among many ref updates. Many users ignore these messages, but the calculation required here slows down their fetches significantly. Keep in mind that they do not have the opportunity to calculate a commit-graph file containing the newly-fetched commits, so these comparisons can be very slow. Add a '--[no-]show-forced-updates' option that allows a user to skip this calculation. The only permanent result is dropping the forced-update bit in the reflog. Include a new fetch.showForcedUpdates config setting that allows this behavior without including the argument in every command. The config setting is overridden by the command-line arguments. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-14tests: use 'test_atexit' to stop httpdLibravatar SZEDER Gábor1-2/+0
Use 'test_atexit' to run cleanup commands to stop httpd at the end of the test script or upon interrupt or failure, as it is shorter, simpler, and more robust than registering such cleanup commands in the trap on EXIT in the test scripts. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-08-20Merge branch 'ab/fetch-tags-noclobber'Libravatar Junio C Hamano1-1/+1
Test and doc clean-ups. * ab/fetch-tags-noclobber: pull doc: fix a long-standing grammar error fetch tests: correct a comment "remove it" -> "remove them" push tests: assert re-pushing annotated tags push tests: add more testing for forced tag pushing push tests: fix logic error in "push" test assertion push tests: remove redundant 'git push' invocation fetch tests: change "Tag" test tag to "testTag"
2018-08-17Merge branch 'jt/refspec-dwim-precedence-fix'Libravatar Junio C Hamano1-0/+35
"git fetch $there refs/heads/s" ought to fetch the tip of the branch 's', but when "refs/heads/refs/heads/s", i.e. a branch whose name is "refs/heads/s" exists at the same time, fetched that one instead by mistake. This has been corrected to honor the usual disambiguation rules for abbreviated refnames. * jt/refspec-dwim-precedence-fix: remote: make refspec follow the same disambiguation rule as local refs
2018-08-13fetch tests: correct a comment "remove it" -> "remove them"Libravatar Ævar Arnfjörð Bjarmason1-1/+1
Correct a comment referring to the removal of just the branch to also refer to the tag. This should have been changed in my ca3065e7e7 ("fetch tests: add a tag to be deleted to the pruning tests", 2018-02-09) when the tag deletion was added, but I missed it at the time. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-08-02Merge branch 'kg/gc-auto-windows-workaround'Libravatar Junio C Hamano1-0/+2
"git gc --auto" opens file descriptors for the packfiles before spawning "git repack/prune", which would upset Windows that does not want a process to work on a file that is open by another process. The issue has been worked around. * kg/gc-auto-windows-workaround: gc --auto: release pack files before auto packing
2018-08-02remote: make refspec follow the same disambiguation rule as local refsLibravatar Junio C Hamano1-0/+35
When matching a non-wildcard LHS of a refspec against a list of refs, find_ref_by_name_abbrev() returns the first ref that matches using any DWIM rules used by refname_match() in refs.c, even if a better match occurs later in the list of refs. This causes unexpected behavior when (for example) fetching using the refspec "refs/heads/s:<something>" from a remote with both "refs/heads/refs/heads/s" and "refs/heads/s"; even if the former was inadvertently created, one would still expect the latter to be fetched. Similarly, when both a tag T and a branch T exist, fetching T should favor the tag, just like how local refname disambiguation rule works. But because the code walks over ls-remote output from the remote, which happens to be sorted in alphabetical order and has refs/heads/T before refs/tags/T, a request to fetch T is (mis)interpreted as fetching refs/heads/T. Update refname_match(), all of whose current callers care only if it returns non-zero (i.e. matches) to see if an abbreviated name can mean the full name being tested, so that it returns a positive integer whose magnitude can be used to tell the precedence, and fix the find_ref_by_name_abbrev() function not to stop at the first match but find the match with the highest precedence. This is based on an earlier work, which special cased only the exact matches, by Jonathan Tan. Helped-by: Jonathan Tan <jonathantanmy@google.com> Helped-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-09gc --auto: release pack files before auto packingLibravatar Kim Gybels1-0/+2
Teach gc --auto to release pack files before auto packing the repository to prevent failures when removing them. Also teach the test 'fetching with auto-gc does not lock up' to complain when it is no longer triggering an auto packing of the repository. Fixes https://github.com/git-for-windows/git/issues/500 Signed-off-by: Kim Gybels <kgybels@infogroep.be> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-03fetch-pack: support negotiation tip whitelistLibravatar Jonathan Tan1-0/+78
During negotiation, fetch-pack eventually reports as "have" lines all commits reachable from all refs. Allow the user to restrict the commits sent in this way by providing a whitelist of tips; only the tips themselves and their ancestors will be sent. Both globs and single objects are supported. This feature is only supported for protocols that support connect or stateless-connect (such as HTTP with protocol v2). This will speed up negotiation when the repository has multiple relatively independent branches (for example, when a repository interacts with multiple repositories, such as with linux-next [1] and torvalds/linux [2]), and the user knows which local branch is likely to have commits in common with the upstream branch they are fetching. [1] https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next/ [2] https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux/ Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-06-01Merge branch 'cc/tests-without-assuming-ref-files-backend'Libravatar Junio C Hamano1-3/+3
Quite a many tests assumed that newly created refs are made as loose refs using the files backend, which have been updated to use proper plumbing like rev-parse and update-ref, to avoid breakage once we start using different ref backends. * cc/tests-without-assuming-ref-files-backend: t990X: use '.git/objects' as 'deep inside .git' path t: make many tests depend less on the refs being files
2018-05-23t: make many tests depend less on the refs being filesLibravatar David Turner1-3/+3
Many tests are very focused on the file system representation of the loose and packed refs code. As there are plans to implement other ref storage systems, let's migrate these tests to a form that test the intent of the refs storage system instead of it internals. This will make clear to readers that these tests do not depend on which ref backend is used. The internals of the loose refs backend are still tested in t1400-update-ref.sh, whereas the tests changed in this patch focus on testing other aspects. This patch just takes care of many low hanging fruits. It does not try to completely solves the issue. Helped-by: Stefan Beller <sbeller@google.com> Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-04-11Merge branch 'pk/test-avoid-pipe-hiding-exit-status'Libravatar Junio C Hamano1-4/+4
Test cleanup. * pk/test-avoid-pipe-hiding-exit-status: test: avoid pipes in git related commands for test
2018-03-28test: avoid pipes in git related commands for testLibravatar Pratik Karki1-4/+4
Avoid using pipes downstream of Git commands since the exit codes of commands upstream of pipes get swallowed, thus potentially hiding failure of those commands. Instead, capture Git command output to a file and apply the downstream command(s) to that file. Signed-off-by: Pratik Karki <predatoramigo@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-06Merge branch 'ab/fetch-prune'Libravatar Junio C Hamano1-44/+194
Clarify how configured fetch refspecs interact with the "--prune" option of "git fetch", and also add a handy short-hand for getting rid of stale tags that are locally held. * ab/fetch-prune: fetch: make the --prune-tags work with <url> fetch: add a --prune-tags option and fetch.pruneTags config fetch tests: add scaffolding for the new fetch.pruneTags git-fetch & config doc: link to the new PRUNING section git remote doc: correct dangerous lies about what prune does git fetch doc: add a new section to explain the ins & outs of pruning fetch tests: fetch <url> <spec> as well as fetch [<remote>] fetch tests: expand case/esac for later change fetch tests: double quote a variable for interpolation fetch tests: test --prune and refspec interaction fetch tests: add a tag to be deleted to the pruning tests fetch tests: re-arrange arguments for future readability fetch tests: refactor in preparation for testing tag pruning remote: add a macro for "refs/tags/*:refs/tags/*" fetch: stop accessing "remote" variable indirectly fetch: trivially refactor assignment to ref_nr fetch: don't redundantly NULL something calloc() gave us
2018-02-09fetch: make the --prune-tags work with <url>Libravatar Ævar Arnfjörð Bjarmason1-9/+7
Make the new --prune-tags option work properly when git-fetch is invoked with a <url> parameter instead of a <remote name> parameter. This change is split off from the introduction of --prune-tags due to the relative complexity of munging the incoming argv, which is easier to review as a separate change. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-02-09fetch: add a --prune-tags option and fetch.pruneTags configLibravatar Ævar Arnfjörð Bjarmason1-0/+70
Add a --prune-tags option to git-fetch, along with fetch.pruneTags config option and a -P shorthand (-p is --prune). This allows for doing any of: git fetch -p -P git fetch --prune --prune-tags git fetch -p -P origin git fetch --prune --prune-tags origin Or simply: git config fetch.prune true && git config fetch.pruneTags true && git fetch Instead of the much more verbose: git fetch --prune origin 'refs/tags/*:refs/tags/*' '+refs/heads/*:refs/remotes/origin/*' Before this feature it was painful to support the use-case of pulling from a repo which is having both its branches *and* tags deleted regularly, and have our local references to reflect upstream. At work we create deployment tags in the repo for each rollout, and there's *lots* of those, so they're archived within weeks for performance reasons. Without this change it's hard to centrally configure such repos in /etc/gitconfig (on servers that are only used for working with them). You need to set fetch.prune=true globally, and then for each repo: git -C {} config --replace-all remote.origin.fetch "refs/tags/*:refs/tags/*" "^\+*refs/tags/\*:refs/tags/\*$" Now I can simply set fetch.pruneTags=true in /etc/gitconfig as well, and users running "git pull" will automatically get the pruning semantics I want. Even though "git remote" has corresponding "prune" and "update --prune" subcommands I'm intentionally not adding a corresponding prune-tags or "update --prune --prune-tags" mode to that command. It's advertised (as noted in my recent "git remote doc: correct dangerous lies about what prune does") as only modifying remote tracking references, whereas any --prune-tags option is always going to modify what from the user's perspective is a local copy of the tag, since there's no such thing as a remote tracking tag. Ideally add_prune_tags_to_fetch_refspec() would be something that would use ALLOC_GROW() to grow the 'fetch` member of the 'remote' struct. Instead I'm realloc-ing remote->fetch and adding the tag_refspec to the end. The reason is that parse_{fetch,push}_refspec which allocate the refspec (ultimately remote->fetch) struct are called many places that don't have access to a 'remote' struct. It would be hard to change all their callsites to be amenable to carry around the bookkeeping variables required for dynamic allocation. All the other callers of the API first incrementally construct the string version of the refspec in remote->fetch_refspec via add_fetch_refspec(), before finally calling parse_fetch_refspec() via some variation of remote_get(). It's less of a pain to deal with the one special case that needs to modify already constructed refspecs than to chase down and change all the other callsites. The API I'm adding is intentionally not generalized because if we add more of these we'd probably want to re-visit how this is done. See my "Re: [BUG] git remote prune removes local tags, depending on fetch config" (87po6ahx87.fsf@evledraar.gmail.com; https://public-inbox.org/git/87po6ahx87.fsf@evledraar.gmail.com/) for more background info. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-02-09fetch tests: add scaffolding for the new fetch.pruneTagsLibravatar Ævar Arnfjörð Bjarmason1-42/+50
The fetch.pruneTags configuration doesn't exist yet, but will be added in a subsequent commit. Since testing for it requires adding new parameters to the test_configured_prune function it's easier to review this patch first to assert that no functional changes are introduced yet. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-02-09fetch tests: fetch <url> <spec> as well as fetch [<remote>]Libravatar Ævar Arnfjörð Bjarmason1-4/+40
When a remote URL is supplied on the command-line the internals of the fetch are different, in particular the code in get_ref_map(). An earlier version of the subsequent fetch.pruneTags patch hid a segfault because the difference wasn't tested for. Now all the tests are run as both of the variants of: git fetch git -c [...] fetch $(git config remote.origin.url) $(git config remote.origin.fetch) I'm using -c because while the [fetch] config just set by set_config_tristate will be picked up, the remote.origin.* config won't override it as intended. Work around that and turn this into a purely command-line test by always setting the variables on the command-line, and translate any setting of remote.origin.X into fetch.X. The reason for choosing the names "name" and "link" as opposed to e.g. "named" and "url" is because they're the same length, which makes the test output easier to read as it will be aligned. Due to shellscript quoting madness it's not worthwhile to do all of this within a test_expect_success, but do the parts that can easily be done there, including the one-time setting of variables that don't change between runs to be used by subsequent runs in the 'prune_type setup' test. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-02-09fetch tests: expand case/esac for later changeLibravatar Ævar Arnfjörð Bjarmason1-2/+6
Expand a compact case/esac statement for a later change that'll add more logic to the body of the "*" case. This is a whitespace-only change. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>