summaryrefslogtreecommitdiff
path: root/builtin/fetch.c
AgeCommit message (Collapse)AuthorFilesLines
2022-04-04Merge branch 'rc/fetch-refetch'Libravatar Junio C Hamano1-2/+32
"git fetch --refetch" learned to fetch everything without telling the other side what we already have, which is useful when you cannot trust what you have in the local object store. * rc/fetch-refetch: docs: mention --refetch fetch option fetch: after refetch, encourage auto gc repacking t5615-partial-clone: add test for fetch --refetch fetch: add --refetch option builtin/fetch-pack: add --refetch option fetch-pack: add refetch fetch-negotiator: add specific noop initializer
2022-03-28fetch: after refetch, encourage auto gc repackingLibravatar Robert Coup1-1/+18
After invoking `fetch --refetch`, the object db will likely contain many duplicate objects. If auto-maintenance is enabled, invoke it with appropriate settings to encourage repacking/consolidation. * gc.autoPackLimit: unless this is set to 0 (disabled), override the value to 1 to force pack consolidation. * maintenance.incremental-repack.auto: unless this is set to 0, override the value to -1 to force incremental repacking. Signed-off-by: Robert Coup <robert@coup.net.nz> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-28fetch: add --refetch optionLibravatar Robert Coup1-1/+14
Teach fetch and transports the --refetch option to force a full fetch without negotiating common commits with the remote. Use when applying a new partial clone filter to refetch all matching objects. Signed-off-by: Robert Coup <robert@coup.net.nz> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-25Merge branch 'gc/recursive-fetch-with-unused-submodules'Libravatar Junio C Hamano1-7/+7
When "git fetch --recurse-submodules" grabbed submodule commits that would be needed to recursively check out newly fetched commits in the superproject, it only paid attention to submodules that are in the current checkout of the superproject. We now do so for all submodules that have been run "git submodule init" on. * gc/recursive-fetch-with-unused-submodules: submodule: fix latent check_has_commit() bug fetch: fetch unpopulated, changed submodules submodule: move logic into fetch_task_create() submodule: extract get_fetch_task() submodule: store new submodule commits oid_array in a struct submodule: inline submodule_commits() into caller submodule: make static functions read submodules from commits t5526: create superproject commits with test helper t5526: stop asserting on stderr literally t5526: introduce test helper to assert on fetches
2022-03-16Merge branch 'ps/fetch-mirror-optim'Libravatar Junio C Hamano1-15/+27
Various optimization for "git fetch". * ps/fetch-mirror-optim: refs/files-backend: optimize reading of symbolic refs remote: read symbolic refs via `refs_read_symbolic_ref()` refs: add ability for backends to special-case reading of symbolic refs fetch: avoid lookup of commits when not appending to FETCH_HEAD upload-pack: look up "want" lines via commit-graph
2022-03-16fetch: fetch unpopulated, changed submodulesLibravatar Glen Choo1-7/+7
"git fetch --recurse-submodules" only considers populated submodules (i.e. submodules that can be found by iterating the index), which makes "git fetch" behave differently based on which commit is checked out. As a result, even if the user has initialized all submodules correctly, they may not fetch the necessary submodule commits, and commands like "git checkout --recurse-submodules" might fail. Teach "git fetch" to fetch cloned, changed submodules regardless of whether they are populated. This is in addition to the current behavior of fetching populated submodules (which is always attempted regardless of what was fetched in the superproject, or even if nothing was fetched in the superproject). A submodule may be encountered multiple times (via the list of populated submodules or via the list of changed submodules). When this happens, "git fetch" only reads the 'populated copy' and ignores the 'changed copy'. Amend the verify_fetch_result() test helper so that we can assert on which 'copy' is being read. Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-13Merge branch 'ps/fetch-atomic'Libravatar Junio C Hamano1-61/+129
"git fetch" can make two separate fetches, but ref updates coming from them were in two separate ref transactions under "--atomic", which has been corrected. * ps/fetch-atomic: fetch: make `--atomic` flag cover pruning of refs fetch: make `--atomic` flag cover backfilling of tags refs: add interface to iterate over queued transactional updates fetch: report errors when backfilling tags fails fetch: control lifecycle of FETCH_HEAD in a single place fetch: backfill tags before setting upstream fetch: increase test coverage of fetches
2022-03-01fetch: avoid lookup of commits when not appending to FETCH_HEADLibravatar Patrick Steinhardt1-15/+27
When fetching from a remote repository we will by default write what has been fetched into the special FETCH_HEAD reference. The order in which references are written depends on whether the reference is for merge or not, which, despite some other conditions, is also determined based on whether the old object ID the reference is being updated from actually exists in the repository. To write FETCH_HEAD we thus loop through all references thrice: once for the references that are about to be merged, once for the references that are not for merge, and finally for all references that are ignored. For every iteration, we then look up the old object ID to determine whether the referenced object exists so that we can label it as "not-for-merge" if it doesn't exist. It goes without saying that this can be expensive in case where we are fetching a lot of references. While this is hard to avoid in the case where we're writing FETCH_HEAD, users can in fact ask us to skip this work via `--no-write-fetch-head`. In that case, we do not care for the result of those lookups at all because we don't have to order writes to FETCH_HEAD in the first place. Skip this busywork in case we're not writing to FETCH_HEAD. The following benchmark performs a mirror-fetch in a repository with about two million references via `git fetch --prune --no-write-fetch-head +refs/*:refs/*`: Benchmark 1: HEAD~ Time (mean ± σ): 75.388 s ± 1.942 s [User: 71.103 s, System: 8.953 s] Range (min … max): 73.184 s … 76.845 s 3 runs Benchmark 2: HEAD Time (mean ± σ): 69.486 s ± 1.016 s [User: 65.941 s, System: 8.806 s] Range (min … max): 68.864 s … 70.659 s 3 runs Summary 'HEAD' ran 1.08 ± 0.03 times faster than 'HEAD~' Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-01Merge branch 'ps/fetch-atomic' into ps/fetch-mirror-optimLibravatar Junio C Hamano1-61/+129
* ps/fetch-atomic: fetch: make `--atomic` flag cover pruning of refs fetch: make `--atomic` flag cover backfilling of tags refs: add interface to iterate over queued transactional updates fetch: report errors when backfilling tags fails fetch: control lifecycle of FETCH_HEAD in a single place fetch: backfill tags before setting upstream fetch: increase test coverage of fetches
2022-02-25Merge branch 'ja/i18n-common-messages'Libravatar Junio C Hamano1-2/+2
Unify more messages to help l10n. * ja/i18n-common-messages: i18n: fix some misformated placeholders in command synopsis i18n: remove from i18n strings that do not hold translatable parts i18n: factorize "invalid value" messages i18n: factorize more 'incompatible options' messages
2022-02-23Merge branch 'ps/fetch-optim-with-commit-graph'Libravatar Junio C Hamano1-2/+6
A couple of optimization to "git fetch". * ps/fetch-optim-with-commit-graph: fetch: skip computing output width when not printing anything fetch-pack: use commit-graph when computing cutoff
2022-02-18Merge branch 'js/short-help-outside-repo-fix'Libravatar Junio C Hamano1-2/+4
"git cmd -h" outside a repository should error out cleanly for many commands, but instead it hit a BUG(), which has been corrected. * js/short-help-outside-repo-fix: t0012: verify that built-ins handle `-h` even without gitdir checkout/fetch/pull/pack-objects: allow `-h` outside a repository
2022-02-18Merge branch 'ab/release-transport-ls-refs-options'Libravatar Junio C Hamano1-1/+1
* ab/release-transport-ls-refs-options: ls-remote & transport API: release "struct transport_ls_refs_options"
2022-02-17fetch: make `--atomic` flag cover pruning of refsLibravatar Patrick Steinhardt1-8/+22
When fetching with the `--prune` flag we will delete any local references matching the fetch refspec which have disappeared on the remote. This step is not currently covered by the `--atomic` flag: we delete branches even though updating of local references has failed, which means that the fetch is not an all-or-nothing operation. Fix this bug by passing in the global transaction into `prune_refs()`: if one is given, then we'll only queue up deletions and not commit them right away. This change also improves performance when pruning many branches in a repository with a big packed-refs file: every references is pruned in its own transaction, which means that we potentially have to rewrite the packed-refs files for every single reference we're about to prune. The following benchmark demonstrates this: it performs a pruning fetch from a repository with a single reference into a repository with 100k references, which causes us to prune all but one reference. This is of course a very artificial setup, but serves to demonstrate the impact of only having to write the packed-refs file once: Benchmark 1: git fetch --prune --atomic +refs/*:refs/* (HEAD~) Time (mean ± σ): 2.366 s ± 0.021 s [User: 0.858 s, System: 1.508 s] Range (min … max): 2.328 s … 2.407 s 10 runs Benchmark 2: git fetch --prune --atomic +refs/*:refs/* (HEAD) Time (mean ± σ): 1.369 s ± 0.017 s [User: 0.715 s, System: 0.641 s] Range (min … max): 1.346 s … 1.400 s 10 runs Summary 'git fetch --prune --atomic +refs/*:refs/* (HEAD)' ran 1.73 ± 0.03 times faster than 'git fetch --prune --atomic +refs/*:refs/* (HEAD~)' Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-17fetch: make `--atomic` flag cover backfilling of tagsLibravatar Patrick Steinhardt1-26/+66
When fetching references from a remote we by default also fetch all tags which point into the history we have fetched. This is a separate step performed after updating local references because it requires us to walk over the history on the client-side to determine whether the remote has announced any tags which point to one of the fetched commits. This backfilling of tags isn't covered by the `--atomic` flag: right now, it only applies to the step where we update our local references. This is an oversight at the time the flag was introduced: its purpose is to either update all references or none, but right now we happily update local references even in the case where backfilling failed. Fix this by pulling up creation of the reference transaction such that we can pass the same transaction to both the code which updates local references and to the code which backfills tags. This allows us to only commit the transaction in case both actions succeed. Note that we also have to start passing the transaction into `find_non_local_tags()`: this function is responsible for finding all tags which we need to backfill. Right now, it will happily return tags which have already been updated with our local references. But when we use a single transaction for both local references and backfilling then it may happen that we try to queue the same reference update twice to the transaction, which consequently triggers a bug. We thus have to skip over any tags which have already been queued. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-17fetch: report errors when backfilling tags failsLibravatar Patrick Steinhardt1-8/+18
When the backfilling of tags fails we do not report this error to the caller, but only report it implicitly at a later point when reporting updated references. This leaves callers unable to act upon the information of whether the backfilling succeeded or not. Refactor the function to return an error code and pass it up the callstack. This causes us to correctly propagate the error back to the user of git-fetch(1). Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-17fetch: control lifecycle of FETCH_HEAD in a single placeLibravatar Patrick Steinhardt1-16/+19
There are two different locations where we're appending to FETCH_HEAD: first when storing updated references, and second when backfilling tags. Both times we open the file, append to it and then commit it into place, which is essentially duplicate work. Improve the lifecycle of updating FETCH_HEAD by opening and committing it once in `do_fetch()`, where we pass the structure down to the code which wants to append to it. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-17fetch: backfill tags before setting upstreamLibravatar Patrick Steinhardt1-17/+18
The fetch code flow is a bit hard to understand right now: 1. We optionally prune all references which have vanished on the remote side. 2. We fetch and update all other references locally. 3. We update the upstream branch in the gitconfig. 4. We backfill tags pointing into the history we have just fetched. It is quite confusing that we fetch objects and update references in both (2) and (4), which is further stressed by the point that we use a `skip` goto label to jump from (3) to (4) in case we fail to update the gitconfig as expected. Reorder the code to first update all local references, and only after we have done so update the upstream branch information. This improves the code flow and furthermore makes it easier to refactor the way we update references together. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-11Merge branch 'tg/fetch-prune-exit-code-fix'Libravatar Junio C Hamano1-4/+6
When "git fetch --prune" failed to prune the refs it wanted to prune, the command issued error messages but exited with exit status 0, which has been corrected. * tg/fetch-prune-exit-code-fix: fetch --prune: exit with error if pruning fails
2022-02-11Merge branch 'rc/negotiate-only-typofix'Libravatar Junio C Hamano1-1/+1
Typofix. * rc/negotiate-only-typofix: fetch: fix negotiate-only error message
2022-02-10fetch: skip computing output width when not printing anythingLibravatar Patrick Steinhardt1-2/+6
When updating references via git-fetch(1), then by default we report to the user which references have been changed. This output is formatted in a nice table such that the different columns are aligned. Because the first column contains abbreviated object IDs we thus need to iterate over all refs which have changed and compute the minimum length for their respective abbreviated hashes. While this effort makes sense in most cases, it is wasteful when the user passes the `--quiet` flag: we don't print the summary, but still compute the length. Skip computing the summary width when the user asked for us to be quiet. This gives us a speedup of nearly 10% when doing a mirror-fetch in a repository with thousands of references being updated: Benchmark 1: git fetch --quiet +refs/*:refs/* (HEAD~) Time (mean ± σ): 96.078 s ± 0.508 s [User: 91.378 s, System: 10.870 s] Range (min … max): 95.449 s … 96.760 s 5 runs Benchmark 2: git fetch --quiet +refs/*:refs/* (HEAD) Time (mean ± σ): 88.214 s ± 0.192 s [User: 83.274 s, System: 10.978 s] Range (min … max): 87.998 s … 88.446 s 5 runs Summary 'git fetch --quiet +refs/*:refs/* (HEAD)' ran 1.09 ± 0.01 times faster than 'git fetch --quiet +refs/*:refs/* (HEAD~)' Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-09Merge branch 'gc/fetch-negotiate-only-early-return'Libravatar Junio C Hamano1-3/+38
"git fetch --negotiate-only" is an internal command used by "git push" to figure out which part of our history is missing from the other side. It should never recurse into submodules even when fetch.recursesubmodules configuration variable is set, nor it should trigger "gc". The code has been tightened up to ensure it only does common ancestry discovery and nothing else. * gc/fetch-negotiate-only-early-return: fetch: help translators by reusing the same message template fetch --negotiate-only: do not update submodules fetch: skip tasks related to fetching objects fetch: use goto cleanup in cmd_fetch()
2022-02-08checkout/fetch/pull/pack-objects: allow `-h` outside a repositoryLibravatar Johannes Schindelin1-2/+4
When we taught these commands about the sparse index, we did not account for the fact that the `cmd_*()` functions _can_ be called without a gitdir, namely when `-h` is passed to show the usage. A plausible approach to address this is to move the `prepare_repo_settings()` calls right after the `parse_options()` calls: The latter will never return when it handles `-h`, and therefore it is safe to assume that we have a `gitdir` at that point, as long as the built-in is marked with the `RUN_SETUP` flag. However, it is unfortunately not that simple. In `cmd_pack_objects()`, for example, the repo settings need to be fully populated so that the command-line options `--sparse`/`--no-sparse` can override them, not the other way round. Therefore, we choose to imitate the strategy taken in `cmd_diff()`, where we simply do not bother to prepare and initialize the repo settings unless we have a `gitdir`. This fixes https://github.com/git-for-windows/git/issues/3688 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-06ls-remote & transport API: release "struct transport_ls_refs_options"Libravatar Ævar Arnfjörð Bjarmason1-1/+1
Fix a memory leak in codepaths that use the "struct transport_ls_refs_options" API. Since the introduction of the struct in 39835409d10 (connect, transport: encapsulate arg in struct, 2021-02-05) the caller has been responsible for freeing it. That commit in turn migrated code originally added in 402c47d9391 (clone: send ref-prefixes when using protocol v2, 2018-07-20) and b4be74105fe (ls-remote: pass ref prefixes when requesting a remote's refs, 2018-03-15). Only some of those codepaths were releasing the allocated resources of the struct, now all of them will. Mark the "t/t5511-refspec.sh" test as passing when git is compiled with SANITIZE=leak. They'll now be listed as running under the "GIT_TEST_PASSING_SANITIZE_LEAK=true" test mode (the "linux-leaks" CI target). Previously 24/47 tests would fail. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-04i18n: factorize "invalid value" messagesLibravatar Jean-Noël Avila1-2/+2
Use the same message when an invalid value is passed to a command line option or a configuration variable. Signed-off-by: Jean-Noël Avila <jn.avila@free.fr> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-01-31fetch --prune: exit with error if pruning failsLibravatar Thomas Gummerer1-4/+6
When pruning refs fails, we print an error to stderr, but still exit 0 from 'git fetch'. Since this is a genuine error, fetch should be exiting with some non-zero exit code. Make it so. The --prune option was introduced in f360d844de ("builtin-fetch: add --prune option", 2009-11-10). Unfortunately it's unclear from that commit whether ignoring the exit code was an oversight or intentional, but it feels like an oversight. Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-01-28fetch: fix negotiate-only error messageLibravatar Robert Coup1-1/+1
The error message when invoking a negotiate-only fetch without providing any tips incorrectly refers to a --negotiate-tip=* argument. Fix this to use the actual argument, --negotiation-tip=*. Signed-off-by: Robert Coup <robert@coup.net.nz> Reviewed-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-01-20fetch: help translators by reusing the same message templateLibravatar Junio C Hamano1-1/+2
Follow the example set by 12909b6b (i18n: turn "options are incompatible" into "cannot be used together", 2022-01-05) and use the same message string to reduce the need for translation. Reported-by: Jiang Xin <worldhello.net@gmail.com> Helped-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-01-18fetch --negotiate-only: do not update submodulesLibravatar Glen Choo1-1/+23
`git fetch --negotiate-only` is an implementation detail of push negotiation and, unlike most `git fetch` invocations, does not actually update the main repository. Thus it should not update submodules even if submodule recursion is enabled. This is not just slow, it is wrong e.g. push negotiation with "submodule.recurse=true" will cause submodules to be updated because it invokes `git fetch --negotiate-only`. Fix this by disabling submodule recursion if --negotiate-only was given. Since this makes --negotiate-only and --recurse-submodules incompatible, check for this invalid combination and die. This does not use the "goto cleanup" introduced in the previous commit because we want to recurse through submodules whenever a ref is fetched, and this can happen without introducing new objects. Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-01-18fetch: skip tasks related to fetching objectsLibravatar Glen Choo1-0/+11
cmd_fetch() does the following with the assumption that objects are fetched: * Run gc * Write commit graphs (if enabled by fetch.writeCommitGraph=true) However, neither of these tasks makes sense if objects are not fetched e.g. `git fetch --negotiate-only` never fetches objects. Speed up cmd_fetch() by bailing out early if we know for certain that objects will not be fetched. cmd_fetch() can bail out early whenever objects are not fetched, but for now this only considers --negotiate-only. The same optimization does not apply to `git fetch --dry-run` because that actually fetches objects; the dry run refers to not updating refs. Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-01-18fetch: use goto cleanup in cmd_fetch()Libravatar Glen Choo1-3/+4
Replace an early return with 'goto cleanup' in cmd_fetch() so that the string_list is always cleared (the string_list_clear() call is purely cleanup; the string_list is not reused). This makes cleanup consistent so that a subsequent commit can use 'goto cleanup' to bail out early. Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-01-12Merge branch 'ps/lockfile-cleanup-fix'Libravatar Junio C Hamano1-6/+11
Some lockfile code called free() in signal-death code path, which has been corrected. * ps/lockfile-cleanup-fix: fetch: fix deadlock when cleaning up lockfiles in async signals
2022-01-10Merge branch 'ja/i18n-similar-messages'Libravatar Junio C Hamano1-4/+4
Similar message templates have been consolidated so that translators need to work on fewer number of messages. * ja/i18n-similar-messages: i18n: turn even more messages into "cannot be used together" ones i18n: ref-filter: factorize "%(foo) atom used without %(bar) atom" i18n: factorize "--foo outside a repository" i18n: refactor "unrecognized %(foo) argument" strings i18n: factorize "no directory given for --foo" i18n: factorize "--foo requires --bar" and the like i18n: tag.c factorize i18n strings i18n: standardize "cannot open" and "cannot read" i18n: turn "options are incompatible" into "cannot be used together" i18n: refactor "%s, %s and %s are mutually exclusive" i18n: refactor "foo and bar are mutually exclusive"
2022-01-10Merge branch 'ds/fetch-pull-with-sparse-index'Libravatar Junio C Hamano1-0/+2
"git fetch" and "git pull" are now declared sparse-index clean. Also "git ls-files" learns the "--sparse" option to help debugging. * ds/fetch-pull-with-sparse-index: test-read-cache: remove --table, --expand options t1091/t3705: remove 'test-tool read-cache --table' t1092: replace 'read-cache --table' with 'ls-files --sparse' ls-files: add --sparse option fetch/pull: use the sparse index
2022-01-07fetch: fix deadlock when cleaning up lockfiles in async signalsLibravatar Patrick Steinhardt1-6/+11
When fetching packfiles, we write a bunch of lockfiles for the packfiles we're writing into the repository. In order to not leave behind any cruft in case we exit or receive a signal, we register both an exit handler as well as signal handlers for common signals like SIGINT. These handlers will then unlink the locks and free the data structure tracking them. We have observed a deadlock in this logic though: (gdb) bt #0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 #1 0x00007f4932bea2cd in _int_free (av=0x7f4932f2eb20 <main_arena>, p=0x3e3e4200, have_lock=0) at malloc.c:3969 #2 0x00007f4932bee58c in __GI___libc_free (mem=<optimized out>) at malloc.c:2975 #3 0x0000000000662ab1 in string_list_clear () #4 0x000000000044f5bc in unlock_pack_on_signal () #5 <signal handler called> #6 _int_free (av=0x7f4932f2eb20 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:4024 #7 0x00007f4932bee58c in __GI___libc_free (mem=<optimized out>) at malloc.c:2975 #8 0x000000000065afd5 in strbuf_release () #9 0x000000000066ddb9 in delete_tempfile () #10 0x0000000000610d0b in files_transaction_cleanup.isra () #11 0x0000000000611718 in files_transaction_abort () #12 0x000000000060d2ef in ref_transaction_abort () #13 0x000000000060d441 in ref_transaction_prepare () #14 0x000000000060e0b5 in ref_transaction_commit () #15 0x00000000004511c2 in fetch_and_consume_refs () #16 0x000000000045279a in cmd_fetch () #17 0x0000000000407c48 in handle_builtin () #18 0x0000000000408df2 in cmd_main () #19 0x00000000004078b5 in main () The process was killed with a signal, which caused the signal handler to kick in and try free the data structures after we have unlinked the locks. It then deadlocks while calling free(3P). The root cause of this is that it is not allowed to call certain functions in async-signal handlers, as specified by signal-safety(7). Next to most I/O functions, this list of disallowed functions also includes memory-handling functions like malloc(3P) and free(3P) because they may not be reentrant. As a result, if we execute such functions in the signal handler, then they may operate on inconistent state and fail in unexpected ways. Fix this bug by not calling non-async-signal-safe functions when running in the signal handler. We're about to re-raise the signal anyway and will thus exit, so it's not much of a problem to keep the string list of lockfiles untouched. Note that it's fine though to call unlink(2), so we'll still clean up the lockfiles correctly. Signed-off-by: Patrick Steinhardt <ps@pks.im> Reviewed-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-01-05i18n: standardize "cannot open" and "cannot read"Libravatar Jean-Noël Avila1-2/+2
Signed-off-by: Jean-Noël Avila <jn.avila@free.fr> Reviewed-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-01-05i18n: refactor "foo and bar are mutually exclusive"Libravatar Jean-Noël Avila1-2/+2
Use static strings for constant parts of the sentences. They are all turned into "cannot be used together". Signed-off-by: Jean-Noël Avila <jn.avila@free.fr> Reviewed-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-22Merge branch 'ab/fetch-set-upstream-while-detached'Libravatar Junio C Hamano1-0/+10
"git fetch --set-upstream" did not check if there is a current branch, leading to a segfault when it is run on a detached HEAD, which has been corrected. * ab/fetch-set-upstream-while-detached: pull, fetch: fix segfault in --set-upstream option
2021-12-22fetch/pull: use the sparse indexLibravatar Derrick Stolee1-0/+2
The 'git fetch' and 'git pull' commands parse the index in order to determine if submodules exist. Without command_requires_full_index=0, this will expand a sparse index, causing slow performance even when there is no new data to fetch. The .gitmodules file will never be inside a sparse directory entry, and even if it was, the index_name_pos() method would expand the sparse index if needed as we search for the path by name. These commands do not iterate over the index, which is the typical thing we are careful about when integrating with the sparse index. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-07pull, fetch: fix segfault in --set-upstream optionLibravatar Ævar Arnfjörð Bjarmason1-0/+10
Fix a segfault in the --set-upstream option added in 24bc1a12926 (pull, fetch: add --set-upstream option, 2019-08-19) added in v2.24.0. The code added there did not do the same checking we do for "git branch" itself since 8efb8899cfe (branch: segfault fixes and validation, 2013-02-23), which in turn fixed the same sort of segfault I'm fixing now in "git branch --set-upstream-to", see 6183d826ba6 (branch: introduce --set-upstream-to, 2012-08-20). The warning message I'm adding here is an amalgamation of the error added for "git branch" in 8efb8899cfe, and the error output install_branch_config() itself emits, i.e. it trims "refs/heads/" from the name and says "branch X on remote", not "branch refs/heads/X on remote". I think it would make more sense to simply die() here, but in the other checks for --set-upstream added in 24bc1a12926 we issue a warning() instead. Let's do the same here for consistency for now. There was an earlier submitted alternate way of fixing this in [1], due to that patch breaking threading with the original report at [2] I didn't notice it before authoring this version. I think the more detailed warning message here is better, and we should also have tests for this behavior. The --no-rebase option to "git pull" is needed as of the recently merged 7d0daf3f12f (Merge branch 'en/pull-conflicting-options', 2021-08-30). 1. https://lore.kernel.org/git/20210706162238.575988-1-clemens@endorphin.org/ 2. https://lore.kernel.org/git/CAG6gW_uHhfNiHGQDgGmb1byMqBA7xa8kuH1mP-wAPEe5Tmi2Ew@mail.gmail.com/ Reported-by: Clemens Fruhwirth <clemens@endorphin.org> Reported-by: Jan Pokorný <poki@fnusa.cz> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-01fetch: protect branches checked out in all worktreesLibravatar Anders Kaseorg1-35/+40
Refuse to fetch into the currently checked out branch of any working tree, not just the current one. Fixes this previously reported bug: https://lore.kernel.org/git/cb957174-5e9a-5603-ea9e-ac9b58a2eaad@mathema.de/ As a side effect of using find_shared_symref, we’ll also refuse the fetch when we’re on a detached HEAD because we’re rebasing or bisecting on the branch in question. This seems like a sensible change. Signed-off-by: Anders Kaseorg <andersk@mit.edu> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-01fetch: lowercase error messagesLibravatar Anders Kaseorg1-24/+26
Documentation/CodingGuidelines says “do not end error messages with a full stop” and “do not capitalize the first word”. Clean up existing messages, some of which we will be touching in later steps in the series, that deviate from these rules in this file, as a preparation for the main part of the topic. Signed-off-by: Anders Kaseorg <andersk@mit.edu> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-20Merge branch 'js/run-command-close-packs'Libravatar Junio C Hamano1-2/+0
The run-command API has been updated so that the callers can easily ask the file descriptors open for packfiles to be closed immediately before spawning commands that may trigger auto-gc. * js/run-command-close-packs: Close object store closer to spawning child processes run_auto_maintenance(): implicitly close the object store run-command: offer to close the object store before running run-command: prettify the `RUN_COMMAND_*` flags pull: release packs before fetching commit-graph: when closing the graph, also release the slab
2021-09-20Merge branch 'ps/fetch-optim'Libravatar Junio C Hamano1-34/+40
Optimize code that handles large number of refs in the "git fetch" code path. * ps/fetch-optim: fetch: avoid second connectivity check if we already have all objects fetch: merge fetching and consuming refs fetch: refactor fetch refs to be more extendable fetch-pack: optimize loading of refs via commit graph connected: refactor iterator to return next object ID directly fetch: avoid unpacking headers in object existence check fetch: speed up lookup of want refs via commit-graph
2021-09-10Merge branch 'ab/retire-advice-config'Libravatar Junio C Hamano1-1/+1
Code clean up to migrate callers from older advice_config[] based API to newer advice_if_enabled() and advice_enabled() API. * ab/retire-advice-config: advice: move advice.graftFileDeprecated squashing to commit.[ch] advice: remove use of global advice_add_embedded_repo advice: remove read uses of most global `advice_` variables advice: add enum variants for missing advice variables
2021-09-10Merge branch 'ps/fetch-omit-formatting-under-quiet'Libravatar Junio C Hamano1-5/+12
"git fetch --quiet" optimization to avoid useless computation of info that will never be displayed. * ps/fetch-omit-formatting-under-quiet: fetch: skip formatting updated refs with `--quiet`
2021-09-09run_auto_maintenance(): implicitly close the object storeLibravatar Johannes Schindelin1-2/+0
Before spawning the auto maintenance, we need to make sure that we release all open file handles to all the `.pack` files (and MIDX files and commit-graph files and...) so that the maintenance process has the freedom to delete those files. So far, we did this manually every time before calling `run_auto_maintenance()`. With the new `close_object_store` flag, we can do that implicitly in that function, which is more robust because future callers won't be able to forget to close the object store. Note: this changes behavior slightly, as we previously _always_ closed the object store, but now we only close the object store when actually running the auto maintenance. In practice, this should not matter (if anything, it might speed up operations where auto maintenance is disabled). Suggested-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-01fetch: avoid second connectivity check if we already have all objectsLibravatar Patrick Steinhardt1-4/+3
When fetching refs, we are doing two connectivity checks: - The first one is done such that we can skip fetching refs in the case where we already have all objects referenced by the updated set of refs. - The second one verifies that we have all objects after we have fetched objects. We always execute both connectivity checks, but this is wasteful in case the first connectivity check already notices that we have all objects locally available. Skip the second connectivity check in case we already had all objects available. This gives us a nice speedup when doing a mirror-fetch in a repository with about 2.3M refs where the fetching repo already has all objects: Benchmark #1: HEAD~: git-fetch Time (mean ± σ): 30.025 s ± 0.081 s [User: 27.070 s, System: 4.933 s] Range (min … max): 29.900 s … 30.111 s 5 runs Benchmark #2: HEAD: git-fetch Time (mean ± σ): 25.574 s ± 0.177 s [User: 22.855 s, System: 4.683 s] Range (min … max): 25.399 s … 25.765 s 5 runs Summary 'HEAD: git-fetch' ran 1.17 ± 0.01 times faster than 'HEAD~: git-fetch' Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-01fetch: merge fetching and consuming refsLibravatar Patrick Steinhardt1-21/+9
The functions `fetch_refs()` and `consume_refs()` must always be called together such that we first obtain all missing objects and then update our local refs to match the remote refs. In a subsequent patch, we'll further require that `fetch_refs()` must always be called before `consume_refs()` such that it can correctly assert that we have all objects after the fetch given that we're about to move the connectivity check. Make this requirement explicit by merging both functions into a single `fetch_and_consume_refs()` function. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-01fetch: refactor fetch refs to be more extendableLibravatar Patrick Steinhardt1-7/+17
Refactor `fetch_refs()` code to make it more extendable by explicitly handling error cases. The refactored code should behave the same. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>