summaryrefslogtreecommitdiff
path: root/run-command.h
AgeCommit message (Collapse)AuthorFilesLines
2021-11-25run-command API: remove "env" member, always use "env_array"Libravatar Ævar Arnfjörð Bjarmason1-17/+17
Remove the "env" member from "struct child_process" in favor of always using the "env_array". As with the preceding removal of "argv" in favor of "args" this gets rid of current and future oddities around memory management at the API boundary (see the amended API docs). For some of the conversions we can replace patterns like: child.env = env->v; With: strvec_pushv(&child.env_array, env->v); But for others we need to guard the strvec_pushv() with a NULL check, since we're not passing in the "v" member of a "struct strvec", e.g. in the case of tmp_objdir_env()'s return value. Ideally we'd rename the "env_array" member to simply "env" as a follow-up, since it and "args" are now inconsistent in not having an "_array" suffix, and seemingly without any good reason, unless we look at the history of how they came to be. But as we've currently got 122 in-tree hits for a "git grep env_array" let's leave that for now (and possibly forever). Doing that rename would be too disruptive. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-11-25run-command API: remove "argv" member, always use "args"Libravatar Ævar Arnfjörð Bjarmason1-12/+8
Remove the "argv" member from the run-command API, ever since "args" was added in c460c0ecdca (run-command: store an optional argv_array, 2014-05-15) being able to provide either "argv" or "args" has led to some confusion and bugs. If we hadn't gone in that direction and only had an "argv" our problems wouldn't have been solved either, as noted in [1] (and in the documentation amended here) it comes with inherent memory management issues: The caller would have to hang on to the "argv" until the run-command API was finished. If the "argv" was an argument to main() this wasn't an issue, but if it it was manually constructed using the API might be painful. We also have a recent report[2] of a user of the API segfaulting, which is a direct result of it being complex to use. This commit addresses the root cause of that bug. This change is larger than I'd like, but there's no easy way to avoid it that wouldn't involve even more verbose intermediate steps. We use the "argv" as the source of truth over the "args", so we need to change all parts of run-command.[ch] itself, as well as the trace2 logging at the same time. The resulting Windows-specific code in start_command() is a bit nasty, as we're now assigning to a strvec's "v" member, instead of to our own "argv". There was a suggestion of some alternate approaches in reply to an earlier version of this commit[3], but let's leave larger a larger and needless refactoring of this code for now. 1. http://lore.kernel.org/git/YT6BnnXeAWn8BycF@coredump.intra.peff.net 2. https://lore.kernel.org/git/20211120194048.12125-1-ematsumiya@suse.de/ 3. https://lore.kernel.org/git/patch-5.5-ea1011f7473-20211122T153605Z-avarab@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-13Merge branch 'jh/builtin-fsmonitor-part1'Libravatar Junio C Hamano1-0/+57
Built-in fsmonitor (part 1). * jh/builtin-fsmonitor-part1: t/helper/simple-ipc: convert test-simple-ipc to use start_bg_command run-command: create start_bg_command simple-ipc/ipc-win32: add Windows ACL to named pipe simple-ipc/ipc-win32: add trace2 debugging simple-ipc: move definition of ipc_active_state outside of ifdef simple-ipc: preparations for supporting binary messages. trace2: add trace2_child_ready() to report on background children
2021-09-27hook.[ch]: move find_hook() from run-command.c to hook.cLibravatar Ævar Arnfjörð Bjarmason1-7/+0
Move the find_hook() function from run-command.c to a new hook.c library. This change establishes a stub library that's pretty pointless right now, but will see much wider use with Emily Shaffer's upcoming "configuration-based hooks" series. Eventually all the hook related code will live in hook.[ch]. Let's start that process by moving the simple find_hook() function over as-is. Signed-off-by: Emily Shaffer <emilyshaffer@google.com> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-23Merge branch 'mr/bisect-in-c-4'Libravatar Junio C Hamano1-0/+12
Rewrite of "git bisect" in C continues. * mr/bisect-in-c-4: bisect--helper: retire `--bisect-next-check` subcommand bisect--helper: reimplement `bisect_run` shell function in C bisect--helper: reimplement `bisect_visualize()` shell function in C run-command: make `exists_in_PATH()` non-static t6030-bisect-porcelain: add test for bisect visualize t6030-bisect-porcelain: add tests to control bisect run exit cases
2021-09-20run-command: create start_bg_commandLibravatar Jeff Hostetler1-0/+57
Create a variation of `run_command()` and `start_command()` to launch a command into the background and optionally wait for it to become "ready" before returning. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-13run-command: make `exists_in_PATH()` non-staticLibravatar Pranit Bauva1-0/+12
Remove the `static` keyword from `exists_in_PATH()` function and declare the function in `run-command.h` file. The function will be used in bisect_visualize() in a later commit. Mentored by: Christian Couder <chriscool@tuxfamily.org> Mentored by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Signed-off-by: Tanushree Tumane <tanushreetumane@gmail.com> Signed-off-by: Miriam Rubio <mirucam@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-09run-command: offer to close the object store before runningLibravatar Johannes Schindelin1-0/+9
Especially on Windows, where files cannot be deleted if _any_ process holds an open file handle to them, it is important to close the object store (releasing all handles to all `.pack` files) before running a command that might spawn a garbage collection. This scenario is so common that we frequently see the pattern of closing the object store before running auto maintenance or another Git command. Let's make this much more convenient by teaching the `run_command()` machinery a new flag to release the object store before spawning the process. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-09run-command: prettify the `RUN_COMMAND_*` flagsLibravatar Johannes Schindelin1-7/+7
The values were listed unaligned, and with powers of two spelled out in decimal. The list is easier to parse for human readers if the numbers are aligned and spelled out as powers of two (using the bit-shift operator `<<`). While at it, remove a code comment that was unclear at best, and confusing at worst. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-16Merge branch 'jt/partial-clone-submodule-1'Libravatar Junio C Hamano1-0/+10
Prepare the internals for lazily fetching objects in submodules from their promisor remotes. * jt/partial-clone-submodule-1: promisor-remote: teach lazy-fetch in any repo run-command: refactor subprocess env preparation submodule: refrain from filtering GIT_CONFIG_COUNT promisor-remote: support per-repository config repository: move global r_f_p_c to repo struct
2021-07-01*.h: move some *_INIT to designated initializersLibravatar Ævar Arnfjörð Bjarmason1-1/+4
Move *_INIT macros I'll use in a subsequent commits to designated initializers. This isn't required for those follow-up changes, but since next commits will change things in this area, let's use the modern pattern over the old one while we're at it. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28run-command: refactor subprocess env preparationLibravatar Jonathan Tan1-0/+10
submodule.c has functionality that prepares the environment for running a subprocess in a new repo. The lazy-fetching code (used in partial clones) will need this in a subsequent commit, so move it to a more central location. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-01-22run-command: document use_shell optionLibravatar Jeff King1-1/+8
It's unclear how run-command's use_shell option should impact the arguments fed to a command. Plausibly it could mean that we glue all of the arguments together into a string to pass to the shell, in which case that opens the question of whether the caller needs to quote them. But in fact we don't implement it that way (and even if we did, we'd probably auto-quote the arguments as part of the glue step). And we must not receive quoted arguments, because we might actually optimize out the shell entirely (i.e., the caller does not even know if a shell will be involved in the end or not). Since this ambiguity may have been the cause of a recent bug, let's document the option a bit. Signed-off-by: Jeff King <peff@peff.net> Reviewed-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-17maintenance: replace run_auto_gc()Libravatar Derrick Stolee1-1/+1
The run_auto_gc() method is used in several places to trigger a check for repo maintenance after some Git commands, such as 'git commit' or 'git fetch'. To allow for extra customization of this maintenance activity, replace the 'git gc --auto [--quiet]' call with one to 'git maintenance run --auto [--quiet]'. As we extend the maintenance builtin with other steps, users will be able to select different maintenance activities. Rename run_auto_gc() to run_auto_maintenance() to be clearer what is happening on this call, and to expose all callers in the current diff. Rewrite the method to use a struct child_process to simplify the calls slightly. Since 'git fetch' already allows disabling the 'git gc --auto' subprocess, add an equivalent option with a different name to be more descriptive of the new behavior: '--[no-]maintenance'. Update the documentation to include these options at the same time. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-28strvec: convert remaining callers away from argv_array nameLibravatar Jeff King1-5/+5
We eventually want to drop the argv_array name and just use strvec consistently. There's no particular reason we have to do it all at once, or care about interactions between converted and unconverted bits. Because of our preprocessor compat layer, the names are interchangeable to the compiler (so even a definition and declaration using different names is OK). This patch converts all of the remaining files, as the resulting diff is reasonably sized. The conversion was done purely mechanically with: git ls-files '*.c' '*.h' | xargs perl -i -pe ' s/ARGV_ARRAY/STRVEC/g; s/argv_array/strvec/g; ' We'll deal with any indentation/style fallouts separately. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-28strvec: rename files from argv-array to strvecLibravatar Jeff King1-1/+1
This requires updating #include lines across the code-base, but that's all fairly mechanical, and was done with: git ls-files '*.c' '*.h' | xargs perl -i -pe 's/argv-array.h/strvec.h/' Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-15Merge branch 'ta/wait-on-aliased-commands-upon-signal' into masterLibravatar Junio C Hamano1-0/+1
When an aliased command, whose output is piped to a pager by git, gets killed by a signal, the pager got into a funny state, which has been corrected (again). * ta/wait-on-aliased-commands-upon-signal: Wait for child on signal death for aliases to externals Wait for child on signal death for aliases to builtins
2020-07-07Wait for child on signal death for aliases to builtinsLibravatar Trygve Aaberge1-0/+1
When you hit ^C all the processes in the tree receives it. When a git command uses a pager, git ignores this and waits until the pager quits. However, when using an alias there is an additional process in the tree which didn't ignore the signal. That caused it to exit which in turn caused the pager to exit. This fixes that for aliases to builtins. This was originally fixed in 46df6906 (execv_dashed_external: wait for child on signal death, 2017-01-06), but was broken by ee4512ed (trace2: create new combined trace facility, 2019-02-22) and then b9140840 (git: avoid calling aliased builtins via their dashed form, 2019-07-29). Signed-off-by: Trygve Aaberge <trygveaa@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-05-07auto-gc: extract a reusable helper from "git fetch"Libravatar Junio C Hamano1-0/+5
Back in 1991006c (fetch: convert argv_gc_auto to struct argv_array, 2014-08-16), we taught "git fetch --quiet" to pass the "--quiet" option down to "gc --auto". This issue, however, is not limited to "fetch": $ git grep -e 'gc.*--auto' \*.c finds hits in "am", "commit", "merge", and "rebase" and these commands do not pass "--quiet" down to "gc --auto" when they themselves are told to be quiet. As a preparatory step, let's introduce a helper function run_auto_gc(), that the caller can pass a boolean "quiet", and redo the fix to "git fetch" using the helper. Signed-off-by: Junio C Hamano <gitster@pobox.com> Reviewed-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-22run-command.h: fix mis-indented struct memberLibravatar Jeff King1-1/+1
An accidental conversion of a tab to 4 spaces snuck into 4c4066d95d (run-command: move doc to run-command.h, 2019-11-17), messing up the alignment when you have the project-recommended 8-width tabstops. Let's revert that line. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-18run-command: move doc to run-command.hLibravatar Heba Waly1-7/+245
Move the documentation from Documentation/technical/api-run-command.txt to run-command.h as it's easier for the developers to find the usage information beside the code instead of looking for it in another doc file. Documentation/technical/api-run-command.txt is removed because the information it has is now redundant and it'll be hard to keep it up to date and synchronized with the documentation in the header file. Signed-off-by: Heba Waly <heba.waly@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-05-05*.[ch]: remove extern from function declarations using sedLibravatar Denton Liu1-1/+1
There has been a push to remove extern from function declarations. Finish the job by removing all instances of "extern" for function declarations in headers using sed. This was done by running the following on my system with sed 4.2.2: $ git ls-files \*.{c,h} | grep -v ^compat/ | xargs sed -i'' -e 's/^\(\s*\)extern \([^(]*([^*]\)/\1\2/' Files under `compat/` are intentionally excluded as some are directly copied from external sources and we should avoid churning them as much as possible. Then, leftover instances of extern were found by running $ git grep -w -C3 extern \*.{c,h} and manually checking the output. No other instances were found. Note that the regex used specifically excludes function variables which _should_ be left as extern. Not the most elegant way to do it but it gets the job done. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-05-05*.[ch]: remove extern from function declarations using spatchLibravatar Denton Liu1-3/+3
There has been a push to remove extern from function declarations. Remove some instances of "extern" for function declarations which are caught by Coccinelle. Note that Coccinelle has some difficulty with processing functions with `__attribute__` or varargs so some `extern` declarations are left behind to be dealt with in a future patch. This was the Coccinelle patch used: @@ type T; identifier f; @@ - extern T f(...); and it was run with: $ git ls-files \*.{c,h} | grep -v ^compat/ | xargs spatch --sp-file contrib/coccinelle/noextern.cocci --in-place Files under `compat/` are intentionally excluded as some are directly copied from external sources and we should avoid churning them as much as possible. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-22trace2: create new combined trace facilityLibravatar Jeff Hostetler1-1/+12
Create a new unified tracing facility for git. The eventual intent is to replace the current trace_printf* and trace_performance* routines with a unified set of git_trace2* routines. In addition to the usual printf-style API, trace2 provides higer-level event verbs with fixed-fields allowing structured data to be written. This makes post-processing and analysis easier for external tools. Trace2 defines 3 output targets. These are set using the environment variables "GIT_TR2", "GIT_TR2_PERF", and "GIT_TR2_EVENT". These may be set to "1" or to an absolute pathname (just like the current GIT_TRACE). * GIT_TR2 is intended to be a replacement for GIT_TRACE and logs command summary data. * GIT_TR2_PERF is intended as a replacement for GIT_TRACE_PERFORMANCE. It extends the output with columns for the command process, thread, repo, absolute and relative elapsed times. It reports events for child process start/stop, thread start/stop, and per-thread function nesting. * GIT_TR2_EVENT is a new structured format. It writes event data as a series of JSON records. Calls to trace2 functions log to any of the 3 output targets enabled without the need to call different trace_printf* or trace_performance* routines. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-05send-pack.c: move async's #ifdef NO_PTHREADS back to run-command.cLibravatar Nguyễn Thái Ngọc Duy1-0/+1
On systems that do not support multithread, start_async() is implemented with fork(). This implementation details unfortunately leak out at least in send-pack.c [1]. To keep the code base clean of NO_PTHREADS, move the this #ifdef back to run-command.c. The new wrapper function async_with_fork() at least helps suggest that this special "close()" is related to async in fork mode. [1] 09c9957cf7 (send-pack: avoid deadlock when pack-object dies early - 2011-04-25) Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-05run-command.h: include thread-utils.h instead of pthread.hLibravatar Nguyễn Thái Ngọc Duy1-3/+1
run-command.c may use threads for its async support. But instead of including pthread.h directly, let's include thread-utils.h. run-command.c probably never needs the dummy bits in thread-utils.h when NO_PTHREADS is defined. But this makes sure we have consistent HAVE_THREADS behavior everywhere. From now on outside compat/, thread-utils.h is the only place that includes pthread.h Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-25run-command: expose is_executable functionLibravatar Brandon Williams1-0/+1
Move the logic for 'is_executable()' from help.c to run_command.c and expose it so that callers from outside help.c can access the function. This is to enable run-command to be able to query if a file is executable in a future patch. Signed-off-by: Brandon Williams <bmwill@google.com> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-09execv_dashed_external: wait for child on signal deathLibravatar Jeff King1-0/+1
When you hit ^C to interrupt a git command going to a pager, this usually leaves the pager running. But when a dashed external is in use, the pager ends up in a funny state and quits (but only after eating one more character from the terminal!). This fixes it. Explaining the reason will require a little background. When git runs a pager, it's important for the git process to hang around and wait for the pager to finish, even though it has no more data to feed it. This is because git spawns the pager as a child, and thus the git process is the session leader on the terminal. After it dies, the pager will finish its current read from the terminal (eating the one character), and then get EIO trying to read again. When you hit ^C, that sends SIGINT to git and to the pager, and it's a similar situation. The pager ignores it, but the git process needs to hang around until the pager is done. We addressed that long ago in a3da882120 (pager: do wait_for_pager on signal death, 2009-01-22). But when you have a dashed external (or an alias pointing to a builtin, which will re-exec git for the builtin), there's an extra process in the mix. For instance, running: $ git -c alias.l=log l will end up with a process tree like: git (parent) \ git-log (child) \ less (pager) If you hit ^C, SIGINT goes to all of them. The pager ignores it, and the child git process will end up in wait_for_pager(). But the parent git process will die, and the usual EIO trouble happens. So we really want the parent git process to wait_for_pager(), but of course it doesn't know anything about the pager at all, since it was started by the child. However, we can have it wait on the git-log child, which in turn is waiting on the pager. And that's what this patch does. There are a few design decisions here worth explaining: 1. The new feature is attached to run-command's clean_on_exit feature. Partly this is convenience, since that feature already has a signal handler that deals with child cleanup. But it's also a meaningful connection. The main reason that dashed externals use clean_on_exit is to bind the two processes together. If somebody kills the parent with a signal, we propagate that to the child (in this instance with SIGINT, we do propagate but it doesn't matter because the original signal went to the whole process group). Likewise, we do not want the parent to go away until the child has done so. In a traditional Unix world, we'd probably accomplish this binding by just having the parent execve() the child directly. But since that doesn't work on Windows, everything goes through run_command's more spawn-like interface. 2. We do _not_ automatically waitpid() on any clean_on_exit children. For dashed externals this makes sense; we know that the parent is doing nothing but waiting for the child to exit anyway. But with other children, it's possible that the child, after getting the signal, could be waiting on the parent to do something (like closing a descriptor). If we were to wait on such a child, we'd end up in a deadlock. So this errs on the side of caution, and lets callers enable the feature explicitly. 3. When we send children the cleanup signal, we send all the signals first, before waiting on any children. This is to avoid the case where one child might be waiting on another one to exit, causing a deadlock. We inform all of them that it's time to die before reaping any. In practice, there is only ever one dashed external run from a given process, so this doesn't matter much now. But it future-proofs us if other callers start using the wait_after_clean mechanism. There's no automated test here, because it would end up racy and unportable. But it's easy to reproduce the situation by running the log command given above and hitting ^C. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-17run-command: add clean_on_exit_handlerLibravatar Lars Schneider1-0/+2
Some processes might want to perform cleanup tasks before Git kills them due to the 'clean_on_exit' flag. Let's give them an interface for doing this. The feature is used in a subsequent patch. Please note, that the cleanup callback is not executed if Git dies of a signal. The reason is that only "async-signal-safe" functions would be allowed to be call in that case. Since we cannot control what functions the callback will use, we will not support the case. See 507d7804 for more details. Helped-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Lars Schneider <larsxschneider@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-17run-command: move check_pipe() from write_or_die to run_commandLibravatar Lars Schneider1-1/+1
Move check_pipe() to run_command and make it public. This is necessary to call the function from pkt-line in a subsequent patch. While at it, make async_exit() static to run_command.c as it is no longer used from outside. Signed-off-by: Lars Schneider <larsxschneider@gmail.com> Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-06-17run-command: add pipe_command helperLibravatar Jeff King1-7/+24
We already have capture_command(), which captures the stdout of a command in a way that avoids deadlocks. But sometimes we need to do more I/O, like capturing stderr as well, or sending data to stdin. It's easy to write code that deadlocks racily in these situations depending on how fast the command reads its input, or in which order it writes its output. Let's give callers an easy interface for doing this the right way, similar to what capture_command() did for the simple case. The whole thing is backed by a generic poll() loop that can feed an arbitrary number of buffers to descriptors, and fill an arbitrary number of strbufs from other descriptors. This seems like overkill, but the resulting code is actually a bit cleaner than just handling the three descriptors (because the output code for stdout/stderr is effectively duplicated, so being able to loop is a benefit). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-04-29Merge branch 'jk/push-client-deadlock-fix'Libravatar Junio C Hamano1-0/+1
"git push" from a corrupt repository that attempts to push a large number of refs deadlocked; the thread to relay rejection notices for these ref updates blocked on writing them to the main thread, after the main thread at the receiving end notices that the push failed and decides not to read these notices and return a failure. * jk/push-client-deadlock-fix: t5504: drop sigpipe=ok from push tests fetch-pack: isolate sigpipe in demuxer thread send-pack: isolate sigpipe in demuxer thread run-command: teach async threads to ignore SIGPIPE send-pack: close demux pipe before finishing async process
2016-04-20run-command: teach async threads to ignore SIGPIPELibravatar Jeff King1-0/+1
Async processes can be implemented as separate forked processes, or as threads (depending on the NO_PTHREADS setting). In the latter case, if an async thread gets SIGPIPE, it takes down the whole process. This is obviously bad if the main process was not otherwise going to die, but even if we were going to die, it means the main process does not have a chance to report a useful error message. There's also the small matter that forked async processes will not take the main process down on a signal, meaning git will behave differently depending on the NO_PTHREADS setting. This patch fixes it by adding a new flag to "struct async" to block SIGPIPE just in the async thread. In theory, this should always be on (which makes async threads behave more like async processes), but we would first want to make sure that each async process we spawn is careful about checking return codes from write() and would not spew endlessly into a dead pipe. So let's start with it as optional, and we can enable it for specific sites in future patches. The natural name for this option would be "ignore_sigpipe", since that's what it does for the threaded case. But since that name might imply that we are ignoring it in all cases (including the separate-process one), let's call it "isolate_sigpipe". What we are really asking for is isolation. I.e., not to have our main process taken down by signals spawned by the async process. How that is implemented is up to the run-command code. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-04-06Merge branch 'sb/submodule-parallel-update'Libravatar Junio C Hamano1-5/+5
A major part of "git submodule update" has been ported to C to take advantage of the recently added framework to run download tasks in parallel. * sb/submodule-parallel-update: clone: allow an explicit argument for parallel submodule clones submodule update: expose parallelism to the user submodule helper: remove double 'fatal: ' prefix git submodule update: have a dedicated helper for cloning run_processes_parallel: rename parameters for the callbacks run_processes_parallel: treat output of children as byte array submodule update: direct error message to stderr fetching submodules: respect `submodule.fetchJobs` config option submodule-config: drop check against NULL submodule-config: keep update strategy around
2016-03-04Merge branch 'sb/submodule-parallel-fetch'Libravatar Junio C Hamano1-6/+3
Simplify the two callback functions that are triggered when the child process terminates to avoid misuse of the child-process structure that has already been cleaned up. * sb/submodule-parallel-fetch: run-command: do not pass child process data into callbacks
2016-03-01run_processes_parallel: rename parameters for the callbacksLibravatar Stefan Beller1-5/+5
The refs code has a similar pattern of passing around 'struct strbuf *err', which is strictly used for error reporting. This is not the case here, as the strbuf is used to accumulate all the output (whether it is error or not) for the user. Rename it to 'out'. Suggested-by: Jonathan Nieder <jrnieder@gmail.com> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-03-01run-command: do not pass child process data into callbacksLibravatar Stefan Beller1-6/+3
The expected way to pass data into the callback is to pass them via the customizable callback pointer. The error reporting in default_{start_failure, task_finished} is not user friendly enough, that we want to encourage using the child data for such purposes. Furthermore the struct child data is cleaned by the run-command API, before we access them in the callbacks, leading to use-after-free situations. Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-02-26Merge branch 'jk/epipe-in-async'Libravatar Junio C Hamano1-0/+1
Handling of errors while writing into our internal asynchronous process has been made more robust, which reduces flakiness in our tests. * jk/epipe-in-async: t5504: handle expected output from SIGPIPE death test_must_fail: report number of unexpected signal fetch-pack: ignore SIGPIPE in sideband demuxer write_or_die: handle EPIPE in async threads
2016-02-25write_or_die: handle EPIPE in async threadsLibravatar Jeff King1-0/+1
When write_or_die() sees EPIPE, it treats it specially by converting it into a SIGPIPE death. We obviously cannot ignore it, as the write has failed and the caller expects us to die. But likewise, we cannot just call die(), because printing any message at all would be a nuisance during normal operations. However, this is a problem if write_or_die() is called from a thread. Our raised signal ends up killing the whole process, when logically we just need to kill the thread (after all, if we are ignoring SIGPIPE, there is good reason to think that the main thread is expecting to handle it). Inside an async thread, the die() code already does the right thing, because we use our custom die_async() routine, which calls pthread_join(). So ideally we would piggy-back on that, and simply call: die_quietly_with_code(141); or similar. But refactoring the die code to do this is surprisingly non-trivial. The die_routines themselves handle both printing and the decision of the exit code. Every one of them would have to be modified to take new parameters for the code, and to tell us to be quiet. Instead, we can just teach write_or_die() to check for the async case and handle it specially. We do have to build an interface to abstract the async exit, but it's simple and self-contained. If we had many call-sites that wanted to do this die_quietly_with_code(), this approach wouldn't scale as well, but we don't. This is the only place where do this weird exit trick. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-16run-command: add an asynchronous parallel child processorLibravatar Stefan Beller1-0/+80
This allows to run external commands in parallel with ordered output on stderr. If we run external commands in parallel we cannot pipe the output directly to the our stdout/err as it would mix up. So each process's output will flow through a pipe, which we buffer. One subprocess can be directly piped to out stdout/err for a low latency feedback to the user. Example: Let's assume we have 5 submodules A,B,C,D,E and each fetch takes a different amount of time as the different submodules vary in size, then the output of fetches in sequential order might look like this: time --> output: |---A---| |-B-| |-------C-------| |-D-| |-E-| When we schedule these submodules into maximal two parallel processes, a schedule and sample output over time may look like this: process 1: |---A---| |-D-| |-E-| process 2: |-B-| |-------C-------| output: |---A---|B|---C-------|DE So A will be perceived as it would run normally in the single child version. As B has finished by the time A is done, we can dump its whole progress buffer on stderr, such that it looks like it finished in no time. Once that is done, C is determined to be the visible child and its progress will be reported in real time. So this way of output is really good for human consumption, as it only changes the timing, not the actual output. For machine consumption the output needs to be prepared in the tasks, by either having a prefix per line or per block to indicate whose tasks output is displayed, because the output order may not follow the original sequential ordering: |----A----| |--B--| |-C-| will be scheduled to be all parallel: process 1: |----A----| process 2: |--B--| process 3: |-C-| output: |----A----|CB This happens because C finished before B did, so it will be queued for output before B. To detect when a child has finished executing, we check interleaved with other actions (such as checking the liveliness of children or starting new processes) whether the stderr pipe still exists. Once a child closed its stderr stream, we assume it is terminating very soon, and use `finish_command()` from the single external process execution interface to collect the exit status. By maintaining the strong assumption of stderr being open until the very end of a child process, we can avoid other hassle such as an implementation using `waitpid(-1)`, which is not implemented in Windows. Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-11-03Merge branch 'rs/daemon-plug-child-leak'Libravatar Junio C Hamano1-0/+1
"git daemon" uses "run_command()" without "finish_command()", so it needs to release resources itself, which it forgot to do. * rs/daemon-plug-child-leak: daemon: plug memory leak run-command: factor out child_process_clear()
2015-11-02run-command: factor out child_process_clear()Libravatar René Scharfe1-0/+1
Avoid duplication by moving the code to release allocated memory for arguments and environment to its own function, child_process_clear(). Export it to provide a counterpart to child_process_init(). Signed-off-by: Rene Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-10-07Merge branch 'ti/glibc-stdio-mutex-from-signal-handler'Libravatar Junio C Hamano1-0/+1
Allocation related functions and stdio are unsafe things to call inside a signal handler, and indeed killing the pager can cause glibc to deadlock waiting on allocation mutex as our signal handler tries to free() some data structures in wait_for_pager(). Reduce these unsafe calls. * ti/glibc-stdio-mutex-from-signal-handler: pager: don't use unsafe functions in signal handlers
2015-10-05Merge branch 'jk/async-pkt-line'Libravatar Junio C Hamano1-0/+1
The debugging infrastructure for pkt-line based communication has been improved to mark the side-band communication specifically. * jk/async-pkt-line: pkt-line: show packets in async processes as "sideband" run-command: provide in_async query function
2015-09-04pager: don't use unsafe functions in signal handlersLibravatar Takashi Iwai1-0/+1
Since the commit a3da8821208d (pager: do wait_for_pager on signal death), we call wait_for_pager() in the pager's signal handler. The recent bug report revealed that this causes a deadlock in glibc at aborting "git log" [*1*]. When this happens, git process is left unterminated, and it can't be killed by SIGTERM but only by SIGKILL. The problem is that wait_for_pager() function does more than waiting for pager process's termination, but it does cleanups and printing errors. Unfortunately, the functions that may be used in a signal handler are very limited [*2*]. Particularly, malloc(), free() and the variants can't be used in a signal handler because they take a mutex internally in glibc. This was the cause of the deadlock above. Other than the direct calls of malloc/free, many functions calling malloc/free can't be used. strerror() is such one, either. Also the usage of fflush() and printf() in a signal handler is bad, although it seems working so far. In a safer side, we should avoid them, too. This patch tries to reduce the calls of such functions in signal handlers. wait_for_signal() takes a flag and avoids the unsafe calls. Also, finish_command_in_signal() is introduced for the same reason. There the free() calls are removed, and only waits for the children without whining at errors. [*1*] https://bugzilla.opensuse.org/show_bug.cgi?id=942297 [*2*] http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_04_03 Signed-off-by: Takashi Iwai <tiwai@suse.de> Reviewed-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-09-01run-command: provide in_async query functionLibravatar Jeff King1-0/+1
It's not easy for arbitrary code to find out whether it is running in an async process or not. A top-level function which is fed to start_async() can know (you just pass down an argument saying "you are async"). But that function may call other global functions, and we would not want to have to pass the information all the way through the call stack. Nor can we simply set a global variable, as those may be shared between async threads and the main thread (if the platform supports pthreads). We need pthread tricks _or_ a global variable, depending on how start_async is implemented. The callers don't have enough information to do this right, so let's provide a simple query function that does. Fortunately we can reuse the existing infrastructure to make the pthread case simple (and even simplify die_async() by using our new function). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-08-10find_hook: keep our own static bufferLibravatar Jeff King1-0/+5
The find_hook function returns the results of git_path, which is a static buffer shared by other path-related calls. Returning such a buffer is slightly dangerous, because it can be overwritten by seemingly unrelated functions. Let's at least keep our _own_ static buffer, so you can only get in trouble by calling find_hook in quick succession, which is less likely to happen and more obvious to notice. While we're at it, let's add some documentation of the function's limitations. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-05-11Merge branch 'nd/multiple-work-trees'Libravatar Junio C Hamano1-1/+1
A replacement for contrib/workdir/git-new-workdir that does not rely on symbolic links and make sharing of objects and refs safer by making the borrowee and borrowers aware of each other. * nd/multiple-work-trees: (41 commits) prune --worktrees: fix expire vs worktree existence condition t1501: fix test with split index t2026: fix broken &&-chain t2026 needs procondition SANITY git-checkout.txt: a note about multiple checkout support for submodules checkout: add --ignore-other-wortrees checkout: pass whole struct to parse_branchname_arg instead of individual flags git-common-dir: make "modules/" per-working-directory directory checkout: do not fail if target is an empty directory t2025: add a test to make sure grafts is working from a linked checkout checkout: don't require a work tree when checking out into a new one git_path(): keep "info/sparse-checkout" per work-tree count-objects: report unused files in $GIT_DIR/worktrees/... gc: support prune --worktrees gc: factor out gc.pruneexpire parsing code gc: style change -- no SP before closing parenthesis checkout: clean up half-prepared directories in --to mode checkout: reject if the branch is already checked out elsewhere prune: strategies for linked checkouts checkout: support checking out into a new working directory ...
2015-03-22run-command: introduce capture_command helperLibravatar Jeff King1-0/+13
Something as simple as reading the stdout from a command turns out to be rather hard to do right. Doing: cmd.out = -1; run_command(&cmd); strbuf_read(&buf, cmd.out, 0); can result in deadlock if the child process produces a large amount of output. What happens is: 1. The parent spawns the child with its stdout connected to a pipe, of which the parent is the sole reader. 2. The parent calls wait(), blocking until the child exits. 3. The child writes to stdout. If it writes more data than the OS pipe buffer can hold, the write() call will block. This is a deadlock; the parent is waiting for the child to exit, and the child is waiting for the parent to call read(). So we might try instead: start_command(&cmd); strbuf_read(&buf, cmd.out, 0); finish_command(&cmd); But that is not quite right either. We are examining cmd.out and running finish_command whether start_command succeeded or not, which is wrong. Moreover, these snippets do not do any error handling. If our read() fails, we must make sure to still call finish_command (to reap the child process). And both snippets failed to close the cmd.out descriptor, which they must do (provided start_command succeeded). Let's introduce a run-command helper that can make this a bit simpler for callers to get right. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-12-22Merge branch 'jc/hook-cleanup'Libravatar Junio C Hamano1-4/+0
Remove unused code. * jc/hook-cleanup: run-command.c: retire unused run_hook_with_custom_index()