summaryrefslogtreecommitdiff
path: root/t/test-lib.sh
AgeCommit message (Collapse)AuthorFilesLines
2019-05-19Merge branch 'tz/test-lib-check-working-jgit'Libravatar Junio C Hamano1-1/+1
A prerequiste check in the test suite to see if a working jgit is available was made more robust. * tz/test-lib-check-working-jgit: test-lib: try harder to ensure a working jgit
2019-05-15test-lib: try harder to ensure a working jgitLibravatar Todd Zullinger1-1/+1
The JGIT prereq uses `type jgit` to determine whether jgit is present. While this is usually sufficient, it won't help if the jgit found is badly broken. This wastes time running tests which fail due to no fault of our own. Use `jgit --version` instead, to guard against cases where jgit is present on the system, but will fail to run, e.g. because of some JRE issue, or missing Java dependencies. Checking that it gets far enough to process the '--version' argument isn't perfect, but seems to be good enough in practice. It's also consistent with how we detect some other dependencies, see e.g. the CURL and UNZIP prerequisites. Signed-off-by: Todd Zullinger <tmz@pobox.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-25Merge branch 'jc/gettext-test-fix'Libravatar Junio C Hamano1-0/+1
The GETTEXT_POISON test option has been quite broken ever since it was made runtime-tunable, which has been fixed. * jc/gettext-test-fix: gettext tests: export the restored GIT_TEST_GETTEXT_POISON
2019-04-25Merge branch 'ab/test-lib-pass-trace2-env'Libravatar Junio C Hamano1-0/+1
Allow tracing of Git executable while running the testsuite. * ab/test-lib-pass-trace2-env: test-lib: whitelist GIT_TR2_* in the environment
2019-04-25Merge branch 'sg/test-atexit'Libravatar Junio C Hamano1-1/+27
Test framework update to more robustly clean up leftover files and processes after tests are done. * sg/test-atexit: t9811-git-p4-label-import: fix pipeline negation git p4 test: disable '-x' tracing in the p4d watchdog loop git p4 test: simplify timeout handling git p4 test: clean up the p4d cleanup functions git p4 test: use 'test_atexit' to kill p4d and the watchdog process t0301-credential-cache: use 'test_atexit' to stop the credentials helper tests: use 'test_atexit' to stop httpd git-daemon: use 'test_atexit` to stop 'git-daemon' test-lib: introduce 'test_atexit' t/lib-git-daemon: make sure to kill the 'git-daemon' process test-lib: fix interrupt handling with 'dash' and '--verbose-log -x'
2019-04-15gettext tests: export the restored GIT_TEST_GETTEXT_POISONLibravatar Junio C Hamano1-0/+1
6cdccfce ("i18n: make GETTEXT_POISON a runtime option", 2018-11-08) made the gettext-poison test a runtime option (which was a good move) and adjusted the test framework so that Git commands we run as part of the framework, as opposed to the ones that are part of the test proper, are not affected by the setting. The original value for the GIT_TEST_GETTEXT_POISON environment variable is saved away in another variable and gets unset, and then later the saved value is restored to the environment variable. But the code forgot to export the variable again, which is necessary to restore the "export" bit that was lost when the variable was unset. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-15tests: disallow the use of abbreviated options (by default)Libravatar Johannes Schindelin1-0/+7
Git's command-line parsers support uniquely abbreviated options, e.g. `git init --ba` would automatically expand `--ba` to `--bare`. This is a very convenient feature in every day life for Git users, in particular when tab completion is not available. However, it is not a good idea to rely on that in Git's test suite, as something that is a unique abbreviation of a command line option today might no longer be a unique abbreviation tomorrow. For example, if a future contribution added a new mode `git init --babyproofing` and a previously-introduced test case used the fact that `git init --ba` expanded to `git init --bare`, that future contribution would now have to touch seemingly unrelated tests just to keep the test suite from failing. So let's disallow abbreviated options in the test suite by default. Note: for ease of implementation, this patch really only touches the `parse-options` machinery: more and more hand-rolled option parsers are converted to use that internal API, and more and more scripts are converted to built-ins (naturally using the parse-options API, too), so in practice this catches most issues, and is definitely the biggest bang for the buck. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-01test-lib: whitelist GIT_TR2_* in the environmentLibravatar Ævar Arnfjörð Bjarmason1-0/+1
Add GIT_TR2_* to the whitelist of environment variables that we don't clear when running the test suite. This allows us to use the test suite to produce trace2 test data, which is handy to e.g. write consumers that collate the trace data itself. One caveat here is that we produce trace output for not *just* the tests, but also e.g. from this line in test-lib.sh: # It appears that people try to run tests without building... "${GIT_TEST_INSTALLED:-$GIT_BUILD_DIR}/git$X" >/dev/null [...] I consider this not just OK but a feature. Let's log *all* the git commands we're going to execute, not just those within test_expect_*(). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-20Merge branch 'js/stress-test-ui-tweak'Libravatar Junio C Hamano1-1/+8
Dev support. * js/stress-test-ui-tweak: tests: introduce --stress-jobs=<N> tests: let --stress-limit=<N> imply --stress
2019-03-14test-lib: introduce 'test_atexit'Libravatar Johannes Schindelin1-0/+23
When running Apache, 'git daemon', or p4d, we want to kill them at the end of the test script, otherwise a leftover daemon process will keep its port open indefinitely, and thus will interfere with subsequent executions of the same test script. So far, we stop these daemon processes "manually", i.e.: - by registering functions or commands in the trap on EXIT to stop the daemon while preserving the last seen exit code before the trap (to deal with a failure when run with '--immediate' or with interrupts by ctrl-C), - and by invoking these functions/commands last thing before 'test_done' (and sometimes restoring the test framework's default trap on EXIT, to prevent the daemons from being killed twice). On one hand, we do this inconsistently, e.g. 'git p4' tests invoke different functions in the trap on EXIT and in the last test before 'test_done', and they neither restore the test framework's default trap on EXIT nor preserve the last seen exit code. On the other hand, this is error prone, because, as shown in a previous patch in this series, any output from the cleanup commands in the trap on EXIT can prevent a proper cleanup when a test script run with '--verbose-log' and certain shells, notably 'dash', is interrupted. Let's introduce 'test_atexit', which is loosely modeled after 'test_when_finished', but has a broader scope: rather than running the commands after the current test case, run them when the test script finishes, and also run them when the test is interrupted, or exits early in case of a failure while the '--immediate' option is in effect. When running the cleanup commands at the end of a successful test, then they will be run in 'test_done' before it removes the trash directory, i.e. the cleanup commands will still be able to access any pidfiles or socket files in there. When running the cleanup commands after an interrupt or failure with '--immediate', then they will be run in the trap on EXIT. In both cases they will be run in 'test_eval_', i.e. both standard error and output of all cleanup commands will go where they should according to the '-v' or '--verbose-log' options, and thus won't cause any troubles when interrupting a test script run with '--verbose-log'. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-14test-lib: fix interrupt handling with 'dash' and '--verbose-log -x'Libravatar SZEDER Gábor1-1/+4
When a test script run with 'dash' and '--verbose-log -x' is interrupted by ctrl-C, SIGTERM, or closing the terminal window, then most of the time the registered EXIT trap actions are not executed. This is an annoying issue with tests involving daemons, because they should run cleanup commands to kill those daemon processes in the trap on EXIT, but since these cleanup commands are not executed, the daemons are left alive and keep their port open, thus interfering with subsequent execution of the same test script. The cause of this issue is the subtle combination of several factors (bear with me, or skip over the indented part): - Even when the test script is interrupted, the cleanup commands are not run in the trap on INT, TERM, or HUP, but in the trap on EXIT after the trap on the signals invokes 'exit' [1]. - According to POSIX [2]: "The environment in which the shell executes a trap on EXIT shall be identical to the environment immediately after the last command executed before the trap on EXIT was taken." Pertinent to the issue at hand is that all open file descriptors and the state of '-x' tracing should be preserved. All shells I've tried [3] preserve '-x'. Unfortunately, however: - 'dash' doesn't conform to this when it comes to open file descriptors: even when standard output and/or error are redirected somewhere when 'exit' is invoked, anything written to them in the trap on EXIT goes to the script's original stdout and stderr [4]. We can't dismiss this with a simple "it doesn't conform to POSIX, so we don't care", because 'dash' is the default /bin/sh in some of the more popular Linux distros. - As far as I can tell, POSIX doesn't explicitly say anything about the environment of trap actions for various signals. In practice it seems that most shells behave sensibly and preserve both open file descriptors and the state of '-x' tracing for the traps on INT, TERM, and HUP, including even 'dash'. The exceptions are 'mksh' and 'lksh': they do preserve '-x', but not the open file descriptors. - When a test script run with '-x' tracing enabled is interrupted, then it's very likely that the signal arrives mid-test, i.e.: - while '-x' tracing is enabled, and, consequently, our trap actions on INT, TERM, HUP, and EXIT will produce trace output as well. - while standard output and error are redirected to a log file, to the test script's original standard output and error, or to /dev/null, depending on whether the test script was run with '--verbose-log', '-v', or neither. According to the above, we can't rely on these redirections still be in effect when running the traps on INT, TERM, HUP, and/or EXIT. - When a test script is run with '--verbose-log', then the test script is re-executed with its standard output and error piped into 'tee', in order to send the "regular" non-verbose test's output both to the terminal and to the log file. When the test is interrupted, then the signal interrupts the downstream 'tee' as well. Putting these together, when a test script run with 'dash' and '--verbose-log -x' is interrupted, then 'dash' tries to write the trace output from the EXIT trap to the script's original standard error, but it very likely can't, because the 'tee' downstream of the pipe is interrupted as well. This causes the shell running the test script to die because of SIGPIPE, without running any of the commands in the EXIT trap. Disable '-x' tracing in the trap on INT, TERM, and HUP to avoid this issue, as it disables tracing in the chained trap on EXIT as well. Wrap it in a '{ ... } 2>/dev/null' block, so the trace of the command disabling the tracing doesn't go to standard error either [5]. Note that it's not only '-x' tracing that can be problematic, but any shell builtin, e.g. 'echo', that writes to standard output or error in the trap on EXIT, while a test running with 'dash' and '--verbose-log' (even without '-x') is interrupted. As far as I can tell, this is not an issue at the moment: - The cleanup commands to stop the credential-helper, Apache, or 'p4d' don't use any such shell builtins. - stop_git_daemon() does use 'say' and 'error', both wrappers around 'echo', but it redirects 'say' to fd 3, i.e. to the log file, and while 'error' does write to standard output, it comes only after the daemon was killed. - The non-builtin commands that actually stop the daemons ('kill', 'apache2 -k stop', 'git credential-cache exit') are silent, so they won't get SIGPIPE before finishing their job. [1] The trap on EXIT must run cleanup commands, because we want to stop any daemons when a test script run with '--immediate' fails and exits early with error. By chaining up the trap on signals to the trap on EXIT we can deal with cleanup commands a bit simpler, because the tests involving daemons only have to set a single trap. [2] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#trap [3] The shells I tried: dash, Bash, ksh, ksh93, mksh, lksh, yash, BusyBox sh, FreeBSD /bin/sh, NetBSD /bin/sh. [4] $ cat trap-output.sh #!/bin/sh trap "echo output; echo error >&2" EXIT { exit; } >OUT 2>ERR $ dash ./trap-output.sh output error $ wc -c OUT ERR 0 OUT 0 ERR On a related note, 'ksh', 'ksh93', and BusyBox sh don't conform to the specs in this respect, either. [5] This '{ set +x; } 2>/dev/null' trick won't help those shells that show trace output for any redirections and don't preserve open file descriptors for the trap on INT, TERM and HUP. The only such shells I'm aware of are 'mksh' and 'lksh'. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-07Merge branch 'jc/test-yes-doc'Libravatar Junio C Hamano1-1/+5
Test doc update. * jc/test-yes-doc: test: caution on our version of 'yes'
2019-03-04tests: introduce --stress-jobs=<N>Libravatar Johannes Schindelin1-1/+7
The --stress option currently accepts an argument, but it is confusing to at least this user that the argument does not define the maximal number of stress iterations, but instead the number of jobs to run in parallel per stress iteration. Let's introduce a separate option for that, whose name makes it more obvious what it is about, and let --stress=<N> error out with a helpful suggestion about the two options tha could possibly have been meant. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-04tests: let --stress-limit=<N> imply --stressLibravatar Johannes Schindelin1-0/+1
It does not make much sense that running a test with --stress-limit=<N> seemingly ignores that option because it does not stress test at all. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-13Merge branch 'sg/stress-test'Libravatar Junio C Hamano1-3/+17
Test improvement. * sg/stress-test: test-lib: fix non-portable pattern bracket expressions test-lib: make '--stress' more bisect-friendly
2019-02-11test-lib: fix non-portable pattern bracket expressionsLibravatar SZEDER Gábor1-2/+2
Use a '!' character to start a non-matching pattern bracket expression, as specified by POSIX in Shell Command Language section 2.13.1 Patterns Matching a Single Character [1]. I used '^' instead in three places in the previous three commits, to verify that the arguments of the '--stress=' and '--stress-limit=' options and the values of various '*_PORT' environment variables are valid numbers. With certain shells, at least with dash (upstream and in Ubuntu 14.04) and mksh, this led to various undesired behaviors: # error message in case of a valid number $ ~/src/dash/src/dash ./t3903-stash.sh --stress=8 error: --stress=<N> requires the number of jobs to run # not the expected error message $ ~/src/dash/src/dash ./t3903-stash.sh --stress=foo ./t3903-stash.sh: 238: test: Illegal number: foo # no error message at all?! $ mksh ./t3903-stash.sh --stress=foo $ echo $? 0 Some other shells, e.g. Bash (even in posix mode), ksh, dash in Ubuntu 16.04 or later, are apparently happy to accept '^' just as well. [1] http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_13 Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-11test: caution on our version of 'yes'Libravatar Junio C Hamano1-1/+5
During a review of a patch, we noticed that we use our own imitation of 'yes' with the limit of 99 lines. It is very tempting to lift this arbitrary limit, but the limit is there for a reason. Add an in-code comment to prevent future developers from wasting their time. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-08test-lib: make '--stress' more bisect-friendlyLibravatar SZEDER Gábor1-2/+16
Let's suppose that a test somehow becomes flaky between 'master' and 'pu', and tends to fail within the first 50 repetitions when run with '--stress'. In such a case we could use 'git bisect' to find the culprit: if the test script fails with '--stress', then the commit is definitely bad, but if it survives, say, 300 repetitions, then we could consider it good with reasonable confidence. Unfortunately, all this could only be done manually, because '--stress' would run the test script repeatedly for all eternity on a good commit, and it would exit with success even when it found a failure on a bad commit. So let's make '--stress' usable with 'git bisect run': - Make it exit with failure if a failure is found. - Add the '--stress-limit=<N>' option to repeat the test script at most N times in each of the parallel jobs, and exit with success when the limit is reached. And then we could simply run something like: $ git bisect start origin/pu master $ git bisect run sh -c 'make && cd t && ./t1234-foo.sh --stress --stress-limit=300' Sure, as a brand new feature it won't be any useful right now, but in a release or three most cooking topics will already contain this, so we could automatically bisect at least newly introduced flakiness. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-06Merge branch 'js/vsts-ci'Libravatar Junio C Hamano1-8/+130
Prepare to run test suite on Azure Pipeline. * js/vsts-ci: (22 commits) test-date: drop unused parameter to getnanos() ci: parallelize testing on Windows ci: speed up Windows phase tests: optionally skip bin-wrappers/ t0061: workaround issues with --with-dashes and RUNTIME_PREFIX tests: add t/helper/ to the PATH with --with-dashes mingw: try to work around issues with the test cleanup tests: include detailed trace logs with --write-junit-xml upon failure tests: avoid calling Perl just to determine file sizes README: add a build badge (status of the Azure Pipelines build) mingw: be more generous when wrapping up the setitimer() emulation ci: use git-sdk-64-minimal build artifact ci: add a Windows job to the Azure Pipelines definition Add a build definition for Azure DevOps ci/lib.sh: add support for Azure Pipelines tests: optionally write results as JUnit-style .xml test-date: add a subcommand to measure times in shell scripts ci: use a junction on Windows instead of a symlink ci: inherit --jobs via MAKEFLAGS in run-build-and-tests ci/lib.sh: encapsulate Travis-specific things ...
2019-02-05Merge branch 'js/test-git-installed'Libravatar Junio C Hamano1-1/+1
Test fix for Windows. * js/test-git-installed: tests: explicitly use `test-tool.exe` on Windows
2019-01-29tests: optionally skip bin-wrappers/Libravatar Johannes Schindelin1-6/+13
This speeds up the tests by a bit on Windows, where running Unix shell scripts (and spawning processes) is not exactly a cheap operation. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29tests: add t/helper/ to the PATH with --with-dashesLibravatar Johannes Schindelin1-1/+1
We really need to be able to find the test helpers... Really. This change was forgotten when we moved the test helpers into t/helper/ Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29mingw: try to work around issues with the test cleanupLibravatar Johannes Schindelin1-1/+5
It seems that every once in a while in the Git for Windows SDK, there are some transient file locking issues preventing the test clean up to delete the trash directory. Let's be gentle and try again five seconds later, and only error out if it still fails the second time. This change helps Windows, and does not hurt any other platform (normally, it is highly unlikely that said deletion fails, and if it does, normally it will fail again even 5 seconds later). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29tests: include detailed trace logs with --write-junit-xml upon failureLibravatar Johannes Schindelin1-1/+21
The JUnit XML format lends itself to be presented in a powerful UI, where you can drill down to the information you are interested in very quickly. For test failures, this usually means that you want to see the detailed trace of the failing tests. With Travis CI, we passed the `--verbose-log` option to get those traces. However, that seems excessive, as we do not need/use the logs in almost all of those cases: only when a test fails do we have a way to include the trace. So let's do something different when using Azure DevOps: let's run all the tests with `--quiet` first, and only if a failure is encountered, try to trace the commands as they are executed. Of course, we cannot turn on `--verbose-log` after the fact. So let's just re-run the test with all the same options, adding `--verbose-log`. And then munging the output file into the JUnit XML on the fly. Note: there is an off chance that re-running the test in verbose mode "fixes" the failures (and this does happen from time to time!). That is a possibility we should be able to live with. Ideally, we would label this as "Passed upon rerun", and Azure Pipelines even know about that outcome, but it is not available when using the JUnit XML format for now: https://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29tests: optionally write results as JUnit-style .xmlLibravatar Johannes Schindelin1-0/+91
This will come in handy when publishing the results of Git's test suite during an automated Azure DevOps run. Note: we need to make extra sure that invalid UTF-8 encoding is turned into valid UTF-8 (using the Replacement Character, \uFFFD) because t9902's trace contains such invalid byte sequences, and the task in the Azure Pipeline that uploads the test results would refuse to do anything if it was asked to parse an .xml file with invalid UTF-8 in it. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-22tests: explicitly use `test-tool.exe` on WindowsLibravatar Johannes Schindelin1-1/+1
In 8abfdf44c882 (tests: explicitly use `git.exe` on Windows, 2018-11-14), we made sure to use the `.exe` file extension when using an absolute path to `git.exe`, to avoid getting confused with a file or directory in the same place that lacks said file extension. For the same reason, we need to handle test-tool.exe the same way. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: add the '--stress' option to run a test repeatedly under loadLibravatar SZEDER Gábor1-2/+107
Unfortunately, we have a few flaky tests, whose failures tend to be hard to reproduce. We've found that the best we can do to reproduce such a failure is to run the test script repeatedly while the machine is under load, and wait in the hope that the load creates enough variance in the timing of the test's commands that a failure is evenually triggered. I have a command to do that, and I noticed that two other contributors have rolled their own scripts to do the same, all choosing slightly different approaches. To help reproduce failures in flaky tests, introduce the '--stress' option to run a test script repeatedly in multiple parallel jobs until one of them fails, thereby using the test script itself to increase the load on the machine. The number of parallel jobs is determined by, in order of precedence: the number specified as '--stress=<N>', or the value of the GIT_TEST_STRESS_LOAD environment variable, or twice the number of available processors (as reported by the 'getconf' utility), or 8. Make '--stress' imply '--verbose -x --immediate' to get the most information about rare failures; there is really no point in spending all the extra effort to reproduce such a failure, and then not know which command failed and why. To prevent the several parallel invocations of the same test from interfering with each other: - Include the parallel job's number in the name of the trash directory and the various output files under 't/test-results/' as a '.stress-<Nr>' suffix. - Add the parallel job's number to the port number specified by the user or to the test number, so even tests involving daemons listening on a TCP socket can be stressed. - Redirect each parallel test run's verbose output to 't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the output of several parallel running tests to the terminal would create a big ugly mess. For convenience, print the output of the failed test job at the end, and rename its trash directory to end with the '.stress-failed' suffix, so it's easy to find in a predictable path (OTOH, all absolute paths recorded in the trash directory become invalid; we'll see whether this causes any issues in practice). If, in an unlikely case, more than one jobs were to fail nearly at the same time, then print the output of all failed jobs, and rename the trash directory of only the last one (i.e. with the highest job number), as it is the trash directory of the test whose output will be at the bottom of the user's terminal. Based on Jeff King's 'stress' script. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: set $TRASH_DIRECTORY earlierLibravatar SZEDER Gábor1-6/+6
A later patch in this series will need to know the path to the trash directory early in 'test-lib.sh', but $TRASH_DIRECTORY is set much later. Set $TRASH_DIRECTORY earlier, where the other test-specific path variables are set. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: consolidate naming of test-results pathsLibravatar SZEDER Gábor1-11/+11
There are two places where we strip off any leading path components and the '.sh' suffix from the test script's pathname, and there are four places where we construct the name of the 't/test-results' directory or the name of various test-specific files in there. The last patch in this series will add even more. Factor these out into helper variables to avoid repeating ourselves. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: parse command line options earlierLibravatar SZEDER Gábor1-109/+124
'test-lib.sh' looks for the presence of certain options like '--tee' and '--verbose-log', so it can execute the test script again to save its standard output and error. It looks for '--valgrind' as well, to set up some Valgrind-specific stuff. These all happen before the actual option parsing loop, and the conditions looking for these options look a bit odd, too. They are not completely correct, either, because in a bogus invocation like './t1234-foo.sh -r --tee' they recognize '--tee', although it should be handled as the required argument of the '-r' option. This patch series will add two more options to look out for early, and, in addition, will have to extract these options' stuck arguments (i.e. '--opt=arg') as well. So let's move the option parsing loop and the couple of related conditions following it earlier in 'test-lib.sh', before the place where the test script is executed again for '--tee' and its friends. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: parse options in a for loop to keep $@ intactLibravatar SZEDER Gábor1-36/+42
'test-lib.sh' looks for the presence of certain options like '--tee' and '--verbose-log', so it can execute the test script again to save its standard output and error, and to do so it needs the original command line options the test was invoked with. The next patch is about to move the option parsing loop earlier in 'test-lib.sh', but it is implemented using 'shift' in a while loop, effecively destroying "$@" by the end of the option parsing. Not good. As a preparatory step, turn that option parsing loop into a 'for opt in "$@"' loop to preserve "$@" intact while iterating over the options, and taking extra care to handle the '-r' option's required argument (or the lack thereof). Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: extract Bash version check for '-x' tracingLibravatar SZEDER Gábor1-18/+19
One of our test scripts, 't1510-repo-setup.sh' [1], still can't be reliably run with '-x' tracing enabled, unless it's executed with a Bash version supporting BASH_XTRACEFD (since v4.1). We have a lengthy condition to check the version of the shell running the test script, and disable tracing if it's not executed with a suitable Bash version [2]. Move this check out from the option parsing loop, so other options can imply '-x' by setting 'trace=t', without missing this Bash version check. [1] 5827506928 (t1510-repo-setup: mark as untraceable with '-x', 2018-02-24) [2] 5fc98e79fc (t: add means to disable '-x' tracing for individual test scripts, 2018-02-24) Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-03test-lib: translate SIGTERM and SIGHUP to an exitLibravatar SZEDER Gábor1-1/+1
Right now if a test script receives SIGTERM or SIGHUP (e.g., because a test was hanging and the user 'kill'-ed it or simply closed the terminal window the test was running in), the shell exits immediately. This can be annoying if the test script did any global setup, like starting apache or git-daemon, as it will not have an opportunity to clean up after itself. A subsequent run of the test won't be able to start its own daemon, and will either fail or skip the tests. Instead, let's trap SIGTERM and SIGHUP as well to make sure we do a clean shutdown, and just chain it to a normal exit (which will trigger any cleanup). This patch follows suit of da706545f7 (t: translate SIGINT to an exit, 2015-03-13), and even stole its commit message as well. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-03Merge branch 'sg/test-bash-version-fix'Libravatar Junio C Hamano1-2/+2
* sg/test-bash-version-fix: test-lib: check Bash version for '-x' without using shell arrays
2019-01-03test-lib: check Bash version for '-x' without using shell arraysLibravatar SZEDER Gábor1-2/+2
One of our test scripts, 't1510-repo-setup.sh' [1], still can't be reliably run with '-x' tracing enabled, unless it's executed with a Bash version supporting BASH_XTRACEFD (since v4.1). We have a lengthy condition to check the version of the shell running the test script, and disable tracing if it's not executed with a suitable Bash version [2]. This condition uses non-portable shell array accesses to easily get Bash's major and minor version number. This didn't seem to be problematic, because the simple commands expanding those array accesses are only executed when the test script is actually run with Bash. When run with Dash, the only shell I have at hand that doesn't support shell arrays, there are no issues, as it apparently skips right over the non-executed simple commands without noticing the non-supported constructs. Alas, it has been reported that NetBSD's /bin/sh does complain about them: ./test-lib.sh: 327: Syntax error: Bad substitution where line 327 contains the first ${BASH_VERSINFO[0]} array access. To my understanding both shells are right and conform to POSIX, because the standard allows both behaviors by stating the following under '2.8.1 Consequences of Shell Errors' [3]: "An expansion error is one that occurs when the shell expansions define in wordexp are carried out (for example, "${x!y}", because '!' is not a valid operator); an implementation may treat these as syntax errors if it is able to detect them during tokenization, rather than during expansion." Avoid this issue with NetBSD's /bin/sh (and potentially with other, less common shells) by hiding the shell array syntax behind 'eval' that is only executed with Bash. [1] 5827506928 (t1510-repo-setup: mark as untraceable with '-x', 2018-02-24) [2] 5fc98e79fc (t: add means to disable '-x' tracing for individual test scripts, 2018-02-24) [3] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_08_01 Reported-by: Max Kirillov <max@max630.net> Helped-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-20tests: send "bug in the test script" errors to the script's stderrLibravatar SZEDER Gábor1-3/+7
Some of the functions in our test library check that they were invoked properly with conditions like this: test "$#" = 2 || error "bug in the test script: not 2 parameters to test-expect-success" If this particular condition is triggered, then 'error' will abort the whole test script with a bold red error message [1] right away. However, under certain circumstances the test script will be aborted completely silently, namely if: - a similar condition in a test helper function like 'test_line_count' is triggered, - which is invoked from the test script's "main" shell [2], - and the test script is run manually (i.e. './t1234-foo.sh' as opposed to 'make t1234-foo.sh' or 'make test') [3] - and without the '--verbose' option, because the error message is printed from within 'test_eval_', where standard output is redirected either to /dev/null or to a log file. The only indication that something is wrong is that not all tests in the script are executed and at the end of the test script's output there is no "# passed all N tests" message, which are subtle and can easily go unnoticed, as I had to experience myself. Send these "bug in the test script" error messages directly to the test scripts standard error and thus to the terminal, so those bugs will be much harder to overlook. Instead of updating all ~20 such 'error' calls with a redirection, let's add a BUG() function to 'test-lib.sh', wrapping an 'error' call with the proper redirection and also including the common prefix of those error messages, and convert all those call sites [4] to use this new BUG() function instead. [1] That particular error message from 'test_expect_success' is printed in color only when running with or without '--verbose'; with '--tee' or '--verbose-log' the error is printed without color, but it is printed to the terminal nonetheless. [2] If such a condition is triggered in a subshell of a test, then 'error' won't be able to abort the whole test script, but only the subshell, which in turn causes the test to fail in the usual way, indicating loudly and clearly that something is wrong. [3] Well, 'error' aborts the test script the same way when run manually or by 'make' or 'prove', but both 'make' and 'prove' pay attention to the test script's exit status, and even a silently aborted test script would then trigger those tools' usual noticable error messages. [4] Strictly speaking, not all those 'error' calls need that redirection to send their output to the terminal, see e.g. 'test_expect_success' in the opening example, but I think it's better to be consistent. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-19Merge branch 'js/test-git-installed'Libravatar Junio C Hamano1-6/+16
Update the "test installed Git" mode of our test suite to work better. * js/test-git-installed: tests: explicitly use `git.exe` on Windows tests: do not require Git to be built when testing an installed Git t/lib-gettext: test installed git-sh-i18n if GIT_TEST_INSTALLED is set tests: respect GIT_TEST_INSTALLED when initializing repositories tests: fix GIT_TEST_INSTALLED's PATH to include t/helper/
2018-11-19Merge branch 'ab/dynamic-gettext-poison'Libravatar Junio C Hamano1-5/+17
Our testing framework uses a special i18n "poisoned localization" feature to find messages that ought to stay constant but are incorrectly marked to be translated. This feature has been made into a runtime option (it used to be a compile-time option). * ab/dynamic-gettext-poison: Makefile: ease dynamic-gettext-poison transition i18n: make GETTEXT_POISON a runtime option
2018-11-16tests: explicitly use `git.exe` on WindowsLibravatar Johannes Schindelin1-4/+9
On Windows, when we refer to `/an/absolute/path/to/git`, it magically resolves `git.exe` at that location. Except if something of the name `git` exists next to that `git.exe`. So if we call `$BUILD_DIR/git`, it will find `$BUILD_DIR/git.exe` *only* if there is not, say, a directory called `$BUILD_DIR/git`. Such a directory, however, exists in Git for Windows when building with Visual Studio (our Visual Studio project generator defaults to putting the build files into a directory whose name is the base name of the corresponding `.exe`). In the bin-wrappers/* scripts, we already take pains to use `git.exe` rather than `git`, as this could pick up the wrong thing on Windows (i.e. if there exists a `git` file or directory in the build directory). Now we do the same in the tests' start-up code. This also helps when testing an installed Git, as there might be even more likely some stray file or directory in the way. Note: the only way we can record whether the `.exe` suffix is by writing it to the `GIT-BUILD-OPTIONS` file and sourcing it at the beginning of `t/test-lib.sh`. This is not a requirement introduced by this patch, but we move the call to be able to use the `$X` variable that holds the file extension, if any. Note also: the many, many calls to `git this` and `git that` are unaffected, as the regular PATH search will find the `.exe` files on Windows (and not be confused by a directory of the name `git` that is in one of the directories listed in the `PATH` variable), while `/path/to/git` would not, per se, know that it is looking for an executable and happily prefer such a directory. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-16tests: do not require Git to be built when testing an installed GitLibravatar Johannes Schindelin1-2/+7
We really only need the test helpers to be built in the worktree in that case, but that is not what we test for. On the other hand it is a perfect opportunity to verify that `GIT_TEST_INSTALLED` points to a working Git. So let's test the appropriate Git executable. While at it, also adjust the error message in the `GIT_TEST_INSTALLED` case. This patch is best viewed with `-w --patience`. Helped-by: Jeff King <peff@peff.net> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-14tests: fix GIT_TEST_INSTALLED's PATH to include t/helper/Libravatar Johannes Schindelin1-1/+1
We really need to be able to find the test helpers... Really. This change was forgotten when we moved the test helpers into t/helper/ Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-13Merge branch 'js/rebase-p-tests'Libravatar Junio C Hamano1-0/+4
In preparation to the day when we can deprecate and remove the "rebase -p", make sure we can skip and later remove tests for it. * js/rebase-p-tests: tests: optionally skip `git rebase -p` tests t3418: decouple test cases from a previous `rebase -p` test case t3404: decouple some test cases from outcomes of previous test cases
2018-11-09i18n: make GETTEXT_POISON a runtime optionLibravatar Ævar Arnfjörð Bjarmason1-5/+17
Change the GETTEXT_POISON compile-time + runtime GIT_GETTEXT_POISON test parameter to only be a GIT_TEST_GETTEXT_POISON=<non-empty?> runtime parameter, to be consistent with other parameters documented in "Running tests with special setups" in t/README. When I added GETTEXT_POISON in bb946bba76 ("i18n: add GETTEXT_POISON to simulate unfriendly translator", 2011-02-22) I was concerned with ensuring that the _() function would get constant folded if NO_GETTEXT was defined, and likewise that GETTEXT_POISON would be compiled out unless it was defined. But as the benchmark in my [1] shows doing a one-off runtime getenv("GIT_TEST_[...]") is trivial, and since GETTEXT_POISON was originally added the GIT_TEST_* env variables have become the common idiom for turning on special test setups. So change GETTEXT_POISON to work the same way. Now the GETTEXT_POISON=YesPlease compile-time option is gone, and running the tests with GIT_TEST_GETTEXT_POISON=[YesPlease|] can be toggled on/off without recompiling. This allows for conditionally amending tests to test with/without poison, similar to what 859fdc0c3c ("commit-graph: define GIT_TEST_COMMIT_GRAPH", 2018-08-29) did for GIT_TEST_COMMIT_GRAPH. Do some of that, now we e.g. always run the t0205-gettext-poison.sh test. I did enough there to remove the GETTEXT_POISON prerequisite, but its inverse C_LOCALE_OUTPUT is still around, and surely some tests using it can be converted to e.g. always set GIT_TEST_GETTEXT_POISON=. Notes on the implementation: * We still compile a dedicated GETTEXT_POISON build in Travis CI. Perhaps this should be revisited and integrated into the "linux-gcc" build, see ae59a4e44f ("travis: run tests with GIT_TEST_SPLIT_INDEX", 2018-01-07) for prior art in that area. Then again maybe not, see [2]. * We now skip a test in t0000-basic.sh under GIT_TEST_GETTEXT_POISON=YesPlease that wasn't skipped before. This test relies on C locale output, but due to an edge case in how the previous implementation of GETTEXT_POISON worked (reading it from GIT-BUILD-OPTIONS) wasn't enabling poison correctly. Now it does, and needs to be skipped. * The getenv() function is not reentrant, so out of paranoia about code of the form: printf(_("%s"), getenv("some-env")); call use_gettext_poison() in our early setup in git_setup_gettext() so we populate the "poison_requested" variable in a codepath that's won't suffer from that race condition. * We error out in the Makefile if you're still saying GETTEXT_POISON=YesPlease to prompt users to change their invocation. * We should not print out poisoned messages during the test initialization itself to keep it more readable, so the test library hides the variable if set in $GIT_TEST_GETTEXT_POISON_ORIG during setup. See [3]. See also [4] for more on the motivation behind this patch, and the history of the GETTEXT_POISON facility. 1. https://public-inbox.org/git/871s8gd32p.fsf@evledraar.gmail.com/ 2. https://public-inbox.org/git/20181102163725.GY30222@szeder.dev/ 3. https://public-inbox.org/git/20181022202241.18629-2-szeder.dev@gmail.com/ 4. https://public-inbox.org/git/878t2pd6yu.fsf@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-06Merge branch 'sg/test-verbose-log'Libravatar Junio C Hamano1-2/+2
Our test scripts can now take the '-V' option as a synonym for the '--verbose-log' option. * sg/test-verbose-log: test-lib: introduce the '-V' short option for '--verbose-log'
2018-11-02tests: optionally skip `git rebase -p` testsLibravatar Johannes Schindelin1-0/+4
The `--preserve-merges` mode of the `rebase` command is slated to be deprecated soon, as the more powerful `--rebase-merges` mode is available now, and the latter was designed with the express intent to address the shortcomings of `--preserve-merges`' design (e.g. the inability to reorder commits in an interactive rebase). As such, we will eventually even remove the `--preserve-merges` support, and along with it, its tests. In preparation for this, and also to allow the Windows phase of our automated tests to save some well-needed time when running the test suite, this commit introduces a new prerequisite REBASE_P, which can be forced to being unmet by setting the environment variable `GIT_TEST_SKIP_REBASE_P` to any non-empty string. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-30test-lib: introduce the '-V' short option for '--verbose-log'Libravatar SZEDER Gábor1-2/+2
'--verbose-log' is one of the most useful and thus most frequently used test options, but due to its length it's a pain to type on the command line. Let's introduce the corresponding short option '-V' to save some keystrokes. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-19Merge branch 'bp/rename-test-env-var'Libravatar Junio C Hamano1-2/+33
Some environment variables that control the runtime options of Git used during tests are getting renamed for consistency. * bp/rename-test-env-var: t0000: do not get self-test disrupted by environment warnings preload-index: update GIT_FORCE_PRELOAD_TEST support read-cache: update TEST_GIT_INDEX_VERSION support fsmonitor: update GIT_TEST_FSMONITOR support preload-index: use git_env_bool() not getenv() for customization t/README: correct spelling of "uncommon"
2018-09-28t0000: do not get self-test disrupted by environment warningsLibravatar Junio C Hamano1-0/+8
The test framework test-lib.sh itself would want to give warnings and hints, e.g. when it sees a deprecated environment variable is in use that we want to encourage users to migrate to another variable. The self-test of test framework done in t0000 however do not expect to see these warnings and hints, so depending on the settings of environment variables, a running test may or may not produce these messages to the standard error output, breaking the expectations of self-test test framework does on itself. Here is what we see: $ TEST_GIT_INDEX_VERSION=4 sh t0000-basic.sh -i -v ... 'err' is not empty, it contains: warning: TEST_GIT_INDEX_VERSION is now GIT_TEST_INDEX_VERSION hint: set GIT_TEST_INDEX_VERSION too during the transition period not ok 5 - pretend we have a fully passing test suite The following quick attempt to work it around does not work, because some tests in t0000 do want to see expected errors from the test framework itself. t/t0000-basic.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh index 850f651e4e..88c6ed4696 100755 --- a/t/t0000-basic.sh +++ b/t/t0000-basic.sh @@ -88,7 +88,7 @@ _run_sub_test_lib_test_common () { ' # Point to the t/test-lib.sh, which isn't in ../ as usual - . "\$TEST_DIRECTORY"/test-lib.sh + . "\$TEST_DIRECTORY"/test-lib.sh >/dev/null 2>&1 EOF cat >>"$name.sh" && chmod +x "$name.sh" && There are a few possible ways to work this around: * We could strip the warning: and hint: unconditionally from the error output before the error messages are checked in the self-test (helper functions check_sub_test_lib_test_err and check_sub_test_lib_test); the problem with this approach is that it will make it impossible to write self-tests to ensure that right warnings and hints are given. * We could force a sane environment settings before the test helper _run_sub_test_lib_test_common dot-sources test-lib.sh; the problem with this approach is that _run_sub_test_lib_test_common now needs to be aware of what pairs of environment variables are checked in test-lib.sh using check_var_migration helper. The final patch I came up with is probably the solution that is least bad. Set a variable to tell test-lib.sh that we are running a self-test, so that various pieces in test-lib.sh can react to keep the output stable. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-28preload-index: update GIT_FORCE_PRELOAD_TEST supportLibravatar Ben Peart1-0/+1
Rename GIT_FORCE_PRELOAD_TEST to GIT_TEST_PRELOAD_INDEX for consistency with the other GIT_TEST_ special setups and properly document its use. Add logic in t/test-lib.sh to give a warning when the old variable is set to let people know they need to update their environment to use the new variable. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-28read-cache: update TEST_GIT_INDEX_VERSION supportLibravatar Ben Peart1-6/+8
Rename TEST_GIT_INDEX_VERSION to GIT_TEST_INDEX_VERSION for consistency with the other GIT_TEST_ special setups and properly document its use. Add logic in t/test-lib.sh to give a warning when the old variable is set to let people know they need to update their environment to use the new variable. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>