summaryrefslogtreecommitdiff
path: root/t/test-lib.sh
AgeCommit message (Collapse)AuthorFilesLines
2019-03-20Merge branch 'js/stress-test-ui-tweak'Libravatar Junio C Hamano1-1/+8
Dev support. * js/stress-test-ui-tweak: tests: introduce --stress-jobs=<N> tests: let --stress-limit=<N> imply --stress
2019-03-07Merge branch 'jc/test-yes-doc'Libravatar Junio C Hamano1-1/+5
Test doc update. * jc/test-yes-doc: test: caution on our version of 'yes'
2019-03-04tests: introduce --stress-jobs=<N>Libravatar Johannes Schindelin1-1/+7
The --stress option currently accepts an argument, but it is confusing to at least this user that the argument does not define the maximal number of stress iterations, but instead the number of jobs to run in parallel per stress iteration. Let's introduce a separate option for that, whose name makes it more obvious what it is about, and let --stress=<N> error out with a helpful suggestion about the two options tha could possibly have been meant. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-04tests: let --stress-limit=<N> imply --stressLibravatar Johannes Schindelin1-0/+1
It does not make much sense that running a test with --stress-limit=<N> seemingly ignores that option because it does not stress test at all. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-13Merge branch 'sg/stress-test'Libravatar Junio C Hamano1-3/+17
Test improvement. * sg/stress-test: test-lib: fix non-portable pattern bracket expressions test-lib: make '--stress' more bisect-friendly
2019-02-11test-lib: fix non-portable pattern bracket expressionsLibravatar SZEDER Gábor1-2/+2
Use a '!' character to start a non-matching pattern bracket expression, as specified by POSIX in Shell Command Language section 2.13.1 Patterns Matching a Single Character [1]. I used '^' instead in three places in the previous three commits, to verify that the arguments of the '--stress=' and '--stress-limit=' options and the values of various '*_PORT' environment variables are valid numbers. With certain shells, at least with dash (upstream and in Ubuntu 14.04) and mksh, this led to various undesired behaviors: # error message in case of a valid number $ ~/src/dash/src/dash ./t3903-stash.sh --stress=8 error: --stress=<N> requires the number of jobs to run # not the expected error message $ ~/src/dash/src/dash ./t3903-stash.sh --stress=foo ./t3903-stash.sh: 238: test: Illegal number: foo # no error message at all?! $ mksh ./t3903-stash.sh --stress=foo $ echo $? 0 Some other shells, e.g. Bash (even in posix mode), ksh, dash in Ubuntu 16.04 or later, are apparently happy to accept '^' just as well. [1] http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_13 Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-11test: caution on our version of 'yes'Libravatar Junio C Hamano1-1/+5
During a review of a patch, we noticed that we use our own imitation of 'yes' with the limit of 99 lines. It is very tempting to lift this arbitrary limit, but the limit is there for a reason. Add an in-code comment to prevent future developers from wasting their time. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-08test-lib: make '--stress' more bisect-friendlyLibravatar SZEDER Gábor1-2/+16
Let's suppose that a test somehow becomes flaky between 'master' and 'pu', and tends to fail within the first 50 repetitions when run with '--stress'. In such a case we could use 'git bisect' to find the culprit: if the test script fails with '--stress', then the commit is definitely bad, but if it survives, say, 300 repetitions, then we could consider it good with reasonable confidence. Unfortunately, all this could only be done manually, because '--stress' would run the test script repeatedly for all eternity on a good commit, and it would exit with success even when it found a failure on a bad commit. So let's make '--stress' usable with 'git bisect run': - Make it exit with failure if a failure is found. - Add the '--stress-limit=<N>' option to repeat the test script at most N times in each of the parallel jobs, and exit with success when the limit is reached. And then we could simply run something like: $ git bisect start origin/pu master $ git bisect run sh -c 'make && cd t && ./t1234-foo.sh --stress --stress-limit=300' Sure, as a brand new feature it won't be any useful right now, but in a release or three most cooking topics will already contain this, so we could automatically bisect at least newly introduced flakiness. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-06Merge branch 'js/vsts-ci'Libravatar Junio C Hamano1-8/+130
Prepare to run test suite on Azure Pipeline. * js/vsts-ci: (22 commits) test-date: drop unused parameter to getnanos() ci: parallelize testing on Windows ci: speed up Windows phase tests: optionally skip bin-wrappers/ t0061: workaround issues with --with-dashes and RUNTIME_PREFIX tests: add t/helper/ to the PATH with --with-dashes mingw: try to work around issues with the test cleanup tests: include detailed trace logs with --write-junit-xml upon failure tests: avoid calling Perl just to determine file sizes README: add a build badge (status of the Azure Pipelines build) mingw: be more generous when wrapping up the setitimer() emulation ci: use git-sdk-64-minimal build artifact ci: add a Windows job to the Azure Pipelines definition Add a build definition for Azure DevOps ci/lib.sh: add support for Azure Pipelines tests: optionally write results as JUnit-style .xml test-date: add a subcommand to measure times in shell scripts ci: use a junction on Windows instead of a symlink ci: inherit --jobs via MAKEFLAGS in run-build-and-tests ci/lib.sh: encapsulate Travis-specific things ...
2019-02-05Merge branch 'js/test-git-installed'Libravatar Junio C Hamano1-1/+1
Test fix for Windows. * js/test-git-installed: tests: explicitly use `test-tool.exe` on Windows
2019-01-29tests: optionally skip bin-wrappers/Libravatar Johannes Schindelin1-6/+13
This speeds up the tests by a bit on Windows, where running Unix shell scripts (and spawning processes) is not exactly a cheap operation. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29tests: add t/helper/ to the PATH with --with-dashesLibravatar Johannes Schindelin1-1/+1
We really need to be able to find the test helpers... Really. This change was forgotten when we moved the test helpers into t/helper/ Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29mingw: try to work around issues with the test cleanupLibravatar Johannes Schindelin1-1/+5
It seems that every once in a while in the Git for Windows SDK, there are some transient file locking issues preventing the test clean up to delete the trash directory. Let's be gentle and try again five seconds later, and only error out if it still fails the second time. This change helps Windows, and does not hurt any other platform (normally, it is highly unlikely that said deletion fails, and if it does, normally it will fail again even 5 seconds later). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29tests: include detailed trace logs with --write-junit-xml upon failureLibravatar Johannes Schindelin1-1/+21
The JUnit XML format lends itself to be presented in a powerful UI, where you can drill down to the information you are interested in very quickly. For test failures, this usually means that you want to see the detailed trace of the failing tests. With Travis CI, we passed the `--verbose-log` option to get those traces. However, that seems excessive, as we do not need/use the logs in almost all of those cases: only when a test fails do we have a way to include the trace. So let's do something different when using Azure DevOps: let's run all the tests with `--quiet` first, and only if a failure is encountered, try to trace the commands as they are executed. Of course, we cannot turn on `--verbose-log` after the fact. So let's just re-run the test with all the same options, adding `--verbose-log`. And then munging the output file into the JUnit XML on the fly. Note: there is an off chance that re-running the test in verbose mode "fixes" the failures (and this does happen from time to time!). That is a possibility we should be able to live with. Ideally, we would label this as "Passed upon rerun", and Azure Pipelines even know about that outcome, but it is not available when using the JUnit XML format for now: https://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29tests: optionally write results as JUnit-style .xmlLibravatar Johannes Schindelin1-0/+91
This will come in handy when publishing the results of Git's test suite during an automated Azure DevOps run. Note: we need to make extra sure that invalid UTF-8 encoding is turned into valid UTF-8 (using the Replacement Character, \uFFFD) because t9902's trace contains such invalid byte sequences, and the task in the Azure Pipeline that uploads the test results would refuse to do anything if it was asked to parse an .xml file with invalid UTF-8 in it. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-22tests: explicitly use `test-tool.exe` on WindowsLibravatar Johannes Schindelin1-1/+1
In 8abfdf44c882 (tests: explicitly use `git.exe` on Windows, 2018-11-14), we made sure to use the `.exe` file extension when using an absolute path to `git.exe`, to avoid getting confused with a file or directory in the same place that lacks said file extension. For the same reason, we need to handle test-tool.exe the same way. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: add the '--stress' option to run a test repeatedly under loadLibravatar SZEDER Gábor1-2/+107
Unfortunately, we have a few flaky tests, whose failures tend to be hard to reproduce. We've found that the best we can do to reproduce such a failure is to run the test script repeatedly while the machine is under load, and wait in the hope that the load creates enough variance in the timing of the test's commands that a failure is evenually triggered. I have a command to do that, and I noticed that two other contributors have rolled their own scripts to do the same, all choosing slightly different approaches. To help reproduce failures in flaky tests, introduce the '--stress' option to run a test script repeatedly in multiple parallel jobs until one of them fails, thereby using the test script itself to increase the load on the machine. The number of parallel jobs is determined by, in order of precedence: the number specified as '--stress=<N>', or the value of the GIT_TEST_STRESS_LOAD environment variable, or twice the number of available processors (as reported by the 'getconf' utility), or 8. Make '--stress' imply '--verbose -x --immediate' to get the most information about rare failures; there is really no point in spending all the extra effort to reproduce such a failure, and then not know which command failed and why. To prevent the several parallel invocations of the same test from interfering with each other: - Include the parallel job's number in the name of the trash directory and the various output files under 't/test-results/' as a '.stress-<Nr>' suffix. - Add the parallel job's number to the port number specified by the user or to the test number, so even tests involving daemons listening on a TCP socket can be stressed. - Redirect each parallel test run's verbose output to 't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the output of several parallel running tests to the terminal would create a big ugly mess. For convenience, print the output of the failed test job at the end, and rename its trash directory to end with the '.stress-failed' suffix, so it's easy to find in a predictable path (OTOH, all absolute paths recorded in the trash directory become invalid; we'll see whether this causes any issues in practice). If, in an unlikely case, more than one jobs were to fail nearly at the same time, then print the output of all failed jobs, and rename the trash directory of only the last one (i.e. with the highest job number), as it is the trash directory of the test whose output will be at the bottom of the user's terminal. Based on Jeff King's 'stress' script. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: set $TRASH_DIRECTORY earlierLibravatar SZEDER Gábor1-6/+6
A later patch in this series will need to know the path to the trash directory early in 'test-lib.sh', but $TRASH_DIRECTORY is set much later. Set $TRASH_DIRECTORY earlier, where the other test-specific path variables are set. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: consolidate naming of test-results pathsLibravatar SZEDER Gábor1-11/+11
There are two places where we strip off any leading path components and the '.sh' suffix from the test script's pathname, and there are four places where we construct the name of the 't/test-results' directory or the name of various test-specific files in there. The last patch in this series will add even more. Factor these out into helper variables to avoid repeating ourselves. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: parse command line options earlierLibravatar SZEDER Gábor1-109/+124
'test-lib.sh' looks for the presence of certain options like '--tee' and '--verbose-log', so it can execute the test script again to save its standard output and error. It looks for '--valgrind' as well, to set up some Valgrind-specific stuff. These all happen before the actual option parsing loop, and the conditions looking for these options look a bit odd, too. They are not completely correct, either, because in a bogus invocation like './t1234-foo.sh -r --tee' they recognize '--tee', although it should be handled as the required argument of the '-r' option. This patch series will add two more options to look out for early, and, in addition, will have to extract these options' stuck arguments (i.e. '--opt=arg') as well. So let's move the option parsing loop and the couple of related conditions following it earlier in 'test-lib.sh', before the place where the test script is executed again for '--tee' and its friends. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: parse options in a for loop to keep $@ intactLibravatar SZEDER Gábor1-36/+42
'test-lib.sh' looks for the presence of certain options like '--tee' and '--verbose-log', so it can execute the test script again to save its standard output and error, and to do so it needs the original command line options the test was invoked with. The next patch is about to move the option parsing loop earlier in 'test-lib.sh', but it is implemented using 'shift' in a while loop, effecively destroying "$@" by the end of the option parsing. Not good. As a preparatory step, turn that option parsing loop into a 'for opt in "$@"' loop to preserve "$@" intact while iterating over the options, and taking extra care to handle the '-r' option's required argument (or the lack thereof). Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: extract Bash version check for '-x' tracingLibravatar SZEDER Gábor1-18/+19
One of our test scripts, 't1510-repo-setup.sh' [1], still can't be reliably run with '-x' tracing enabled, unless it's executed with a Bash version supporting BASH_XTRACEFD (since v4.1). We have a lengthy condition to check the version of the shell running the test script, and disable tracing if it's not executed with a suitable Bash version [2]. Move this check out from the option parsing loop, so other options can imply '-x' by setting 'trace=t', without missing this Bash version check. [1] 5827506928 (t1510-repo-setup: mark as untraceable with '-x', 2018-02-24) [2] 5fc98e79fc (t: add means to disable '-x' tracing for individual test scripts, 2018-02-24) Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-03test-lib: translate SIGTERM and SIGHUP to an exitLibravatar SZEDER Gábor1-1/+1
Right now if a test script receives SIGTERM or SIGHUP (e.g., because a test was hanging and the user 'kill'-ed it or simply closed the terminal window the test was running in), the shell exits immediately. This can be annoying if the test script did any global setup, like starting apache or git-daemon, as it will not have an opportunity to clean up after itself. A subsequent run of the test won't be able to start its own daemon, and will either fail or skip the tests. Instead, let's trap SIGTERM and SIGHUP as well to make sure we do a clean shutdown, and just chain it to a normal exit (which will trigger any cleanup). This patch follows suit of da706545f7 (t: translate SIGINT to an exit, 2015-03-13), and even stole its commit message as well. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-03Merge branch 'sg/test-bash-version-fix'Libravatar Junio C Hamano1-2/+2
* sg/test-bash-version-fix: test-lib: check Bash version for '-x' without using shell arrays
2019-01-03test-lib: check Bash version for '-x' without using shell arraysLibravatar SZEDER Gábor1-2/+2
One of our test scripts, 't1510-repo-setup.sh' [1], still can't be reliably run with '-x' tracing enabled, unless it's executed with a Bash version supporting BASH_XTRACEFD (since v4.1). We have a lengthy condition to check the version of the shell running the test script, and disable tracing if it's not executed with a suitable Bash version [2]. This condition uses non-portable shell array accesses to easily get Bash's major and minor version number. This didn't seem to be problematic, because the simple commands expanding those array accesses are only executed when the test script is actually run with Bash. When run with Dash, the only shell I have at hand that doesn't support shell arrays, there are no issues, as it apparently skips right over the non-executed simple commands without noticing the non-supported constructs. Alas, it has been reported that NetBSD's /bin/sh does complain about them: ./test-lib.sh: 327: Syntax error: Bad substitution where line 327 contains the first ${BASH_VERSINFO[0]} array access. To my understanding both shells are right and conform to POSIX, because the standard allows both behaviors by stating the following under '2.8.1 Consequences of Shell Errors' [3]: "An expansion error is one that occurs when the shell expansions define in wordexp are carried out (for example, "${x!y}", because '!' is not a valid operator); an implementation may treat these as syntax errors if it is able to detect them during tokenization, rather than during expansion." Avoid this issue with NetBSD's /bin/sh (and potentially with other, less common shells) by hiding the shell array syntax behind 'eval' that is only executed with Bash. [1] 5827506928 (t1510-repo-setup: mark as untraceable with '-x', 2018-02-24) [2] 5fc98e79fc (t: add means to disable '-x' tracing for individual test scripts, 2018-02-24) [3] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_08_01 Reported-by: Max Kirillov <max@max630.net> Helped-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-20tests: send "bug in the test script" errors to the script's stderrLibravatar SZEDER Gábor1-3/+7
Some of the functions in our test library check that they were invoked properly with conditions like this: test "$#" = 2 || error "bug in the test script: not 2 parameters to test-expect-success" If this particular condition is triggered, then 'error' will abort the whole test script with a bold red error message [1] right away. However, under certain circumstances the test script will be aborted completely silently, namely if: - a similar condition in a test helper function like 'test_line_count' is triggered, - which is invoked from the test script's "main" shell [2], - and the test script is run manually (i.e. './t1234-foo.sh' as opposed to 'make t1234-foo.sh' or 'make test') [3] - and without the '--verbose' option, because the error message is printed from within 'test_eval_', where standard output is redirected either to /dev/null or to a log file. The only indication that something is wrong is that not all tests in the script are executed and at the end of the test script's output there is no "# passed all N tests" message, which are subtle and can easily go unnoticed, as I had to experience myself. Send these "bug in the test script" error messages directly to the test scripts standard error and thus to the terminal, so those bugs will be much harder to overlook. Instead of updating all ~20 such 'error' calls with a redirection, let's add a BUG() function to 'test-lib.sh', wrapping an 'error' call with the proper redirection and also including the common prefix of those error messages, and convert all those call sites [4] to use this new BUG() function instead. [1] That particular error message from 'test_expect_success' is printed in color only when running with or without '--verbose'; with '--tee' or '--verbose-log' the error is printed without color, but it is printed to the terminal nonetheless. [2] If such a condition is triggered in a subshell of a test, then 'error' won't be able to abort the whole test script, but only the subshell, which in turn causes the test to fail in the usual way, indicating loudly and clearly that something is wrong. [3] Well, 'error' aborts the test script the same way when run manually or by 'make' or 'prove', but both 'make' and 'prove' pay attention to the test script's exit status, and even a silently aborted test script would then trigger those tools' usual noticable error messages. [4] Strictly speaking, not all those 'error' calls need that redirection to send their output to the terminal, see e.g. 'test_expect_success' in the opening example, but I think it's better to be consistent. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-19Merge branch 'js/test-git-installed'Libravatar Junio C Hamano1-6/+16
Update the "test installed Git" mode of our test suite to work better. * js/test-git-installed: tests: explicitly use `git.exe` on Windows tests: do not require Git to be built when testing an installed Git t/lib-gettext: test installed git-sh-i18n if GIT_TEST_INSTALLED is set tests: respect GIT_TEST_INSTALLED when initializing repositories tests: fix GIT_TEST_INSTALLED's PATH to include t/helper/
2018-11-19Merge branch 'ab/dynamic-gettext-poison'Libravatar Junio C Hamano1-5/+17
Our testing framework uses a special i18n "poisoned localization" feature to find messages that ought to stay constant but are incorrectly marked to be translated. This feature has been made into a runtime option (it used to be a compile-time option). * ab/dynamic-gettext-poison: Makefile: ease dynamic-gettext-poison transition i18n: make GETTEXT_POISON a runtime option
2018-11-16tests: explicitly use `git.exe` on WindowsLibravatar Johannes Schindelin1-4/+9
On Windows, when we refer to `/an/absolute/path/to/git`, it magically resolves `git.exe` at that location. Except if something of the name `git` exists next to that `git.exe`. So if we call `$BUILD_DIR/git`, it will find `$BUILD_DIR/git.exe` *only* if there is not, say, a directory called `$BUILD_DIR/git`. Such a directory, however, exists in Git for Windows when building with Visual Studio (our Visual Studio project generator defaults to putting the build files into a directory whose name is the base name of the corresponding `.exe`). In the bin-wrappers/* scripts, we already take pains to use `git.exe` rather than `git`, as this could pick up the wrong thing on Windows (i.e. if there exists a `git` file or directory in the build directory). Now we do the same in the tests' start-up code. This also helps when testing an installed Git, as there might be even more likely some stray file or directory in the way. Note: the only way we can record whether the `.exe` suffix is by writing it to the `GIT-BUILD-OPTIONS` file and sourcing it at the beginning of `t/test-lib.sh`. This is not a requirement introduced by this patch, but we move the call to be able to use the `$X` variable that holds the file extension, if any. Note also: the many, many calls to `git this` and `git that` are unaffected, as the regular PATH search will find the `.exe` files on Windows (and not be confused by a directory of the name `git` that is in one of the directories listed in the `PATH` variable), while `/path/to/git` would not, per se, know that it is looking for an executable and happily prefer such a directory. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-16tests: do not require Git to be built when testing an installed GitLibravatar Johannes Schindelin1-2/+7
We really only need the test helpers to be built in the worktree in that case, but that is not what we test for. On the other hand it is a perfect opportunity to verify that `GIT_TEST_INSTALLED` points to a working Git. So let's test the appropriate Git executable. While at it, also adjust the error message in the `GIT_TEST_INSTALLED` case. This patch is best viewed with `-w --patience`. Helped-by: Jeff King <peff@peff.net> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-14tests: fix GIT_TEST_INSTALLED's PATH to include t/helper/Libravatar Johannes Schindelin1-1/+1
We really need to be able to find the test helpers... Really. This change was forgotten when we moved the test helpers into t/helper/ Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-13Merge branch 'js/rebase-p-tests'Libravatar Junio C Hamano1-0/+4
In preparation to the day when we can deprecate and remove the "rebase -p", make sure we can skip and later remove tests for it. * js/rebase-p-tests: tests: optionally skip `git rebase -p` tests t3418: decouple test cases from a previous `rebase -p` test case t3404: decouple some test cases from outcomes of previous test cases
2018-11-09i18n: make GETTEXT_POISON a runtime optionLibravatar Ævar Arnfjörð Bjarmason1-5/+17
Change the GETTEXT_POISON compile-time + runtime GIT_GETTEXT_POISON test parameter to only be a GIT_TEST_GETTEXT_POISON=<non-empty?> runtime parameter, to be consistent with other parameters documented in "Running tests with special setups" in t/README. When I added GETTEXT_POISON in bb946bba76 ("i18n: add GETTEXT_POISON to simulate unfriendly translator", 2011-02-22) I was concerned with ensuring that the _() function would get constant folded if NO_GETTEXT was defined, and likewise that GETTEXT_POISON would be compiled out unless it was defined. But as the benchmark in my [1] shows doing a one-off runtime getenv("GIT_TEST_[...]") is trivial, and since GETTEXT_POISON was originally added the GIT_TEST_* env variables have become the common idiom for turning on special test setups. So change GETTEXT_POISON to work the same way. Now the GETTEXT_POISON=YesPlease compile-time option is gone, and running the tests with GIT_TEST_GETTEXT_POISON=[YesPlease|] can be toggled on/off without recompiling. This allows for conditionally amending tests to test with/without poison, similar to what 859fdc0c3c ("commit-graph: define GIT_TEST_COMMIT_GRAPH", 2018-08-29) did for GIT_TEST_COMMIT_GRAPH. Do some of that, now we e.g. always run the t0205-gettext-poison.sh test. I did enough there to remove the GETTEXT_POISON prerequisite, but its inverse C_LOCALE_OUTPUT is still around, and surely some tests using it can be converted to e.g. always set GIT_TEST_GETTEXT_POISON=. Notes on the implementation: * We still compile a dedicated GETTEXT_POISON build in Travis CI. Perhaps this should be revisited and integrated into the "linux-gcc" build, see ae59a4e44f ("travis: run tests with GIT_TEST_SPLIT_INDEX", 2018-01-07) for prior art in that area. Then again maybe not, see [2]. * We now skip a test in t0000-basic.sh under GIT_TEST_GETTEXT_POISON=YesPlease that wasn't skipped before. This test relies on C locale output, but due to an edge case in how the previous implementation of GETTEXT_POISON worked (reading it from GIT-BUILD-OPTIONS) wasn't enabling poison correctly. Now it does, and needs to be skipped. * The getenv() function is not reentrant, so out of paranoia about code of the form: printf(_("%s"), getenv("some-env")); call use_gettext_poison() in our early setup in git_setup_gettext() so we populate the "poison_requested" variable in a codepath that's won't suffer from that race condition. * We error out in the Makefile if you're still saying GETTEXT_POISON=YesPlease to prompt users to change their invocation. * We should not print out poisoned messages during the test initialization itself to keep it more readable, so the test library hides the variable if set in $GIT_TEST_GETTEXT_POISON_ORIG during setup. See [3]. See also [4] for more on the motivation behind this patch, and the history of the GETTEXT_POISON facility. 1. https://public-inbox.org/git/871s8gd32p.fsf@evledraar.gmail.com/ 2. https://public-inbox.org/git/20181102163725.GY30222@szeder.dev/ 3. https://public-inbox.org/git/20181022202241.18629-2-szeder.dev@gmail.com/ 4. https://public-inbox.org/git/878t2pd6yu.fsf@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-06Merge branch 'sg/test-verbose-log'Libravatar Junio C Hamano1-2/+2
Our test scripts can now take the '-V' option as a synonym for the '--verbose-log' option. * sg/test-verbose-log: test-lib: introduce the '-V' short option for '--verbose-log'
2018-11-02tests: optionally skip `git rebase -p` testsLibravatar Johannes Schindelin1-0/+4
The `--preserve-merges` mode of the `rebase` command is slated to be deprecated soon, as the more powerful `--rebase-merges` mode is available now, and the latter was designed with the express intent to address the shortcomings of `--preserve-merges`' design (e.g. the inability to reorder commits in an interactive rebase). As such, we will eventually even remove the `--preserve-merges` support, and along with it, its tests. In preparation for this, and also to allow the Windows phase of our automated tests to save some well-needed time when running the test suite, this commit introduces a new prerequisite REBASE_P, which can be forced to being unmet by setting the environment variable `GIT_TEST_SKIP_REBASE_P` to any non-empty string. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-30test-lib: introduce the '-V' short option for '--verbose-log'Libravatar SZEDER Gábor1-2/+2
'--verbose-log' is one of the most useful and thus most frequently used test options, but due to its length it's a pain to type on the command line. Let's introduce the corresponding short option '-V' to save some keystrokes. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-19Merge branch 'bp/rename-test-env-var'Libravatar Junio C Hamano1-2/+33
Some environment variables that control the runtime options of Git used during tests are getting renamed for consistency. * bp/rename-test-env-var: t0000: do not get self-test disrupted by environment warnings preload-index: update GIT_FORCE_PRELOAD_TEST support read-cache: update TEST_GIT_INDEX_VERSION support fsmonitor: update GIT_TEST_FSMONITOR support preload-index: use git_env_bool() not getenv() for customization t/README: correct spelling of "uncommon"
2018-09-28t0000: do not get self-test disrupted by environment warningsLibravatar Junio C Hamano1-0/+8
The test framework test-lib.sh itself would want to give warnings and hints, e.g. when it sees a deprecated environment variable is in use that we want to encourage users to migrate to another variable. The self-test of test framework done in t0000 however do not expect to see these warnings and hints, so depending on the settings of environment variables, a running test may or may not produce these messages to the standard error output, breaking the expectations of self-test test framework does on itself. Here is what we see: $ TEST_GIT_INDEX_VERSION=4 sh t0000-basic.sh -i -v ... 'err' is not empty, it contains: warning: TEST_GIT_INDEX_VERSION is now GIT_TEST_INDEX_VERSION hint: set GIT_TEST_INDEX_VERSION too during the transition period not ok 5 - pretend we have a fully passing test suite The following quick attempt to work it around does not work, because some tests in t0000 do want to see expected errors from the test framework itself. t/t0000-basic.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh index 850f651e4e..88c6ed4696 100755 --- a/t/t0000-basic.sh +++ b/t/t0000-basic.sh @@ -88,7 +88,7 @@ _run_sub_test_lib_test_common () { ' # Point to the t/test-lib.sh, which isn't in ../ as usual - . "\$TEST_DIRECTORY"/test-lib.sh + . "\$TEST_DIRECTORY"/test-lib.sh >/dev/null 2>&1 EOF cat >>"$name.sh" && chmod +x "$name.sh" && There are a few possible ways to work this around: * We could strip the warning: and hint: unconditionally from the error output before the error messages are checked in the self-test (helper functions check_sub_test_lib_test_err and check_sub_test_lib_test); the problem with this approach is that it will make it impossible to write self-tests to ensure that right warnings and hints are given. * We could force a sane environment settings before the test helper _run_sub_test_lib_test_common dot-sources test-lib.sh; the problem with this approach is that _run_sub_test_lib_test_common now needs to be aware of what pairs of environment variables are checked in test-lib.sh using check_var_migration helper. The final patch I came up with is probably the solution that is least bad. Set a variable to tell test-lib.sh that we are running a self-test, so that various pieces in test-lib.sh can react to keep the output stable. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-28preload-index: update GIT_FORCE_PRELOAD_TEST supportLibravatar Ben Peart1-0/+1
Rename GIT_FORCE_PRELOAD_TEST to GIT_TEST_PRELOAD_INDEX for consistency with the other GIT_TEST_ special setups and properly document its use. Add logic in t/test-lib.sh to give a warning when the old variable is set to let people know they need to update their environment to use the new variable. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-28read-cache: update TEST_GIT_INDEX_VERSION supportLibravatar Ben Peart1-6/+8
Rename TEST_GIT_INDEX_VERSION to GIT_TEST_INDEX_VERSION for consistency with the other GIT_TEST_ special setups and properly document its use. Add logic in t/test-lib.sh to give a warning when the old variable is set to let people know they need to update their environment to use the new variable. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-28fsmonitor: update GIT_TEST_FSMONITOR supportLibravatar Ben Peart1-0/+20
Rename GIT_FSMONITOR_TEST to GIT_TEST_FSMONITOR for consistency with the other GIT_TEST_ special setups and properly document its use. Add logic in t/test-lib.sh to give a warning when the old variable is set to let people know they need to update their environment to use the new variable. Remove the outdated instructions on how to run the test suite utilizing fsmonitor now that it is properly documented in t/README. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-17Merge branch 'nd/unpack-trees-with-cache-tree'Libravatar Junio C Hamano1-0/+6
The unpack_trees() API used in checking out a branch and merging walks one or more trees along with the index. When the cache-tree in the index tells us that we are walking a tree whose flattened contents is known (i.e. matches a span in the index), as linearly scanning a span in the index is much more efficient than having to open tree objects recursively and listing their entries, the walk can be optimized, which is done in this topic. * nd/unpack-trees-with-cache-tree: Document update for nd/unpack-trees-with-cache-tree cache-tree: verify valid cache-tree in the test suite unpack-trees: add missing cache invalidation unpack-trees: reuse (still valid) cache-tree from src_index unpack-trees: reduce malloc in cache-tree walk unpack-trees: optimize walking same trees with cache-tree unpack-trees: add performance tracing trace.h: support nested performance tracing
2018-08-27tests: fix and add lint for non-portable head -c NLibravatar Ævar Arnfjörð Bjarmason1-2/+2
The "head -c BYTES" option is non-portable (not in POSIX[1]). Change such invocations to use the test_copy_bytes wrapper added in 48860819e8 ("t9300: factor out portable "head -c" replacement", 2016-06-30). This fixes a test added in 9d2e330b17 ("ewah_read_mmap: bounds-check mmap reads", 2018-06-14), which has been breaking t5310-pack-bitmaps.sh on OpenBSD since 2.18.0. The OpenBSD ports already have a similar workaround after their upgrade to 2.18.0[2]. I have not tested this on IRIX, but according to 4de0bbd898 ("t9300: use perl "head -c" clone in place of "dd bs=1 count=16000" kluge", 2010-12-13) this invocation would have broken things there too. Also, change a valgrind-specific codepath in test-lib.sh to use this wrapper. Given where valgrind runs I don't think this would ever become a portability issue in practice, but it's easier to just use the wrapper than introduce some exception for the "make test-lint" check being added here. 1. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/head.html 2. https://github.com/openbsd/ports/commit/08d5d82eaefe5cf2f125ecc0c6a57df9cf91350c#diff-f7d3c4fabeed1691620d608f1534f5e5 Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-08-20Merge branch 'wc/make-funnynames-shared-lazy-prereq'Libravatar Junio C Hamano1-0/+14
A test prerequisite defined by various test scripts with slightly different semantics has been consolidated into a single copy and made into a lazily defined one. * wc/make-funnynames-shared-lazy-prereq: t: factor out FUNNYNAMES as shared lazy prereq
2018-08-18cache-tree: verify valid cache-tree in the test suiteLibravatar Nguyễn Thái Ngọc Duy1-0/+6
This makes sure that cache-tree is consistent with the index. The main purpose is to catch potential problems by saving the index in unpack_trees() but the line in write_index() would also help spot missing invalidation in other code. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-08-06t: factor out FUNNYNAMES as shared lazy prereqLibravatar William Chargin1-0/+14
A fair number of tests need to check that the filesystem supports file names including "funny" characters, like newline, tab, and double-quote. Jonathan Nieder suggested that this be extracted into a lazy prereq in the top-level `test-lib.sh`. This patch effects that change. The FUNNYNAMES prereq now uniformly requires support for newlines, tabs, and double-quotes in filenames. This very slightly decreases the power of some tests, which might have run previously on a system that supports (e.g.) newlines and tabs but not double-quotes, but now will not. This seems to me like an acceptable tradeoff for consistency. One test (`t/t9902-completion.sh`) defined FUNNYNAMES to further require the separators \034 through \037, the test for which was implemented using the Bash-specific $'\034' syntax. I've elected to leave this one as is, renaming it to FUNNIERNAMES. After this patch, `git grep 'test_\(set\|lazy\)_prereq.*FUNNYNAMES'` has only one result. Signed-off-by: William Chargin <wchargin@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-17t/test-lib: teach --chain-lint to detect broken &&-chains in subshellsLibravatar Eric Sunshine1-1/+2
The --chain-lint option detects broken &&-chains by forcing the test to exit early (as the very first step) with a sentinel value. If that sentinel is the test's overall exit code, then the &&-chain is intact; if not, then the chain is broken. Unfortunately, this detection does not extend to &&-chains within subshells even when the subshell itself is properly linked into the outer &&-chain. Address this shortcoming by feeding the body of the test to a lightweight "linter" which can peer inside subshells and identify broken &&-chains by pure textual inspection. Although the linter does not actually parse shell scripts, it has enough knowledge of shell syntax to reliably deal with formatting style variations (as evolved over the years) and to avoid being fooled by non-shell content (such as inside here-docs and multi-line strings). It recognizes modern subshell formatting: statement1 && ( statement2 && statement3 ) && statement4 as well as old-style: statement1 && (statement2 && statement3) && statement4 Heuristics are employed to properly identify the extent of a subshell formatted in the old-style since a number of legitimate constructs may superficially appear to close the subshell even though they don't. For example, it understands that neither "x=$(command)" nor "case $x in *)" end a subshell, despite the ")" at the end of line. Due to limitations of the tool used ('sed') and its inherent line-by-line processing, only subshells one level deep are handled, as well as one-liner subshells one level below that. Subshells deeper than that or multi-line subshells at level two are passed through as-is, thus &&-chains in their bodies are not checked. Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-05-30Merge branch 'bc/hash-independent-tests'Libravatar Junio C Hamano1-1/+10
Many tests hardcode the raw object names, which would change once we migrate away from SHA-1. While some of them must test against exact object names, most of them do not have to use hardcoded constants in the test. The latter kind of tests have been updated to test the moral equivalent of the original without hardcoding the actual object names. * bc/hash-independent-tests: (28 commits) t5300: abstract away SHA-1-specific constants t4208: abstract away SHA-1-specific constants t4045: abstract away SHA-1-specific constants t4042: abstract away SHA-1-specific constants t4205: sort log output in a hash-independent way t/lib-diff-alternative: abstract away SHA-1-specific constants t4030: abstract away SHA-1-specific constants t4029: abstract away SHA-1-specific constants t4029: fix test indentation t4022: abstract away SHA-1-specific constants t4020: abstract away SHA-1-specific constants t4014: abstract away SHA-1-specific constants t4008: abstract away SHA-1-specific constants t4007: abstract away SHA-1-specific constants t3905: abstract away SHA-1-specific constants t3702: abstract away SHA-1-specific constants t3103: abstract away SHA-1-specific constants t2203: abstract away SHA-1-specific constants t: skip pack tests if not using SHA-1 t4044: skip test if not using SHA-1 ...
2018-05-23Merge branch 'tb/test-apfs-utf8-normalization'Libravatar Junio C Hamano1-6/+1
A test to see if the filesystem normalizes UTF-8 filename has been updated to check what we need to know in a more direct way, i.e. a path created in NFC form can be accessed with NFD form (or vice versa) to cope with APFS as well as HFS. * tb/test-apfs-utf8-normalization: test: correct detection of UTF8_NFD_TO_NFC for APFS
2018-05-14t/test-lib: introduce OID_REGEXLibravatar brian m. carlson1-1/+2
Currently we have a variable, $_x40, which contains a regex that matches a full 40-character hex constant. However, with NewHash, we'll have object IDs that are longer than 40 characters. In such a case, $_x40 will be a confusing name. Create a $OID_REGEX variable which will always reflect a regex matching the appropriate object ID, regardless of the length of the current hash. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>