summaryrefslogtreecommitdiff
path: root/t/test-lib.sh
AgeCommit message (Collapse)AuthorFilesLines
2020-04-29Merge branch 'dd/ci-swap-azure-pipelines-with-github-actions'Libravatar Junio C Hamano1-1/+13
Update the CI configuration to use GitHub Actions, retiring the one based on Azure Pipelines. * dd/ci-swap-azure-pipelines-with-github-actions: ci: let GitHub Actions upload failed tests' directories ci: add a problem matcher for GitHub Actions tests: when run in Bash, annotate test failures with file name/line number ci: retire the Azure Pipelines definition README: add a build badge for the GitHub Actions runs ci: configure GitHub Actions for CI/PR ci: run gem with sudo to install asciidoctor ci: explicit install all required packages ci: fix the `jobname` of the `GETTEXT_POISON` job ci/lib: set TERM environment variable if not exist ci/lib: allow running in GitHub Actions ci/lib: if CI type is unknown, show the environment variables
2020-04-22Merge branch 'mt/test-lib-bundled-short-options'Libravatar Junio C Hamano1-14/+47
Minor test usability improvement. * mt/test-lib-bundled-short-options: test-lib: allow short options to be bundled
2020-04-22Merge branch 'js/test-junit-finalization-fix'Libravatar Junio C Hamano1-0/+1
Test fix. * js/test-junit-finalization-fix: tests(junit-xml): avoid invalid XML
2020-04-22Merge branch 'js/tests-gpg-integration-on-windows'Libravatar Junio C Hamano1-2/+4
Enable tests that require GnuPG on Windows. * js/tests-gpg-integration-on-windows: tests: increase the verbosity of the GPG-related prereqs tests: turn GPG, GPGSM and RFC1991 into lazy prereqs tests: do not let lazy prereqs inside `test_expect_*` turn off tracing t/lib-gpg.sh: stop pretending to be a stand-alone script tests(gpg): allow the gpg-agent to start on Windows
2020-04-10tests: when run in Bash, annotate test failures with file name/line numberLibravatar Johannes Schindelin1-1/+13
When a test fails, it is nice to see where the corresponding code lives in the worktree. Sadly, it seems that only Bash allows us to infer this information. Let's do it when we detect that we're running in a Bash. This will come in handy in the next commit, where we teach the GitHub Actions workflow to annotate failed test runs with this information. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-03-26tests: do not let lazy prereqs inside `test_expect_*` turn off tracingLibravatar Johannes Schindelin1-2/+4
The `test_expect_*` functions use `test_eval_` and so does `test_run_lazy_prereq_`. If tracing is enabled via the `-x` option, `test_eval_` turns on tracing while evaluating the code block, and turns it off directly after it. This is unwanted for nested invocations. One somewhat surprising example of this is when running a test that calls `test_i18ngrep`: that function requires the `C_LOCALE_OUTPUT` prereq, and that prereq is a lazy one, so it is evaluated via `test_eval_`, the command tracing is turned off, and the test case continues to run _without tracing the commands_. Another somewhat surprising example is when one lazy prereq depends on another lazy prereq: the former will call `test_have_prereq` with the latter one, which in turn calls `test_eval_` and -- you guessed it -- tracing (if enabled) will be turned off _before_ returning to evaluating the other lazy prereq. As we will introduce just such a scenario with the GPG, GPGSM and RFC1991 prereqs, let's fix that by introducing a variable that keeps track of the current trace level: nested `test_eval_` calls will increment and then decrement the level, and only when it reaches 0, the tracing will _actually_ be turned off. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-03-25test-lib: allow short options to be bundledLibravatar Matheus Tavares1-14/+47
When debugging a test (or a set of tests), it's common to execute it with some combination of short options, such as: $ ./txxx-testname.sh -d -x -i In cases like this, CLIs usually allow the short options to be bundled in a single argument, for convenience and agility. Let's add this feature to test-lib, allowing the above command to be run as: $ ./txxx-testname.sh -dxi (or any other permutation, e.g. '-ixd') Note: Short options that require an argument can also be used in a bundle, in any position. So, for example, '-r 5 -x', '-xr 5' and '-rx 5' are all valid and equivalent. A special case would be having a bundle with more than one of such options. To keep things simple, this case is not allowed for now. This shouldn't be a major limitation, though, as the only short option that requires an argument today is '-r'. And concatenating '-r's as in '-rr 5 6' would probably not be very practical: its unbundled format would be '-r 5 -r 6', for which test-lib currently considers only the last argument. Therefore, if '-rr 5 6' were to be allowed, it would have the same effect as just typing '-r 6'. Note: the test-lib currently doesn't support '-r5', as an alternative for '-r 5', so the former is not supported in bundles as well. Helped-by: Jeff King <peff@peff.net> Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-03-23tests(junit-xml): avoid invalid XMLLibravatar Johannes Schindelin1-0/+1
When a test case is run in a subshell, we finalize the JUnit-style XML when said subshell exits. But then we continue to write into that XML as if nothing had happened. This leads to Azure Pipelines' Publish Test Results task complaining: Failed to read /home/vsts/work/1/s/t/out/TEST-t0000-basic.xml. Error : Unexpected end tag. Line 110, position 5. And indeed, the resulting XML is incorrect. Let's "re-open" the XML in such a case, i.e. remove the previously added closing tags. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-24t: use hash-specific lookup tables to define test constantsLibravatar brian m. carlson1-15/+14
In the future, we'll allow developers to run the testsuite with a hash algorithm of their choice. To make this easier, compute the fixed constants using test_oid. Move the constant initialization down below the point where test-lib-functions.sh is loaded so the functions are defined. Note that we don't provide a value for the OID_REGEX value directly because writing a large number of instances of "[0-9a-f]" in the oid-info files is unwieldy and there isn't a way to compute it based on those values. Instead, compute it based on ZERO_OID. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-17Merge branch 'js/test-write-junit-xml-fix'Libravatar Junio C Hamano1-1/+2
Testfix. * js/test-write-junit-xml-fix: tests: fix --write-junit-xml with subshells
2020-02-12tests: fix --write-junit-xml with subshellsLibravatar Johannes Schindelin1-1/+2
In t0000, more precisely in its `test_bool_env` test case, there are two subshells that are supposed to fail. To be even _more_ precise, they fail by calling the `error` function, and that is okay, because it is in a subshell, and it is expected that those two subshell invocations fail. However, the `error` function also tries to finalize the JUnit XML (if that XML was asked for, via `--write-junit-xml`. As a consequence, the XML is edited to add a `time` attribute for the `testsuite` tag. And since there are two expected `error` calls in addition to the final `test_done`, the `finalize_junit_xml` function is called three times and naturally the `time` attribute is added _three times_. Azure Pipelines is not happy with that, complaining thusly: ##[warning]Failed to read D:\a\1\s\t\out\TEST-t0000-basic.xml. Error : 'time' is a duplicate attribute name. Line 2, position 82.. One possible way to address this would be to unset `write_junit_xml` in the `test_bool_env` test case. But that would be fragile, as other `error` calls in subshells could be introduced. So let's just modify `finalize_junit_xml` to remove any `time` attribute before adding the authoritative one. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-10Merge branch 'em/test-skip-regex-illseq'Libravatar Junio C Hamano1-1/+7
Test portability fix. * em/test-skip-regex-illseq: t4210: skip i18n tests that don't work on FreeBSD
2019-12-05Merge branch 'sg/test-bool-env'Libravatar Junio C Hamano1-5/+5
Recently we have declared that GIT_TEST_* variables take the usual boolean values (it used to be that some used "non-empty means true" and taking GIT_TEST_VAR=YesPlease as true); make sure we notice and fail when non-bool strings are given to these variables. * sg/test-bool-env: t5608-clone-2gb.sh: turn GIT_TEST_CLONE_2GB into a bool tests: add 'test_bool_env' to catch non-bool GIT_TEST_* values
2019-12-01Merge branch 'sg/skip-skipped-prereq'Libravatar Junio C Hamano1-6/+6
Test update to avoid wasted cycles. * sg/skip-skipped-prereq: test-lib: don't check prereqs of test cases that won't be run anyway
2019-11-30t4210: skip i18n tests that don't work on FreeBSDLibravatar Ed Maste1-1/+7
A number of t4210-log-i18n tests added in 4e2443b181 set LC_ALL to a UTF-8 locale (is_IS.UTF-8) but then pass an invalid UTF-8 string to --grep. FreeBSD's regcomp() fails in this case with REG_ILLSEQ, "illegal byte sequence," which git then passes to die(): fatal: command line: '�': illegal byte sequence When these tests were added the commit message stated: | It's possible that this | test breaks the "basic" and "extended" backends on some systems that | are more anal than glibc about the encoding of locale issues with | POSIX functions that I can remember which seems to be the case here. Extend test-lib.sh to add a REGEX_ILLSEQ prereq, set it on FreeBSD, and add !REGEX_ILLSEQ to the two affected tests. Signed-off-by: Ed Maste <emaste@freebsd.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-23tests: add 'test_bool_env' to catch non-bool GIT_TEST_* valuesLibravatar SZEDER Gábor1-5/+5
Since 3b072c577b (tests: replace test_tristate with "git env--helper", 2019-06-21) we get the normalized bool values of various GIT_TEST_* environment variables via 'git env--helper'. Now, while the 'git env--helper' command itself does catch invalid values in the environment variable or in the given --default and exits with error (exit code 128 or 129, respectively), it's invoked in conditions like 'if ! git env--helper ...', which means that all invalid bool values are interpreted the same as the ordinary 'false' (exit code 1). This has led to inadvertently skipped httpd tests in our CI builds for a couple of weeks, see 3960290675 (ci: restore running httpd tests, 2019-09-06). Let's be more careful about what the test suite accepts as bool values in GIT_TEST_* environment variables, and error out loud and clear on invalid values instead of simply skipping tests. Add the 'test_bool_env' helper function to encapsulate the invocation of 'git env--helper' and the verification of its exit code, and replace all invocations of that command in our test framework and test suite with a call to this new helper (except in 't0017-env-helper.sh', of course). $ GIT_TEST_GIT_DAEMON=YesPlease ./t5570-git-daemon.sh fatal: bad numeric config value 'YesPlease' for 'GIT_TEST_GIT_DAEMON': invalid unit error: test_bool_env requires bool values both for $GIT_TEST_GIT_DAEMON and for the default fallback Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-13test-lib: don't check prereqs of test cases that won't be run anywayLibravatar SZEDER Gábor1-6/+6
With './t1234-foo.sh -r 5,6' we can run only specific test cases in a test script, but our test framwork still evaluates all lazy prereqs that the excluded test cases might depend on. This is unnecessary and produces verbose and trace output that can be distracting. This has been an issue ever since the '-r|--run=' options were introduced in 0445e6f0a1 (test-lib: '--run' to run only specific tests, 2014-04-30), because that commit added the check of the list of test cases specified with '-r' after evaluating the prereqs. Avoid this unnecessary prereq evaluation by checking the list of test cases specified with '-r' before looking at the prereqs. Note that GIT_SKIP_TESTS has always been checked before the prereqs, so prereqs necessary for tests skipped that way were not evaluated. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-30pretty: add "%aL" etc. to show local-part of email addressesLibravatar Prarit Bhargava1-2/+6
In many projects the number of contributors is low enough that users know each other and the full email address doesn't need to be displayed. Displaying only the author's username saves a lot of columns on the screen. Existing 'e/E' (as in "%ae" and "%aE") placeholders would show the author's address as "prarit@redhat.com", which would waste columns to show the same domain-part for all contributors when used in a project internal to redhat. Introduce 'l/L' placeholders that strip '@' and domain part from the e-mail address. Signed-off-by: Prarit Bhargava <prarit@redhat.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-15Merge branch 'js/azure-pipelines-msvc'Libravatar Junio C Hamano1-16/+22
CI updates. * js/azure-pipelines-msvc: ci: also build and test with MS Visual Studio on Azure Pipelines ci: really use shallow clones on Azure Pipelines tests: let --immediate and --write-junit-xml play well together test-tool run-command: learn to run (parts of) the testsuite vcxproj: include more generated files vcxproj: only copy `git-remote-http.exe` once it was built msvc: work around a bug in GetEnvironmentVariable() msvc: handle DEVELOPER=1 msvc: ignore some libraries when linking compat/win32/path-utils.h: add #include guards winansi: use FLEX_ARRAY to avoid compiler warning msvc: avoid using minus operator on unsigned types push: do not pretend to return `int` from `die_push_simple()`
2019-10-06tests: let --immediate and --write-junit-xml play well togetherLibravatar Johannes Schindelin1-16/+22
When the `--immediate` option is in effect, any test failure will immediately exit the test script. Together with `--write-junit-xml`, we will want the JUnit-style `.xml` file to be finalized (and not leave the XML incomplete). Let's make it so. This comes in particularly handy when trying to debug via Azure Pipelines, where the JUnit-style XML is consumed to present the test results in an informative and helpful way. While at it, also handle the `error()` code path. The only remaining code path that sets `GIT_EXIT_OK` happens whenever the trash directory could not be set up, i.e. long before the JUnit XML was written, therefore we should _not_ try to finalize that XML in that case. It is tempting to change the `immediate` code path to just hand off to `error`, simplifying the code in the process. That would, however, result in a change of behavior (an additional error message) in the test suite, which is outside of the purview of the current patch series: its goal is to allow building Git with Visual Studio and testing it with a portable version of Git for Windows. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-09-06t: use common $SQ variableLibravatar Denton Liu1-0/+3
In many test scripts, there are bespoke definitions of the single quote that are some variation of this: SQ="'" Define a common $SQ variable in test-lib.sh and replace all usages of these bespoke variables with the common one. This change was done by running `git grep =\"\'\" t/` and `git grep =\\\\\'` and manually changing the resulting definitions and corresponding usages. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-08-22Merge branch 'sg/show-failed-test-names'Libravatar Junio C Hamano1-0/+2
The first line of verbose output from each test piece now carries the test name and number to help scanning with eyeballs. * sg/show-failed-test-names: tests: show the test name and number at the start of verbose output t0000-basic: use realistic test script names in the verbose tests
2019-08-05tests: show the test name and number at the start of verbose outputLibravatar SZEDER Gábor1-0/+2
The verbose output of every test looks something like this: expecting success: echo content >file && git add file && git commit -m "add file" [master (root-commit) d1fbfbd] add file Author: A U Thor <author@example.com> 1 file changed, 1 insertion(+) create mode 100644 file ok 1 - commit works i.e. first an "expecting success" (or "checking known breakage") line followed by the commands to be executed, then the output of those comamnds, and finally an "ok"/"not ok" line containing the test name. Note that the test's name is only shown at the very end. With '-x' tracing enabled and/or in longer tests the verbose output might be several screenfulls long, making it harder than necessary to find where the output of the test with a given name starts (especially when the outputs to different file descriptors are racing, and the "expecting success"/command block arrives earlier than the "ok" line of the previous test). Print the test name at the start of the test's verbose output, i.e. at the end of the "expecting success" and "checking known breakage" lines, to make the start of a particular test a bit easier to recognize. Also print the test script and test case numbers, to help those poor souls who regularly have to scan through the combined verbose output of several test scripts. So the dummy test above would start like this: expecting success of 9999.1 'commit works': echo content >file && [...] Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-07-25Merge branch 'ab/test-env'Libravatar Junio C Hamano1-9/+22
Many GIT_TEST_* environment variables control various aspects of how our tests are run, but a few followed "non-empty is true, empty or unset is false" while others followed the usual "there are a few ways to spell true, like yes, on, etc., and also ways to spell false, like no, off, etc." convention. * ab/test-env: env--helper: mark a file-local symbol as static tests: make GIT_TEST_FAIL_PREREQS a boolean tests: replace test_tristate with "git env--helper" tests README: re-flow a previously changed paragraph tests: make GIT_TEST_GETTEXT_POISON a boolean t6040 test: stop using global "script" variable config.c: refactor die_bad_number() to not call gettext() early env--helper: new undocumented builtin wrapping git_env_*() config tests: simplify include cycle test
2019-07-09Merge branch 'sg/trace2-rename'Libravatar Junio C Hamano1-1/+0
Dev support update to help tracing out tests. * sg/trace2-rename: trace2: correct typo in technical documentation Revert "test-lib: whitelist GIT_TR2_* in the environment"
2019-06-21tests: make GIT_TEST_FAIL_PREREQS a booleanLibravatar Ævar Arnfjörð Bjarmason1-4/+19
Change the GIT_TEST_FAIL_PREREQS variable from being "non-empty?" to being a more standard boolean variable. I recently added the variable in dfe1a17df9 ("tests: add a special setup where prerequisites fail", 2019-05-13), having to add another "non-empty?" special-case is what prompted me to write the "git env--helper" utility being used here. Converting this one is a bit tricky since we use it so early and frequently in the guts of the test code itself, so let's set a GIT_TEST_FAIL_PREREQS_INTERNAL which can be tested with the old "test -n" for the purposes of the shell code, and change the user-exposed and documented GIT_TEST_FAIL_PREREQS variable to a boolean. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-21tests: make GIT_TEST_GETTEXT_POISON a booleanLibravatar Ævar Arnfjörð Bjarmason1-5/+3
Change the GIT_TEST_GETTEXT_POISON variable from being "non-empty?" to being a more standard boolean variable. Since it needed to be checked in both C code and shellscript (via test -n) it was one of the remaining shellscript-like variables. Now that we have "env--helper" we can change that. There's a couple of tricky edge cases that arise because we're using git_env_bool() early, and the config-reading "env--helper". If GIT_TEST_GETTEXT_POISON is set to an invalid value die_bad_number() will die, but to do so it would usually call gettext(). Let's detect the special case of GIT_TEST_GETTEXT_POISON and always emit that message in the C locale, lest we infinitely loop. As seen in the updated tests in t0017-env-helper.sh there's also a caveat related to "env--helper" needing to read the config for trace2 purposes. Since the C_LOCALE_OUTPUT prerequisite is lazy and relies on "env--helper" we could get invalid results if we failed to read the config (e.g. because we'd loop on includes) when combined with e.g. "test_i18ngrep" wanting to check with "env--helper" if GIT_TEST_GETTEXT_POISON was true or not. I'm crossing my fingers and hoping that a test similar to the one I removed in the earlier "config tests: simplify include cycle test" change in this series won't happen again, and testing for this explicitly in "env--helper"'s own tests. This change breaks existing uses of e.g. GIT_TEST_GETTEXT_POISON=YesPlease, which we've documented in po/README and other places. As noted in [1] we might want to consider also accepting "YesPlease" in "env--helper" as a special-case. But as the lack of uproar over 6cdccfce1e ("i18n: make GETTEXT_POISON a runtime option", 2018-11-08) demonstrates the audience for this option is a really narrow set of git developers, who shouldn't have much trouble modifying their test scripts, so I think it's better to deal with that minor headache now and make all the relevant GIT_TEST_* variables boolean in the same way than carry the "YesPlease" special-case forward. 1. https://public-inbox.org/git/xmqqtvckm3h8.fsf@gitster-ct.c.googlers.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-13Merge branch 'ab/fail-prereqs-in-test'Libravatar Junio C Hamano1-0/+4
Developer support to emulate unsatisfied prerequisites in tests to ensure that the remainer of the tests still succeeds when tests with prerequisites are skipped. * ab/fail-prereqs-in-test: tests: add a special setup where prerequisites fail
2019-06-12Revert "test-lib: whitelist GIT_TR2_* in the environment"Libravatar Ævar Arnfjörð Bjarmason1-1/+0
This reverts my commit c1ee5796dc ("test-lib: whitelist GIT_TR2_* in the environment", 2019-03-30), which is now redundant. Since e4b75d6a1d ("trace2: rename environment variables to GIT_TRACE2*", 2019-05-19) the GIT_TRACE2* variables match the existing GIT_TRACE* pattern added in 95a1d12e9b ("tests: scrub environment of GIT_* variables", 2011-03-15), so we no longer need to list TR2 here. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-05-19Merge branch 'tz/test-lib-check-working-jgit'Libravatar Junio C Hamano1-1/+1
A prerequiste check in the test suite to see if a working jgit is available was made more robust. * tz/test-lib-check-working-jgit: test-lib: try harder to ensure a working jgit
2019-05-15test-lib: try harder to ensure a working jgitLibravatar Todd Zullinger1-1/+1
The JGIT prereq uses `type jgit` to determine whether jgit is present. While this is usually sufficient, it won't help if the jgit found is badly broken. This wastes time running tests which fail due to no fault of our own. Use `jgit --version` instead, to guard against cases where jgit is present on the system, but will fail to run, e.g. because of some JRE issue, or missing Java dependencies. Checking that it gets far enough to process the '--version' argument isn't perfect, but seems to be good enough in practice. It's also consistent with how we detect some other dependencies, see e.g. the CURL and UNZIP prerequisites. Signed-off-by: Todd Zullinger <tmz@pobox.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-05-14tests: add a special setup where prerequisites failLibravatar Ævar Arnfjörð Bjarmason1-0/+4
As discussed in [1] there's a regression in the "pu" branch now because a new test implicitly assumed that a previous test guarded by a prerequisite had been run. Add a "GIT_TEST_FAIL_PREREQS" special test setup where we'll skip (nearly) all tests guarded by prerequisites, allowing us to easily emulate those platform where we don't run these tests. As noted in the documentation I'm adding I'm whitelisting the SYMLINKS prerequisite for now. A lot of tests started failing if we lied about not supporting symlinks. It's also unlikely that we'll have a failing test due to a hard dependency on symlinks without that being the obvious cause, so for now it's not worth the effort to make it work. 1. https://public-inbox.org/git/nycvar.QRO.7.76.6.1905131531000.44@tvgsbejvaqbjf.bet/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-25Merge branch 'jc/gettext-test-fix'Libravatar Junio C Hamano1-0/+1
The GETTEXT_POISON test option has been quite broken ever since it was made runtime-tunable, which has been fixed. * jc/gettext-test-fix: gettext tests: export the restored GIT_TEST_GETTEXT_POISON
2019-04-25Merge branch 'ab/test-lib-pass-trace2-env'Libravatar Junio C Hamano1-0/+1
Allow tracing of Git executable while running the testsuite. * ab/test-lib-pass-trace2-env: test-lib: whitelist GIT_TR2_* in the environment
2019-04-25Merge branch 'sg/test-atexit'Libravatar Junio C Hamano1-1/+27
Test framework update to more robustly clean up leftover files and processes after tests are done. * sg/test-atexit: t9811-git-p4-label-import: fix pipeline negation git p4 test: disable '-x' tracing in the p4d watchdog loop git p4 test: simplify timeout handling git p4 test: clean up the p4d cleanup functions git p4 test: use 'test_atexit' to kill p4d and the watchdog process t0301-credential-cache: use 'test_atexit' to stop the credentials helper tests: use 'test_atexit' to stop httpd git-daemon: use 'test_atexit` to stop 'git-daemon' test-lib: introduce 'test_atexit' t/lib-git-daemon: make sure to kill the 'git-daemon' process test-lib: fix interrupt handling with 'dash' and '--verbose-log -x'
2019-04-15gettext tests: export the restored GIT_TEST_GETTEXT_POISONLibravatar Junio C Hamano1-0/+1
6cdccfce ("i18n: make GETTEXT_POISON a runtime option", 2018-11-08) made the gettext-poison test a runtime option (which was a good move) and adjusted the test framework so that Git commands we run as part of the framework, as opposed to the ones that are part of the test proper, are not affected by the setting. The original value for the GIT_TEST_GETTEXT_POISON environment variable is saved away in another variable and gets unset, and then later the saved value is restored to the environment variable. But the code forgot to export the variable again, which is necessary to restore the "export" bit that was lost when the variable was unset. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-15tests: disallow the use of abbreviated options (by default)Libravatar Johannes Schindelin1-0/+7
Git's command-line parsers support uniquely abbreviated options, e.g. `git init --ba` would automatically expand `--ba` to `--bare`. This is a very convenient feature in every day life for Git users, in particular when tab completion is not available. However, it is not a good idea to rely on that in Git's test suite, as something that is a unique abbreviation of a command line option today might no longer be a unique abbreviation tomorrow. For example, if a future contribution added a new mode `git init --babyproofing` and a previously-introduced test case used the fact that `git init --ba` expanded to `git init --bare`, that future contribution would now have to touch seemingly unrelated tests just to keep the test suite from failing. So let's disallow abbreviated options in the test suite by default. Note: for ease of implementation, this patch really only touches the `parse-options` machinery: more and more hand-rolled option parsers are converted to use that internal API, and more and more scripts are converted to built-ins (naturally using the parse-options API, too), so in practice this catches most issues, and is definitely the biggest bang for the buck. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-01test-lib: whitelist GIT_TR2_* in the environmentLibravatar Ævar Arnfjörð Bjarmason1-0/+1
Add GIT_TR2_* to the whitelist of environment variables that we don't clear when running the test suite. This allows us to use the test suite to produce trace2 test data, which is handy to e.g. write consumers that collate the trace data itself. One caveat here is that we produce trace output for not *just* the tests, but also e.g. from this line in test-lib.sh: # It appears that people try to run tests without building... "${GIT_TEST_INSTALLED:-$GIT_BUILD_DIR}/git$X" >/dev/null [...] I consider this not just OK but a feature. Let's log *all* the git commands we're going to execute, not just those within test_expect_*(). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-20Merge branch 'js/stress-test-ui-tweak'Libravatar Junio C Hamano1-1/+8
Dev support. * js/stress-test-ui-tweak: tests: introduce --stress-jobs=<N> tests: let --stress-limit=<N> imply --stress
2019-03-14test-lib: introduce 'test_atexit'Libravatar Johannes Schindelin1-0/+23
When running Apache, 'git daemon', or p4d, we want to kill them at the end of the test script, otherwise a leftover daemon process will keep its port open indefinitely, and thus will interfere with subsequent executions of the same test script. So far, we stop these daemon processes "manually", i.e.: - by registering functions or commands in the trap on EXIT to stop the daemon while preserving the last seen exit code before the trap (to deal with a failure when run with '--immediate' or with interrupts by ctrl-C), - and by invoking these functions/commands last thing before 'test_done' (and sometimes restoring the test framework's default trap on EXIT, to prevent the daemons from being killed twice). On one hand, we do this inconsistently, e.g. 'git p4' tests invoke different functions in the trap on EXIT and in the last test before 'test_done', and they neither restore the test framework's default trap on EXIT nor preserve the last seen exit code. On the other hand, this is error prone, because, as shown in a previous patch in this series, any output from the cleanup commands in the trap on EXIT can prevent a proper cleanup when a test script run with '--verbose-log' and certain shells, notably 'dash', is interrupted. Let's introduce 'test_atexit', which is loosely modeled after 'test_when_finished', but has a broader scope: rather than running the commands after the current test case, run them when the test script finishes, and also run them when the test is interrupted, or exits early in case of a failure while the '--immediate' option is in effect. When running the cleanup commands at the end of a successful test, then they will be run in 'test_done' before it removes the trash directory, i.e. the cleanup commands will still be able to access any pidfiles or socket files in there. When running the cleanup commands after an interrupt or failure with '--immediate', then they will be run in the trap on EXIT. In both cases they will be run in 'test_eval_', i.e. both standard error and output of all cleanup commands will go where they should according to the '-v' or '--verbose-log' options, and thus won't cause any troubles when interrupting a test script run with '--verbose-log'. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-14test-lib: fix interrupt handling with 'dash' and '--verbose-log -x'Libravatar SZEDER Gábor1-1/+4
When a test script run with 'dash' and '--verbose-log -x' is interrupted by ctrl-C, SIGTERM, or closing the terminal window, then most of the time the registered EXIT trap actions are not executed. This is an annoying issue with tests involving daemons, because they should run cleanup commands to kill those daemon processes in the trap on EXIT, but since these cleanup commands are not executed, the daemons are left alive and keep their port open, thus interfering with subsequent execution of the same test script. The cause of this issue is the subtle combination of several factors (bear with me, or skip over the indented part): - Even when the test script is interrupted, the cleanup commands are not run in the trap on INT, TERM, or HUP, but in the trap on EXIT after the trap on the signals invokes 'exit' [1]. - According to POSIX [2]: "The environment in which the shell executes a trap on EXIT shall be identical to the environment immediately after the last command executed before the trap on EXIT was taken." Pertinent to the issue at hand is that all open file descriptors and the state of '-x' tracing should be preserved. All shells I've tried [3] preserve '-x'. Unfortunately, however: - 'dash' doesn't conform to this when it comes to open file descriptors: even when standard output and/or error are redirected somewhere when 'exit' is invoked, anything written to them in the trap on EXIT goes to the script's original stdout and stderr [4]. We can't dismiss this with a simple "it doesn't conform to POSIX, so we don't care", because 'dash' is the default /bin/sh in some of the more popular Linux distros. - As far as I can tell, POSIX doesn't explicitly say anything about the environment of trap actions for various signals. In practice it seems that most shells behave sensibly and preserve both open file descriptors and the state of '-x' tracing for the traps on INT, TERM, and HUP, including even 'dash'. The exceptions are 'mksh' and 'lksh': they do preserve '-x', but not the open file descriptors. - When a test script run with '-x' tracing enabled is interrupted, then it's very likely that the signal arrives mid-test, i.e.: - while '-x' tracing is enabled, and, consequently, our trap actions on INT, TERM, HUP, and EXIT will produce trace output as well. - while standard output and error are redirected to a log file, to the test script's original standard output and error, or to /dev/null, depending on whether the test script was run with '--verbose-log', '-v', or neither. According to the above, we can't rely on these redirections still be in effect when running the traps on INT, TERM, HUP, and/or EXIT. - When a test script is run with '--verbose-log', then the test script is re-executed with its standard output and error piped into 'tee', in order to send the "regular" non-verbose test's output both to the terminal and to the log file. When the test is interrupted, then the signal interrupts the downstream 'tee' as well. Putting these together, when a test script run with 'dash' and '--verbose-log -x' is interrupted, then 'dash' tries to write the trace output from the EXIT trap to the script's original standard error, but it very likely can't, because the 'tee' downstream of the pipe is interrupted as well. This causes the shell running the test script to die because of SIGPIPE, without running any of the commands in the EXIT trap. Disable '-x' tracing in the trap on INT, TERM, and HUP to avoid this issue, as it disables tracing in the chained trap on EXIT as well. Wrap it in a '{ ... } 2>/dev/null' block, so the trace of the command disabling the tracing doesn't go to standard error either [5]. Note that it's not only '-x' tracing that can be problematic, but any shell builtin, e.g. 'echo', that writes to standard output or error in the trap on EXIT, while a test running with 'dash' and '--verbose-log' (even without '-x') is interrupted. As far as I can tell, this is not an issue at the moment: - The cleanup commands to stop the credential-helper, Apache, or 'p4d' don't use any such shell builtins. - stop_git_daemon() does use 'say' and 'error', both wrappers around 'echo', but it redirects 'say' to fd 3, i.e. to the log file, and while 'error' does write to standard output, it comes only after the daemon was killed. - The non-builtin commands that actually stop the daemons ('kill', 'apache2 -k stop', 'git credential-cache exit') are silent, so they won't get SIGPIPE before finishing their job. [1] The trap on EXIT must run cleanup commands, because we want to stop any daemons when a test script run with '--immediate' fails and exits early with error. By chaining up the trap on signals to the trap on EXIT we can deal with cleanup commands a bit simpler, because the tests involving daemons only have to set a single trap. [2] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#trap [3] The shells I tried: dash, Bash, ksh, ksh93, mksh, lksh, yash, BusyBox sh, FreeBSD /bin/sh, NetBSD /bin/sh. [4] $ cat trap-output.sh #!/bin/sh trap "echo output; echo error >&2" EXIT { exit; } >OUT 2>ERR $ dash ./trap-output.sh output error $ wc -c OUT ERR 0 OUT 0 ERR On a related note, 'ksh', 'ksh93', and BusyBox sh don't conform to the specs in this respect, either. [5] This '{ set +x; } 2>/dev/null' trick won't help those shells that show trace output for any redirections and don't preserve open file descriptors for the trap on INT, TERM and HUP. The only such shells I'm aware of are 'mksh' and 'lksh'. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-07Merge branch 'jc/test-yes-doc'Libravatar Junio C Hamano1-1/+5
Test doc update. * jc/test-yes-doc: test: caution on our version of 'yes'
2019-03-04tests: introduce --stress-jobs=<N>Libravatar Johannes Schindelin1-1/+7
The --stress option currently accepts an argument, but it is confusing to at least this user that the argument does not define the maximal number of stress iterations, but instead the number of jobs to run in parallel per stress iteration. Let's introduce a separate option for that, whose name makes it more obvious what it is about, and let --stress=<N> error out with a helpful suggestion about the two options tha could possibly have been meant. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-04tests: let --stress-limit=<N> imply --stressLibravatar Johannes Schindelin1-0/+1
It does not make much sense that running a test with --stress-limit=<N> seemingly ignores that option because it does not stress test at all. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-13Merge branch 'sg/stress-test'Libravatar Junio C Hamano1-3/+17
Test improvement. * sg/stress-test: test-lib: fix non-portable pattern bracket expressions test-lib: make '--stress' more bisect-friendly
2019-02-11test-lib: fix non-portable pattern bracket expressionsLibravatar SZEDER Gábor1-2/+2
Use a '!' character to start a non-matching pattern bracket expression, as specified by POSIX in Shell Command Language section 2.13.1 Patterns Matching a Single Character [1]. I used '^' instead in three places in the previous three commits, to verify that the arguments of the '--stress=' and '--stress-limit=' options and the values of various '*_PORT' environment variables are valid numbers. With certain shells, at least with dash (upstream and in Ubuntu 14.04) and mksh, this led to various undesired behaviors: # error message in case of a valid number $ ~/src/dash/src/dash ./t3903-stash.sh --stress=8 error: --stress=<N> requires the number of jobs to run # not the expected error message $ ~/src/dash/src/dash ./t3903-stash.sh --stress=foo ./t3903-stash.sh: 238: test: Illegal number: foo # no error message at all?! $ mksh ./t3903-stash.sh --stress=foo $ echo $? 0 Some other shells, e.g. Bash (even in posix mode), ksh, dash in Ubuntu 16.04 or later, are apparently happy to accept '^' just as well. [1] http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_13 Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-11test: caution on our version of 'yes'Libravatar Junio C Hamano1-1/+5
During a review of a patch, we noticed that we use our own imitation of 'yes' with the limit of 99 lines. It is very tempting to lift this arbitrary limit, but the limit is there for a reason. Add an in-code comment to prevent future developers from wasting their time. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-08test-lib: make '--stress' more bisect-friendlyLibravatar SZEDER Gábor1-2/+16
Let's suppose that a test somehow becomes flaky between 'master' and 'pu', and tends to fail within the first 50 repetitions when run with '--stress'. In such a case we could use 'git bisect' to find the culprit: if the test script fails with '--stress', then the commit is definitely bad, but if it survives, say, 300 repetitions, then we could consider it good with reasonable confidence. Unfortunately, all this could only be done manually, because '--stress' would run the test script repeatedly for all eternity on a good commit, and it would exit with success even when it found a failure on a bad commit. So let's make '--stress' usable with 'git bisect run': - Make it exit with failure if a failure is found. - Add the '--stress-limit=<N>' option to repeat the test script at most N times in each of the parallel jobs, and exit with success when the limit is reached. And then we could simply run something like: $ git bisect start origin/pu master $ git bisect run sh -c 'make && cd t && ./t1234-foo.sh --stress --stress-limit=300' Sure, as a brand new feature it won't be any useful right now, but in a release or three most cooking topics will already contain this, so we could automatically bisect at least newly introduced flakiness. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-06Merge branch 'js/vsts-ci'Libravatar Junio C Hamano1-8/+130
Prepare to run test suite on Azure Pipeline. * js/vsts-ci: (22 commits) test-date: drop unused parameter to getnanos() ci: parallelize testing on Windows ci: speed up Windows phase tests: optionally skip bin-wrappers/ t0061: workaround issues with --with-dashes and RUNTIME_PREFIX tests: add t/helper/ to the PATH with --with-dashes mingw: try to work around issues with the test cleanup tests: include detailed trace logs with --write-junit-xml upon failure tests: avoid calling Perl just to determine file sizes README: add a build badge (status of the Azure Pipelines build) mingw: be more generous when wrapping up the setitimer() emulation ci: use git-sdk-64-minimal build artifact ci: add a Windows job to the Azure Pipelines definition Add a build definition for Azure DevOps ci/lib.sh: add support for Azure Pipelines tests: optionally write results as JUnit-style .xml test-date: add a subcommand to measure times in shell scripts ci: use a junction on Windows instead of a symlink ci: inherit --jobs via MAKEFLAGS in run-build-and-tests ci/lib.sh: encapsulate Travis-specific things ...
2019-02-05Merge branch 'js/test-git-installed'Libravatar Junio C Hamano1-1/+1
Test fix for Windows. * js/test-git-installed: tests: explicitly use `test-tool.exe` on Windows