summaryrefslogtreecommitdiff
path: root/t/README
AgeCommit message (Collapse)AuthorFilesLines
2018-10-19Merge branch 'bp/read-cache-parallel'Libravatar Junio C Hamano1-0/+5
A new extension to the index file has been introduced, which allows the file to be read in parallel. * bp/read-cache-parallel: read-cache: load cache entries on worker threads ieot: add Index Entry Offset Table (IEOT) extension read-cache: load cache extensions on a worker thread config: add new index.threads config setting eoie: add End of Index Entry (EOIE) extension read-cache: clean up casting and byte decoding read-cache.c: optimize reading index format v4
2018-10-19Merge branch 'bp/rename-test-env-var'Libravatar Junio C Hamano1-1/+12
Some environment variables that control the runtime options of Git used during tests are getting renamed for consistency. * bp/rename-test-env-var: t0000: do not get self-test disrupted by environment warnings preload-index: update GIT_FORCE_PRELOAD_TEST support read-cache: update TEST_GIT_INDEX_VERSION support fsmonitor: update GIT_TEST_FSMONITOR support preload-index: use git_env_bool() not getenv() for customization t/README: correct spelling of "uncommon"
2018-10-16Merge branch 'md/test-cleanup'Libravatar Junio C Hamano1-22/+47
Various test scripts have been updated for style and also correct handling of exit status of various commands. * md/test-cleanup: tests: order arguments to git-rev-list properly t9109: don't swallow Git errors upstream of pipes tests: don't swallow Git errors upstream of pipes t/*: fix ordering of expected/observed arguments tests: standardize pipe placement Documentation: add shell guidelines t/README: reformat Do, Don't, Keep in mind lists
2018-10-11config: add new index.threads config settingLibravatar Ben Peart1-0/+5
Add support for a new index.threads config setting which will be used to control the threading code in do_read_index(). A value of 0 will tell the index code to automatically determine the correct number of threads to use. A value of 1 will make the code single threaded. A value greater than 1 will set the maximum number of threads to use. For testing purposes, this setting can be overwritten by setting the GIT_TEST_INDEX_THREADS=<n> environment variable to a value greater than 0. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-10Merge branch 'bc/hash-independent-tests'Libravatar Junio C Hamano1-0/+22
Various tests have been updated to make it easier to swap the hash function used for object identification. * bc/hash-independent-tests: t5318: use test_oid for HASH_LEN t1407: make hash size independent t1406: make hash-size independent t1405: make hash size independent t1400: switch hard-coded object ID to variable t1006: make hash size independent t0064: make hash size independent t0002: abstract away SHA-1 specific constants t0000: update tests for SHA-256 t0000: use hash translation table t: add test functions to translate hash-related values
2018-10-07Documentation: add shell guidelinesLibravatar Matthew DeVore1-0/+27
Add the following guideline to Documentation/CodingGuidelines: Break overlong lines after "&&", "||", and "|", not before them; that way the command can continue to subsequent lines without backslash at the end. And the following to t/README (since it is specific to writing tests): Pipes and $(git ...) should be avoided when they swallow exit codes of Git processes Signed-off-by: Matthew DeVore <matvore@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-07t/README: reformat Do, Don't, Keep in mind listsLibravatar Matthew DeVore1-22/+20
The list of Don'ts for test writing has grown large such that it is hard to see at a glance which section an item is in. In other words, if I ignore a little bit of surrounding context, the "don'ts" look like "do's." To make the list more readable, prefix "Don't" in front of every first sentence in the items. Also, the "Keep in mind" list is out of place and awkward, because it was a very short "list" beneath two very long ones, and it seemed easy to miss under the list of "don'ts," and it only had one item. So move this item to the list of "do's" and phrase as "Remember..." Signed-off-by: Matthew DeVore <matvore@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-28preload-index: update GIT_FORCE_PRELOAD_TEST supportLibravatar Ben Peart1-0/+3
Rename GIT_FORCE_PRELOAD_TEST to GIT_TEST_PRELOAD_INDEX for consistency with the other GIT_TEST_ special setups and properly document its use. Add logic in t/test-lib.sh to give a warning when the old variable is set to let people know they need to update their environment to use the new variable. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-28read-cache: update TEST_GIT_INDEX_VERSION supportLibravatar Ben Peart1-0/+4
Rename TEST_GIT_INDEX_VERSION to GIT_TEST_INDEX_VERSION for consistency with the other GIT_TEST_ special setups and properly document its use. Add logic in t/test-lib.sh to give a warning when the old variable is set to let people know they need to update their environment to use the new variable. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-28fsmonitor: update GIT_TEST_FSMONITOR supportLibravatar Ben Peart1-0/+4
Rename GIT_FSMONITOR_TEST to GIT_TEST_FSMONITOR for consistency with the other GIT_TEST_ special setups and properly document its use. Add logic in t/test-lib.sh to give a warning when the old variable is set to let people know they need to update their environment to use the new variable. Remove the outdated instructions on how to run the test suite utilizing fsmonitor now that it is properly documented in t/README. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-20t/README: correct spelling of "uncommon"Libravatar Ben Peart1-1/+1
Correct a spelling error in the documentation for GIT_TEST_OE_DELTA_SIZE. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-17Merge branch 'ds/commit-graph-tests'Libravatar Junio C Hamano1-0/+4
We can now optionally run tests with commit-graph enabled. * ds/commit-graph-tests: commit-graph: define GIT_TEST_COMMIT_GRAPH
2018-09-17Merge branch 'nd/unpack-trees-with-cache-tree'Libravatar Junio C Hamano1-0/+4
The unpack_trees() API used in checking out a branch and merging walks one or more trees along with the index. When the cache-tree in the index tells us that we are walking a tree whose flattened contents is known (i.e. matches a span in the index), as linearly scanning a span in the index is much more efficient than having to open tree objects recursively and listing their entries, the walk can be optimized, which is done in this topic. * nd/unpack-trees-with-cache-tree: Document update for nd/unpack-trees-with-cache-tree cache-tree: verify valid cache-tree in the test suite unpack-trees: add missing cache invalidation unpack-trees: reuse (still valid) cache-tree from src_index unpack-trees: reduce malloc in cache-tree walk unpack-trees: optimize walking same trees with cache-tree unpack-trees: add performance tracing trace.h: support nested performance tracing
2018-09-13t: add test functions to translate hash-related valuesLibravatar brian m. carlson1-0/+22
Add several test functions to make working with various hash-related values easier. Add test_oid_init, which loads common hash-related constants and placeholder object IDs from the newly added files in t/oid-info. Provide values for these constants for both SHA-1 and SHA-256. Add test_oid_cache, which accepts data on standard input in the form of hash-specific key-value pairs that can be looked up later, using the same format as the files in t/oid-info. Document this format in a t/oid-info/README directory so that it's easier to use in the future. Add test_oid, which is used to specify look up a per-hash value (produced on standard output) based on the key specified as its argument. Usually the data to be looked up will be a hash-related constant (such as the size of the hash in binary or hexadecimal), a well-known or placeholder object ID (such as the all-zeros object ID or one consisting of "deadbeef" repeated), or something similar. For these reasons, test_oid will usually be used within a command substitution. Consequently, redirect the error output to standard error, since otherwise it will not be displayed. Add test_detect_hash, which currently only detects SHA-1, and test_set_hash, which can be used to set a different hash algorithm for test purposes. In the future, test_detect_hash will learn to actually detect the hash depending on how the testsuite is to be run. Use the local keyword within these functions to avoid overwriting other shell variables. We have had a test balloon in place for a couple of releases to catch shells that don't have this keyword and have not received any reports of failure. Note that the varying usages of local used here are supported by all common open-source shells supporting the local keyword. Test these new functions as part of t0000, which also serves to demonstrate basic usage of them. In addition, add documentation on how to format the lookup data and how to use the test functions. Implement two basic lookup charts, one for common invalid or synthesized object IDs, and one for various facts about the hash function in use. Provide versions of the data for both SHA-1 and SHA-256. Since we use shell variables for storage, names used for lookup can currently consist only of shell identifier characters. If this is a problem in the future, we can hash the names before use. Improved-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-08-29commit-graph: define GIT_TEST_COMMIT_GRAPHLibravatar Derrick Stolee1-0/+4
The commit-graph feature is tested in isolation by t5318-commit-graph.sh and t6600-test-reach.sh, but there are many more interesting scenarios involving commit walks. Many of these scenarios are covered by the existing test suite, but we need to maintain coverage when the optional commit-graph structure is not present. To allow running the full test suite with the commit-graph present, add a new test environment variable, GIT_TEST_COMMIT_GRAPH. Similar to GIT_TEST_SPLIT_INDEX, this variable makes every Git command try to load the commit-graph when parsing commits, and writes the commit-graph file after every 'git commit' command. There are a few tests that rely on commits not existing in pack-files to trigger important events, so manually set GIT_TEST_COMMIT_GRAPH to false for the necessary commands. There is one test in t6024-recursive-merge.sh that relies on the merge-base algorithm picking one of two ambiguous merge-bases, and the commit-graph feature changes which merge-base is picked. Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-08-27Document update for nd/unpack-trees-with-cache-treeLibravatar Nguyễn Thái Ngọc Duy1-0/+4
Fix an incorrect comment in the new code added in b4da37380b (unpack-trees: optimize walking same trees with cache-tree - 2018-08-18) and document about the new test variable that is enabled by default in test-lib.sh in 4592e6080f (cache-tree: verify valid cache-tree in the test suite - 2018-08-18) Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-23pack-objects: fix performance issues on packing large deltasLibravatar Nguyễn Thái Ngọc Duy1-0/+4
Let's start with some background about oe_delta_size() and oe_set_delta_size(). If you already know, skip the next paragraph. These two are added in 0aca34e826 (pack-objects: shrink delta_size field in struct object_entry - 2018-04-14) to help reduce 'struct object_entry' size. The delta size field in this struct is reduced to only contain max 1MB. So if any new delta is produced and larger than 1MB, it's dropped because we can't really save such a large size anywhere. Fallback is provided in case existing packfiles already have large deltas, then we can retrieve it from the pack. While this should help small machines repacking large repos without large deltas (i.e. less memory pressure), dropping large deltas during the delta selection process could end up with worse pack files. And if existing packfiles already have >1MB delta and pack-objects is instructed to not reuse deltas, all of them will be dropped on the floor, and the resulting pack would be definitely bigger. There is also a regression in terms of CPU/IO if we have large on-disk deltas because fallback code needs to parse the pack every time the delta size is needed and just access to the mmap'd pack data is enough for extra page faults when memory is under pressure. Both of these issues were reported on the mailing list. Here's some numbers for comparison. Version Pack (MB) MaxRSS(kB) Time (s) ------- --------- ---------- -------- 2.17.0 5498 43513628 2494.85 2.18.0 10531 40449596 4168.94 This patch provides a better fallback that is - cheaper in terms of cpu and io because we won't have to read existing pack files as much - better in terms of pack size because the pack heuristics is back to 2.17.0 time, we do not drop large deltas at all If we encounter any delta (on-disk or created during try_delta phase) that is larger than the 1MB limit, we stop using delta_size_ field for this because it can't contain such size anyway. A new array of delta size is dynamically allocated and can hold all the deltas that 2.17.0 can. This array only contains delta sizes that delta_size_ can't contain. With this, we do not have to drop deltas in try_delta() anymore. Of course the downside is we use slightly more memory, even compared to 2.17.0. But since this is considered an uncommon case, a bit more memory consumption should not be a problem. Delta size limit is also raised from 1MB to 16MB to better cover common case and avoid that extra memory consumption (99.999% deltas in this reported repo are under 12MB; Jeff noted binary artifacts topped out at about 3MB in some other private repos). Other fields are shuffled around to keep this struct packed tight. We don't use more memory in common case even with this limit update. A note about thread synchronization. Since this code can be run in parallel during delta searching phase, we need a mutex. The realloc part in packlist_alloc() is not protected because it only happens during the object counting phase, which is always single-threaded. Access to e->delta_size_ (and by extension pack->delta_size[e - pack->objects]) is unprotected as before, the thread scheduler in pack-objects must make sure "e" is never updated by two different threads. The area under the new lock is as small as possible, avoiding locking at all in common case, since lock contention with high thread count could be expensive (most blobs are small enough that delta compute time is short and we end up taking the lock very often). The previous attempt to always hold a lock in oe_delta_size() and oe_set_delta_size() increases execution time by 33% when repacking linux.git with with 40 threads. Reported-by: Elijah Newren <newren@gmail.com> Helped-by: Elijah Newren <newren@gmail.com> Helped-by: Jeff King <peff@peff.net> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-04-16pack-objects: shrink size field in struct object_entryLibravatar Nguyễn Thái Ngọc Duy1-0/+6
It's very very rare that an uncompressed object is larger than 4GB (partly because Git does not handle those large files very well to begin with). Let's optimize it for the common case where object size is smaller than this limit. Shrink size field down to 31 bits and one overflow bit. If the size is too large, we read it back from disk. As noted in the previous patch, we need to return the delta size instead of canonical size when the to-be-reused object entry type is a delta instead of a canonical one. Add two compare helpers that can take advantage of the overflow bit (e.g. if the file is 4GB+, chances are it's already larger than core.bigFileThreshold and there's no point in comparing the actual value). Another note about oe_get_size_slow(). This function MUST be thread safe because SIZE() macro is used inside try_delta() which may run in parallel. Outside parallel code, no-contention locking should be dirt cheap (or insignificant compared to i/o access anyway). To exercise this code, it's best to run the test suite with something like make test GIT_TEST_OE_SIZE=4 which forces this code on all objects larger than 3 bytes. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-04-16pack-objects: move in_pack out of struct object_entryLibravatar Nguyễn Thái Ngọc Duy1-0/+5
Instead of using 8 bytes (on 64 bit arch) to store a pointer to a pack. Use an index instead since the number of packs should be relatively small. This limits the number of packs we can handle to 1k. Since we can't be sure people can never run into the situation where they have more than 1k pack files. Provide a fall back route for it. If we find out they have too many packs, the new in_pack_by_idx[] array (which has at most 1k elements) will not be used. Instead we allocate in_pack[] array that holds nr_objects elements. This is similar to how the optional in_pack_pos field is handled. The new simple test is just to make sure the too-many-packs code path is at least executed. The true test is running make test GIT_TEST_FULL_IN_PACK_ARRAY=1 to take advantage of other special case tests. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-04-16read-cache.c: make $GIT_TEST_SPLIT_INDEX booleanLibravatar Nguyễn Thái Ngọc Duy1-0/+11
While at there, document about this special mode when running the test suite. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-14Merge branch 'sg/test-x'Libravatar Junio C Hamano1-3/+20
Running test scripts under -x option of the shell is often not a useful way to debug them, because the error messages from the commands tests try to capture and inspect are contaminated by the tracing output by the shell. An earlier work done to make it more pleasant to run tests under -x with recent versions of bash is extended to cover posix shells that do not support BASH_XTRACEFD. * sg/test-x: travis-ci: run tests with '-x' tracing t/README: add a note about don't saving stderr of compound commands t1510-repo-setup: mark as untraceable with '-x' t9903-bash-prompt: don't check the stderr of __git_ps1() t5570-git-daemon: don't check the stderr of a subshell t5526: use $TRASH_DIRECTORY to specify the path of GIT_TRACE log file t5500-fetch-pack: don't check the stderr of a subshell t3030-merge-recursive: don't check the stderr of a subshell t1507-rev-parse-upstream: don't check the stderr of a shell function t: add means to disable '-x' tracing for individual test scripts t: prevent '-x' tracing from interfering with test helpers' stderr
2018-02-28t/README: add a note about don't saving stderr of compound commandsLibravatar SZEDER Gábor1-3/+17
Explain in 't/README' why it is a bad idea to redirect and verify the stderr of compound commands, in the hope that future contributions will follow this advice and the test suite will keep working with '-x' tracing and /bin/sh. While at it, since we can now run the test suite with '-x' without needing a Bash version supporting BASH_XTRACEFD, remove the now outdated caution note about non-Bash shells from the description of the '-x' option. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-02-27t: add means to disable '-x' tracing for individual test scriptsLibravatar SZEDER Gábor1-0/+3
The previous patch resolved most of the test failures caused by running our test suite with '-x' tracing and /bin/sh, and the following patches in this series will resolve almost all of the remaining failures. Unfortunately, not yet all. Add means to disable '-x' tracing for individual test scripts by setting the $test_untraceable variable to a non-empty value in the test script before sourcing 'test-lib.sh'. However, since '-x' tracing is not an issue with recent Bash versions supporting BASH_XTRACEFD, i.e. v4.1 and later, don't disable tracing when the test script is run with such a Bash version even when $test_untraceable is set. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-02-27Merge branch 'sg/doc-test-must-fail-args'Libravatar Junio C Hamano1-2/+12
Devdoc update. * sg/doc-test-must-fail-args: t: document 'test_must_fail ok=<signal-name>'
2018-02-12t: document 'test_must_fail ok=<signal-name>'Libravatar SZEDER Gábor1-2/+12
Since 'test_might_fail' is implemented as a thin wrapper around 'test_must_fail', it also accepts the same options. Mention this in the docs as well. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-12-13Merge branch 'tg/t-readme-updates'Libravatar Junio C Hamano1-4/+6
Developer doc updates. * tg/t-readme-updates: t/README: document test_cmp_rev t/README: remove mention of adding copyright notices
2017-11-27t/README: document test_cmp_revLibravatar Thomas Gummerer1-0/+5
test_cmp_rev is a useful function that's used in quite a few test scripts. It is however not documented in t/README. Document it. Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-11-27t/README: remove mention of adding copyright noticesLibravatar Thomas Gummerer1-4/+1
We generally no longer include copyright notices in new test scripts. However t/README still mentions it as something to include at the top of every new script. Remove that mention as it's outdated. Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-11-24test-lib: add LIBPCRE1 & LIBPCRE2 prerequisitesLibravatar Ævar Arnfjörð Bjarmason1-0/+12
Add LIBPCRE1 and LIBPCRE2 prerequisites which are true when git is compiled with USE_LIBPCRE1=YesPlease or USE_LIBPCRE2=YesPlease, respectively. The syntax of PCRE1 and PCRE2 isn't the same in all cases (see pcresyntax(3) and pcre2syntax(3)). If test are added that test for those they'll need to be guarded by these new prerequisites. The subsequent patch will make use of LIBPCRE2, so LIBPCRE1 isn't strictly needed for now, but let's add it for consistency and so that checking for it doesn't have to be done with the less obvious "PCRE, !LIBPCRE2", which while semantically the same is more confusing, and would lead to bugs if PCRE v3 is ever released as the tests would mean v1, not any non-v2 version. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-19t/README: fix typo and grammatically improve a sentenceLibravatar Kaartic Sivaraam1-3/+3
Signed-off-by: Kaartic Sivaraam <kaarticsivaraam91196@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-26test-lib: add a PTHREADS prerequisiteLibravatar Ævar Arnfjörð Bjarmason1-0/+4
Add a PTHREADS prerequisite which is false when git is compiled with NO_PTHREADS=YesPlease. There's lots of custom code that runs when threading isn't available, but before this prerequisite there was no way to test it. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-21test-lib: rename the LIBPCRE prerequisite to PCRELibravatar Ævar Arnfjörð Bjarmason1-2/+2
Rename the LIBPCRE prerequisite to PCRE. This is for preparation for libpcre2 support, where having just "LIBPCRE" would be confusing as it implies v1 of the library. None of these tests are incompatible between versions 1 & 2 of libpcre, it's less confusing to give them a more general name to make it clear that they work on both library versions. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-26t/README: clarify the test_have_prereq documentationLibravatar Ævar Arnfjörð Bjarmason1-3/+5
Clarify the test_have_prereq documentation so that it's clear in the reader's mind when the text says "most common use of this directly" what the answer to "as opposed to what?" is. Usually this function isn't used in lieu of using the prerequisite support built into test_expect_*, mention that explicitly. This changes documentation that I added in commit 9a897893a7 ("t/README: Document the prereq functions, and 3-arg test_*", 2010-07-02). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-22t/README: change "Inside <X> part" to "Inside the <X> part"Libravatar Ævar Arnfjörð Bjarmason1-1/+1
Change the mention of "Inside the <script> part, the standard output..." to use the definite article, which makes more sense in this context. This changes documentation I originally added back in commit 20873f45e7 ("t/README: Document the do's and don'ts of tests", 2010-07-02). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-22t/README: link to metacpan.org, not search.cpan.orgLibravatar Ævar Arnfjörð Bjarmason1-1/+1
Change a link to the web version of the TAP::Parser::Grammar documentation to link to metacpan.org instead of search.cpan.org. This is something I added back in commit 20873f45e7 ("t/README: Document the do's and don'ts of tests", 2010-07-02), at the time search.cpan.org was the more actively maintained CPAN web-interface, nowadays that's metacpan.org. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-26Merge branch 'jk/tap-verbose-fix'Libravatar Junio C Hamano1-0/+6
The Travis CI configuration we ship ran the tests with --verbose option but this risks non-TAP output that happens to be "ok" to be misinterpreted as TAP signalling a test that passed. This resulted in unnecessary failure. This has been corrected by introducing a new mode to run our tests in the test harness to send the verbose output separately to the log file. * jk/tap-verbose-fix: test-lib: bail out when "-v" used under "prove" travis: use --verbose-log test option test-lib: add --verbose-log option test-lib: handle TEST_OUTPUT_DIRECTORY with spaces
2016-10-21test-lib: add --verbose-log optionLibravatar Jeff King1-0/+6
The "--verbose" option redirects output from arbitrary test commands to stdout. This is useful for examining the output manually, like: ./t5547-push-quarantine.sh -v | less But it also means that the output is intermingled with the TAP directives, which can confuse a TAP parser like "prove". This has always been a potential problem, but became an issue recently when one test happened to output the word "ok" on a line by itself, which prove interprets as a test success: $ prove t5547-push-quarantine.sh :: -v t5547-push-quarantine.sh .. 1/? To dest.git * [new branch] HEAD -> master To dest.git ! [remote rejected] reject -> reject (pre-receive hook declined) error: failed to push some refs to 'dest.git' fatal: git cat-file d08c8eba97f4e683ece08654c7c8d2ba0c03b129: bad file t5547-push-quarantine.sh .. Failed -1/4 subtests Test Summary Report ------------------- t5547-push-quarantine.sh (Wstat: 0 Tests: 5 Failed: 0) Parse errors: Tests out of sequence. Found (2) but expected (3) Tests out of sequence. Found (3) but expected (4) Tests out of sequence. Found (4) but expected (5) Bad plan. You planned 4 tests but ran 5. Files=1, Tests=5, 0 wallclock secs ( 0.01 usr + 0.01 sys = 0.02 CPU) Result: FAIL One answer is "if it hurts, don't do it", but that's not quite the whole story. The Travis tests use "--verbose --tee" so that they can get the benefit of prove's parallel options, along with a verbose log in case there is a failure. We just need the verbose output to go to the log, but keep stdout clean. Getting this right turns out to be surprisingly difficult. Here's the progression of alternatives I considered: 1. Add an option to write verbose output to stderr. This is hard to capture, though, because we want each test to have its own log (because they're all run in parallel and the jumbled output would be useless). 2. Add an option to write verbose output to a file in test-results. This works, but the log is missing all of the non-verbose output, which gives context. 3. Like (2), but teach say_color() to additionally output to the log. This mostly works, but misses any output that happens outside of the say() functions (which isn't a lot, but is a potential maintenance headache). 4. Like (2), but make the log file the same as the "--tee" file. That almost works, but now we have two processes opening the same file. That gives us two separate descriptors, each with their own idea of the current position. They'll each start writing at offset 0, and overwrite each other's data. 5. Like (4), but in each case open the file for appending. That atomically positions each write at the end of the file. It's possible we may still get sheared writes between the two processes, but this is already the case when writing to stdout. It's not a problem in practice because the test harness generally waits for snippets to finish before writing the TAP output. We can ignore buffering issues with tee, because POSIX mandates that it does not buffer. Likewise, POSIX specifies "tee -a", so it should be available everywhere. This patch implements option (5), which seems to work well in practice. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-11Spelling fixesLibravatar Ville Skyttä1-1/+1
<BAD> <CORRECTED> accidently accidentally commited committed dependancy dependency emtpy empty existance existence explicitely explicitly git-upload-achive git-upload-archive hierachy hierarchy indegee indegree intial initial mulitple multiple non-existant non-existent precendence. precedence. priviledged privileged programatically programmatically psuedo-binary pseudo-binary soemwhere somewhere successfull successful transfering transferring uncommited uncommitted unkown unknown usefull useful writting writing Signed-off-by: Ville Skyttä <ville.skytta@iki.fi> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-05-31Merge branch 'jk/test-send-sh-x-trace-elsewhere' into maintLibravatar Junio C Hamano1-3/+3
Running tests with '-x' option to trace the individual command executions is a useful way to debug test scripts, but some tests that capture the standard error stream and check what the command said can be broken with the trace output mixed in. When running our tests under "bash", however, we can redirect the trace output to another file descriptor to keep the standard error of programs being tested intact. * jk/test-send-sh-x-trace-elsewhere: test-lib: set BASH_XTRACEFD automatically
2016-05-17Merge branch 'jk/test-send-sh-x-trace-elsewhere'Libravatar Junio C Hamano1-3/+3
Running tests with '-x' option to trace the individual command executions is a useful way to debug test scripts, but some tests that capture the standard error stream and check what the command said can be broken with the trace output mixed in. When running our tests under "bash", however, we can redirect the trace output to another file descriptor to keep the standard error of programs being tested intact. * jk/test-send-sh-x-trace-elsewhere: test-lib: set BASH_XTRACEFD automatically
2016-05-11test-lib: set BASH_XTRACEFD automaticallyLibravatar Jeff King1-3/+3
Passing "-x" to a test script enables the shell's "set -x" tracing, which can help with tracking down the command that is causing a failure. Unfortunately, it can also _cause_ failures in some tests that redirect the stderr of a shell function. Inside the function the shell continues to respect "set -x", and the trace output is collected along with whatever stderr is generated normally by the function. You can see an example of this by running: ./t0040-parse-options.sh -x -i which will fail immediately in the first test, as it expects: test_must_fail some-cmd 2>output.err to leave output.err empty (but with "-x" it has our trace output). Unfortunately there isn't a portable or scalable solution to this. We could teach test_must_fail to disable "set -x", but that doesn't help any of the other functions or subshells. However, we can work around it by pointing the "set -x" output to our descriptor 4, which always points to the original stderr of the test script. Unfortunately this only works for bash, but it's better than nothing (and other shells will just ignore the BASH_XTRACEFD variable). The patch itself is a simple one-liner, but note the caveats in the accompanying comments. Automatic tests for our "-x" option may be a bit too meta (and a pain, because they are bash-specific), but I did confirm that it works correctly both with regular "-x" and with "--verbose-only=1". This works because the latter flips "set -x" off and on for particular tests (if it didn't, we would get tracing for all tests, as going to descriptor 4 effectively circumvents the verbose flag). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-10-30test: facilitate debugging Git executables in tests with gdbLibravatar Johannes Schindelin1-0/+5
When prefixing a Git call in the test suite with 'debug ', it will now be run with GDB, allowing the developer to debug test failures more conveniently. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-20t/test-lib: introduce --chain-lint optionLibravatar Jeff King1-0/+10
It's easy to miss an "&&"-chain in a test script, like: test_expect_success 'check something important' ' cmd1 && cmd2 cmd3 ' The test harness will notice if cmd3 fails, but a failure of cmd1 or cmd2 will go unnoticed, as their exit status is lost after cmd3 runs. The toy example above is easy to spot because the "cmds" are all the same length, but real code is much more complicated. It's also difficult to detect these situations by statically analyzing the shell code with regexps (like the check-non-portable-shell script does); there's too much context required to know whether a &&-chain is appropriate on a given line or not. This patch instead lets the shell check each test by sticking a command with a specific and unusual return code at the top of each test, like: (exit 117) && cmd1 && cmd2 cmd3 In a well-formed test, the non-zero exit from the first command prevents any of the rest from being run, and the test's exit code is 117. In a bad test (like the one above), the 117 is lost, and cmd3 is run. When we encounter a failure of this check, we abort the test script entirely. For one thing, we have no clue which subset of the commands in the test snippet were actually run. Running further tests would be pointless, because we're now in an unknown state. And two, this is not a "test failure" in the traditional sense. The test script is buggy, not the code it is testing. We should be able to fix these problems in the script once, and not have them come back later as a regression in git's code. After checking a test snippet for --chain-lint, we do still run the test itself. We could actually have a pure-lint mode which just checks each test, but there are a few reasons not to. One, because the tests are executing arbitrary code, which could impact the later environment (e.g., that could impact which set of tests we run at all). And two, because a pure-lint mode would still be expensive to run, because a significant amount of code runs outside of the test_expect_* blocks. Instead, this option is designed to be used as part of a normal test suite run, where it adds very little overhead. Turning on this option detects quite a few problems in existing tests, which will be fixed in subsequent patches. However, there are a number of places it cannot reach: - it cannot find a failure to break out of loops on error, like: cmd1 && for i in a b c; do cmd2 $i done && cmd3 which will not notice failures of "cmd2 a" or "cmd b" - it cannot find a missing &&-chain inside a block or subfunction, like: foo () { cmd1 cmd2 } foo && bar which will not notice a failure of cmd1. - it only checks tests that you run; every platform will have some tests skipped due to missing prequisites, so it's impossible to say from one run that the test suite is free of broken &&-chains. However, all tests get run by _somebody_, so eventually we will notice problems. - it does not operate on test_when_finished or prerequisite blocks. It could, but these tends to be much shorter and less of a problem, so I punted on them in this patch. This patch was inspired by an earlier patch by Jonathan Nieder: http://article.gmane.org/gmane.comp.version-control.git/235913 This implementation and all bugs are mine. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-12-22Merge branch 'jc/refer-to-t-readme-from-submitting-patches'Libravatar Junio C Hamano1-1/+2
* jc/refer-to-t-readme-from-submitting-patches: t/README: justify why "! grep foo" is sufficient SubmittingPatches: refer to t/README for tests
2014-11-24t/README: justify why "! grep foo" is sufficientLibravatar Junio C Hamano1-1/+2
We require use of test_must_fail to check expected non-zero exit by Git itself, but discourage test_must_fail to be used for checking exit status of non Git commands that are supplied by the system. The current text explains the reason for the former but not the latter. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-10-13test-lib.sh: support -x option for shell-tracingLibravatar Jeff King1-0/+6
Usually running a test under "-v" makes it clear which command is failing. However, sometimes it can be useful to also see a complete trace of the shell commands being run in the test. You can do so without any support from the test suite by running "sh -x tXXXX-foo.sh". However, this produces quite a large bit of output, as we see a trace of the entire test suite. This patch instead introduces a "-x" option to the test scripts (i.e., "./tXXXX-foo.sh -x"). When enabled, this turns on "set -x" only for the tests themselves. This can still be a bit verbose, but should keep things to a more manageable level. You can even use "--verbose-only" to see the trace only for a specific test. The implementation is a little invasive. We turn on the "set -x" inside the "eval" of the test code. This lets the eval itself avoid being reported in the trace (which would be long, and redundant with the verbose listing we already showed). And then after the eval runs, we do some trickery with stderr to avoid showing the "set +x" to the user. We also show traces for test_cleanup functions (since they can impact the test outcome, too). However, we do avoid running the noop ":" cleanup (the default if the test does not use test_cleanup at all), as it creates unnecessary noise in the "set -x" output. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-16Merge branch 'ib/test-selectively-run'Libravatar Junio C Hamano1-4/+81
Allow specifying only certain individual test pieces to be run using a range notation (e.g. "t1234-test.sh --run='1-4 6 8 9-'"). * ib/test-selectively-run: t0000-*.sh: fix the GIT_SKIP_TESTS sub-tests test-lib: '--run' to run only specific tests test-lib: tests skipped by GIT_SKIP_TESTS say so test-lib: document short options in t/README
2014-06-10test doc: test_write_lines does not split its argumentsLibravatar Jonathan Nieder1-4/+3
test_write_lines carefully quotes its arguments as "$@", so test_write_lines "a b" c writes two lines as requested, not three. Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-10test: add test_write_lines helperLibravatar Michael S. Tsirkin1-0/+22
API and implementation as suggested by Junio. Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-06test-lib: '--run' to run only specific testsLibravatar Ilya Bobyr1-4/+77
Allow better control of the set of tests that will be executed for a single test suite. Mostly useful while debugging or developing as it allows to focus on a specific test. Signed-off-by: Ilya Bobyr <ilya.bobyr@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>