summaryrefslogtreecommitdiff
path: root/ci
AgeCommit message (Collapse)AuthorFilesLines
2021-10-11Merge branch 'ab/sanitize-leak-ci'Libravatar Junio C Hamano2-2/+9
CI learns to run the leak sanitizer builds. * ab/sanitize-leak-ci: tests: add a test mode for SANITIZE=leak, run it in CI Makefile: add SANITIZE=leak flag to GIT-BUILD-OPTIONS
2021-09-23tests: add a test mode for SANITIZE=leak, run it in CILibravatar Ævar Arnfjörð Bjarmason2-2/+9
While git can be compiled with SANITIZE=leak, we have not run regression tests under that mode. Memory leaks have only been fixed as one-offs without structured regression testing. This change adds CI testing for it. We'll now build and small set of whitelisted t00*.sh tests under Linux with a new job called "linux-leaks". The CI target uses a new GIT_TEST_PASSING_SANITIZE_LEAK=true test mode. When running in that mode, we'll assert that we were compiled with SANITIZE=leak. We'll then skip all tests, except those that we've opted-in by setting "TEST_PASSES_SANITIZE_LEAK=true". A test setting "TEST_PASSES_SANITIZE_LEAK=true" setting can in turn make use of the "SANITIZE_LEAK" prerequisite, should they wish to selectively skip tests even under "GIT_TEST_PASSING_SANITIZE_LEAK=true". In the preceding commit we started doing this in "t0004-unwritable.sh" under SANITIZE=leak, now it'll combine nicely with "GIT_TEST_PASSING_SANITIZE_LEAK=true". This is how tests that don't set "TEST_PASSES_SANITIZE_LEAK=true" will be skipped under GIT_TEST_PASSING_SANITIZE_LEAK=true: $ GIT_TEST_PASSING_SANITIZE_LEAK=true ./t0001-init.sh 1..0 # SKIP skip all tests in t0001 under SANITIZE=leak, TEST_PASSES_SANITIZE_LEAK not set The intent is to add more TEST_PASSES_SANITIZE_LEAK=true annotations as follow-up change, but let's start small to begin with. In ci/run-build-and-tests.sh we make use of the default "*" case to run "make test" without any GIT_TEST_* modes. SANITIZE=leak is known to fail in combination with GIT_TEST_SPLIT_INDEX=true in t0016-oidmap.sh, and we're likely to have other such failures in various GIT_TEST_* modes. Let's focus on getting the base tests passing, we can expand coverage to GIT_TEST_* modes later. It would also be possible to implement a more lightweight version of this by only relying on setting "LSAN_OPTIONS". See <YS9OT/pn5rRK9cGB@coredump.intra.peff.net>[1] and <YS9ZIDpANfsh7N+S@coredump.intra.peff.net>[2] for a discussion of that. I've opted for this approach of adding a GIT_TEST_* mode instead because it's consistent with how we handle other special test modes. Being able to add a "!SANITIZE_LEAK" prerequisite and calling "test_done" early if it isn't satisfied also means that we can more incrementally add regression tests without being forced to fix widespread and hard-to-fix leaks at the same time. We have tests that do simple checking of some tool we're interested in, but later on in the script might be stressing trace2, or common sources of leaks like "git log" in combination with the tool (e.g. the commit-graph tests). To be clear having a prerequisite could also be accomplished by using "LSAN_OPTIONS" directly. On the topic of "LSAN_OPTIONS": It would be nice to have a mode to aggregate all failures in our various scripts, see [2] for a start at doing that which sets "log_path" in "LSAN_OPTIONS". I've punted on that for now, it can be added later. As of writing this we've got major regressions between master..seen, i.e. the t000*.sh tests and more fixed since 31f9acf9ce2 (Merge branch 'ah/plugleaks', 2021-08-04) have regressed recently. See the discussion at <87czsv2idy.fsf@evledraar.gmail.com>[3] about the lack of this sort of test mode, and 0e5bba53af (add UNLEAK annotation for reducing leak false positives, 2017-09-08) for the initial addition of SANITIZE=leak. See also 09595ab381 (Merge branch 'jk/leak-checkers', 2017-09-19), 7782066f67 (Merge branch 'jk/apache-lsan', 2019-05-19) and the recent 936e58851a (Merge branch 'ah/plugleaks', 2021-05-07) for some of the past history of "one-off" SANITIZE=leak (and more) fixes. As noted in [5] we can't support this on OSX yet until Clang 14 is released, at that point we'll probably want to resurrect that "osx-leaks" job. 1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer 2. https://lore.kernel.org/git/YS9OT%2Fpn5rRK9cGB@coredump.intra.peff.net/ 3. https://lore.kernel.org/git/87czsv2idy.fsf@evledraar.gmail.com/ 4. https://lore.kernel.org/git/YS9ZIDpANfsh7N+S@coredump.intra.peff.net/ 5. https://lore.kernel.org/git/20210916035603.76369-1-carenas@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Carlo Marcelo Arenas Belón <carenas@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-20Merge branch 'tb/multi-pack-bitmaps'Libravatar Junio C Hamano1-0/+1
The reachability bitmap file used to be generated only for a single pack, but now we've learned to generate bitmaps for history that span across multiple packfiles. * tb/multi-pack-bitmaps: (29 commits) pack-bitmap: drop bitmap_index argument from try_partial_reuse() pack-bitmap: drop repository argument from prepare_midx_bitmap_git() p5326: perf tests for MIDX bitmaps p5310: extract full and partial bitmap tests midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' t7700: update to work with MIDX bitmap test knob t5319: don't write MIDX bitmaps in t5319 t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP t5326: test multi-pack bitmap behavior t/helper/test-read-midx.c: add --checksum mode t5310: move some tests to lib-bitmap.sh pack-bitmap: write multi-pack bitmaps pack-bitmap: read multi-pack bitmaps pack-bitmap.c: avoid redundant calls to try_partial_reuse pack-bitmap.c: introduce 'bitmap_is_preferred_refname()' pack-bitmap.c: introduce 'nth_bitmap_object_oid()' pack-bitmap.c: introduce 'bitmap_num_objects()' midx: avoid opening multiple MIDXs when writing midx: close linked MIDXs, avoid leaking memory ...
2021-09-10Merge branch 'cb/ci-build-pedantic'Libravatar Junio C Hamano2-3/+11
CI update. * cb/ci-build-pedantic: ci: run a pedantic build as part of the GitHub workflow
2021-09-01midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'Libravatar Taylor Blau1-0/+1
Introduce a new 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment variable to also write a multi-pack bitmap when 'GIT_TEST_MULTI_PACK_INDEX' is set. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-11ci: run a pedantic build as part of the GitHub workflowLibravatar Carlo Marcelo Arenas Belón2-3/+11
similar to the recently added sparse task, it is nice to know as early as possible. add a dockerized build using fedora (that usually has the latest gcc) to be ahead of the curve and avoid older ISO C issues at the same time. Signed-off-by: Carlo Marcelo Arenas Belón <carenas@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-26ci/install-dependencies: handle "sparse" job package installsLibravatar Jeff King1-0/+5
This just matches the style/location of the package installation for other jobs. There should be no functional change. I did flip the order of the options and command-name ("-y update" instead of "update -y") for consistency with other lines in the same file. Note also that we have to reorder the dependency install with the "checkout" action, so that we actually have the "ci" scripts available. Signed-off-by: Jeff King <peff@peff.net> Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-08Merge branch 'dd/svn-test-wo-locale-a'Libravatar Junio C Hamano1-0/+1
"git-svn" tests assumed that "locale -a", which is used to pick an available UTF-8 locale, is available everywhere. A knob has been introduced to allow testers to specify a suitable locale to use. * dd/svn-test-wo-locale-a: t: use user-specified utf-8 locale for testing svn
2021-06-08t: use user-specified utf-8 locale for testing svnLibravatar Đoàn Trần Công Danh1-0/+1
In some test-cases, UTF-8 locale is required. To find such locale, we're using the first available UTF-8 locale that returned by "locale -a". However, the locale(1) utility is unavailable on some systems, e.g. Linux with musl libc. However, without "locale -a", we can't guess provided UTF-8 locale. Add a Makefile knob GIT_TEST_UTF8_LOCALE and activate it for linux-musl in our CI system. Rename t/lib-git-svn.sh:prepare_a_utf8_locale to prepare_utf8_locale, since we no longer prepare the variable named "a_utf8_locale", but set up a fallback value for GIT_TEST_UTF8_LOCALE instead. The fallback will be LC_ALL, LANG environment variable, or the first UTF-8 locale from output of "locale -a", in that order. Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-16Merge branch 'mt/parallel-checkout-part-3'Libravatar Junio C Hamano1-0/+1
The final part of "parallel checkout". * mt/parallel-checkout-part-3: ci: run test round with parallel-checkout enabled parallel-checkout: add tests related to .gitattributes t0028: extract encoding helpers to lib-encoding.sh parallel-checkout: add tests related to path collisions parallel-checkout: add tests for basic operations checkout-index: add parallel checkout support builtin/checkout.c: complete parallel checkout support make_transient_cache_entry(): optionally alloc from mem_pool
2021-05-05ci: run test round with parallel-checkout enabledLibravatar Matheus Tavares1-0/+1
We already have tests for the basic parallel-checkout operations. But this code can also run be executed by other commands, such as git-read-tree and git-sparse-checkout, which are currently not tested with multiple workers. To promote a wider test coverage without duplicating tests: 1. Add the GIT_TEST_CHECKOUT_WORKERS environment variable, to optionally force parallel-checkout execution during the whole test suite. 2. Set this variable (with a value of 2) in the second test round of our linux-gcc CI job. This round runs `make test` again with some optional GIT_TEST_* variables enabled, so there is no additional overhead in exercising the parallel-checkout code here. Note that tests checking out less than two parallel-eligible entries will fall back to the sequential mode. Nevertheless, it's still a good exercise for the parallel-checkout framework as the fallback codepath also writes the queued entries using the parallel-checkout functions (only without spawning any worker). Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-16Merge branch 'en/ort-readiness'Libravatar Junio C Hamano1-0/+1
Plug the ort merge backend throughout the rest of the system, and start testing it as a replacement for the recursive backend. * en/ort-readiness: Add testing with merge-ort merge strategy t6423: mark remaining expected failure under merge-ort as such Revert "merge-ort: ignore the directory rename split conflict for now" merge-recursive: add a bunch of FIXME comments documenting known bugs merge-ort: write $GIT_DIR/AUTO_MERGE whenever we hit a conflict t: mark several submodule merging tests as fixed under merge-ort merge-ort: implement CE_SKIP_WORKTREE handling with conflicted entries t6428: new test for SKIP_WORKTREE handling and conflicts merge-ort: support subtree shifting merge-ort: let renormalization change modify/delete into clean delete merge-ort: have ll_merge() use a special attr_index for renormalization merge-ort: add a special minimal index just for renormalization merge-ort: use STABLE_QSORT instead of QSORT where required
2021-03-20Add testing with merge-ort merge strategyLibravatar Elijah Newren1-0/+1
In preparation for switching from merge-recursive to merge-ort as the default strategy, have the testsuite default to running with merge-ort. Keep coverage of the recursive backend by having the linux-gcc job run with it. Signed-off-by: Elijah Newren <newren@gmail.com> Reviewed-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-02-12Merge branch 'tb/pack-revindex-on-disk'Libravatar Junio C Hamano1-0/+1
Introduce an on-disk file to record revindex for packdata, which traditionally was always created on the fly and only in-core. * tb/pack-revindex-on-disk: t5325: check both on-disk and in-memory reverse index pack-revindex: ensure that on-disk reverse indexes are given precedence t: support GIT_TEST_WRITE_REV_INDEX t: prepare for GIT_TEST_WRITE_REV_INDEX Documentation/config/pack.txt: advertise 'pack.writeReverseIndex' builtin/pack-objects.c: respect 'pack.writeReverseIndex' builtin/index-pack.c: write reverse indexes builtin/index-pack.c: allow stripping arbitrary extensions pack-write.c: prepare to write 'pack-*.rev' files packfile: prepare for the existence of '*.rev' files
2021-02-10Merge branch 'ab/detox-gettext-tests'Libravatar Junio C Hamano2-3/+2
Get rid of "GETTEXT_POISON" support altogether, which may or may not be controversial. * ab/detox-gettext-tests: tests: remove uses of GIT_TEST_GETTEXT_POISON=false tests: remove support for GIT_TEST_GETTEXT_POISON ci: remove GETTEXT_POISON jobs
2021-01-25t: support GIT_TEST_WRITE_REV_INDEXLibravatar Taylor Blau1-0/+1
Add a new option that unconditionally enables the pack.writeReverseIndex setting in order to run the whole test suite in a mode that generates on-disk reverse indexes. Additionally, enable this mode in the second run of tests under linux-gcc in 'ci/run-build-and-tests.sh'. Once on-disk reverse indexes are proven out over several releases, we can change the default value of that configuration to 'true', and drop this patch. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-01-25Merge branch 'js/default-branch-name-tests-final-stretch'Libravatar Junio C Hamano1-0/+2
Prepare tests not to be affected by the name of the default branch "git init" creates. * js/default-branch-name-tests-final-stretch: (28 commits) tests: drop prereq `PREPARE_FOR_MAIN_BRANCH` where no longer needed t99*: adjust the references to the default branch name "main" tests(git-p4): transition to the default branch name `main` t9[5-7]*: adjust the references to the default branch name "main" t9[0-4]*: adjust the references to the default branch name "main" t8*: adjust the references to the default branch name "main" t7[5-9]*: adjust the references to the default branch name "main" t7[0-4]*: adjust the references to the default branch name "main" t6[4-9]*: adjust the references to the default branch name "main" t64*: preemptively adjust alignment to prepare for `master` -> `main` t6[0-3]*: adjust the references to the default branch name "main" t5[6-9]*: adjust the references to the default branch name "main" t55[4-9]*: adjust the references to the default branch name "main" t55[23]*: adjust the references to the default branch name "main" t551*: adjust the references to the default branch name "main" t550*: adjust the references to the default branch name "main" t5503: prepare aligned comment for replacing `master` with `main` t5[0-4]*: adjust the references to the default branch name "main" t5323: prepare centered comment for `master` -> `main` t4*: adjust the references to the default branch name "main" ...
2021-01-21ci: remove GETTEXT_POISON jobsLibravatar Ævar Arnfjörð Bjarmason2-3/+2
A subsequent commit will remove GETTEXT_POISON entirely, let's start by removing the CI jobs that enable the option. We cannot just remove the job because the CI is implicitly depending on the "poison" job being a sort of "default" job in the sense that it's the job that was otherwise run with the default compiler, no other GIT_TEST_* options etc. So let's keep it under the name "linux-gcc-default". This means we can remove the initial "make test" from the "linux-gcc" job (it does another one after setting a bunch of GIT_TEST_* variables). I'm not doing that because it would conflict with the in-flight 334afbc76fb (tests: mark tests relying on the current default for `init.defaultBranch`, 2020-11-18) (currently on the "seen" branch, so the SHA-1 will almost definitely change). It's going to use that "make test" again for different reasons, so let's preserve it for now. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-01-14ci/install-depends: attempt to fix "brew cask" stuffLibravatar Junio C Hamano1-4/+4
We run "git pull" against "$cask_repo"; clarify that we are expecting not to have any of our own modifications and running "git pull" to merely update, by passing "--ff-only" on the command line. Also, the "brew cask install" command line triggers an error message that says: Error: Calling brew cask install is disabled! Use brew install [--cask] instead. In addition, "brew install caskroom/cask/perforce" step triggers an error that says: Error: caskroom/cask was moved. Tap homebrew/cask instead. Attempt to see if blindly following the suggestion in these error messages gets us into a better shape. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-11-19tests: mark tests relying on the current default for `init.defaultBranch`Libravatar Johannes Schindelin1-0/+2
In addition to the manual adjustment to let the `linux-gcc` CI job run the test suite with `master` and then with `main`, this patch makes sure that GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME is set in all test scripts that currently rely on the initial branch name being `master by default. To determine which test scripts to mark up, the first step was to force-set the default branch name to `master` in - all test scripts that contain the keyword `master`, - t4211, which expects `t/t4211/history.export` with a hard-coded ref to initialize the default branch, - t5560 because it sources `t/t556x_common` which uses `master`, - t8002 and t8012 because both source `t/annotate-tests.sh` which also uses `master`) This trick was performed by this command: $ sed -i '/^ *\. \.\/\(test-lib\|lib-\(bash\|cvs\|git-svn\)\|gitweb-lib\)\.sh$/i\ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\ ' $(git grep -l master t/t[0-9]*.sh) \ t/t4211*.sh t/t5560*.sh t/t8002*.sh t/t8012*.sh After that, careful, manual inspection revealed that some of the test scripts containing the needle `master` do not actually rely on a specific default branch name: either they mention `master` only in a comment, or they initialize that branch specificially, or they do not actually refer to the current default branch. Therefore, the aforementioned modification was undone in those test scripts thusly: $ git checkout HEAD -- \ t/t0027-auto-crlf.sh t/t0060-path-utils.sh \ t/t1011-read-tree-sparse-checkout.sh \ t/t1305-config-include.sh t/t1309-early-config.sh \ t/t1402-check-ref-format.sh t/t1450-fsck.sh \ t/t2024-checkout-dwim.sh \ t/t2106-update-index-assume-unchanged.sh \ t/t3040-subprojects-basic.sh t/t3301-notes.sh \ t/t3308-notes-merge.sh t/t3423-rebase-reword.sh \ t/t3436-rebase-more-options.sh \ t/t4015-diff-whitespace.sh t/t4257-am-interactive.sh \ t/t5323-pack-redundant.sh t/t5401-update-hooks.sh \ t/t5511-refspec.sh t/t5526-fetch-submodules.sh \ t/t5529-push-errors.sh t/t5530-upload-pack-error.sh \ t/t5548-push-porcelain.sh \ t/t5552-skipping-fetch-negotiator.sh \ t/t5572-pull-submodule.sh t/t5608-clone-2gb.sh \ t/t5614-clone-submodules-shallow.sh \ t/t7508-status.sh t/t7606-merge-custom.sh \ t/t9302-fast-import-unpack-limit.sh We excluded one set of test scripts in these commands, though: the range of `git p4` tests. The reason? `git p4` stores the (foreign) remote branch in the branch called `p4/master`, which is obviously not the default branch. Manual analysis revealed that only five of these tests actually require a specific default branch name to pass; They were modified thusly: $ sed -i '/^ *\. \.\/lib-git-p4\.sh$/i\ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\ ' t/t980[0167]*.sh t/t9811*.sh Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-11-17ci: avoid `set-env` construct in print-test-failures.shLibravatar Junio C Hamano1-1/+1
Imitating cac42e47 (ci: avoid using the deprecated `set-env` construct, 2020-11-07), avoid deprecated ::set-env and use the recommended alternative instead in print-test-failures.sh Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-10-08ci: do not skip tagged revisions in GitHub workflowsLibravatar Johannes Schindelin1-0/+2
When `master` is tagged, and then both `master` and the tag are pushed, Travis CI will happily build both. That is a waste of energy, which is why we skip the build for `master` in that case. Our GitHub workflow is also triggered by tags. However, the run would fail because the `windows-test` jobs are _not_ skipped on tags, but the `windows-build` job _is skipped (and therefore fails to upload the build artifacts needed by the test jobs). In addition, we just added logic to our GitHub workflow that will skip runs altogether if there is already a successful run for the same commit or at least for the same tree. Let's just change the GitHub workflow to no longer specifically skip tagged revisions. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-21ci: stop linking built-ins to the dashed versionsLibravatar Johannes Schindelin1-0/+1
Since e4597aae6590 (run test suite without dashed git-commands in PATH, 2009-12-02), we stopped running our tests with `git-foo` binaries found at the top-level directory of a freshly built source tree; instead we have placed only `git` and selected `git-foo` commands that must be on `$PATH` in `bin-wrappers/` and prepended that `bin-wrappers/` to the `PATH` used in the test suite. We did that to catch the tests and scripted Git commands that still try to use the dashed form. Since CI jobs will not install the built Git to anywhere, and the hardlinks we make at the top-level of the source tree for `git-add` and friends are not even used during tests, they are pure waste of resources these days. Thanks to the newly invented `SKIP_DASHED_BUILT_INS` knob, we can now skip creating these links in the source tree. So let's do that. Note that this change introduces a subtle change of behavior: when Git's `cmd_main()` calls `setup_path()`, it inserts the value of `GIT_EXEC_PATH` (defaulting to `<prefix>/libexec/git-core`) at the beginning of the environment variable `PATH`. This is necessary to find e.g. scripted commands that are installed in that location. For the purposes of Git's test suite, the `bin-wrappers/` scripts override `GIT_EXEC_PATH` to point to the top-level directory of the source code. In other words, if a scripted command had used a dashed invocation of a built-in Git command, it would not have been caught previously, which is fixed by this change. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-08-11Merge branch 'bc/sha-256-part-3'Libravatar Junio C Hamano1-0/+6
The final leg of SHA-256 transition. * bc/sha-256-part-3: (39 commits) t: remove test_oid_init in tests docs: add documentation for extensions.objectFormat ci: run tests with SHA-256 t: make SHA1 prerequisite depend on default hash t: allow testing different hash algorithms via environment t: add test_oid option to select hash algorithm repository: enable SHA-256 support by default setup: add support for reading extensions.objectformat bundle: add new version for use with SHA-256 builtin/verify-pack: implement an --object-format option http-fetch: set up git directory before parsing pack hashes t0410: mark test with SHA1 prerequisite t5308: make test work with SHA-256 t9700: make hash size independent t9500: ensure that algorithm info is preserved in config t9350: make hash size independent t9301: make hash size independent t9300: use $ZERO_OID instead of hard-coded object ID t9300: abstract away SHA-1-specific constants t8011: make hash size independent ...
2020-07-30ci: run tests with SHA-256Libravatar brian m. carlson1-0/+6
Now that we have Git supporting SHA-256, we'd like to make sure that we don't regress that state. Unfortunately, it's easy to do so, so to help, let's add code to run one of our CI jobs with SHA-256 as the default hash. This will help us detect any problems that may occur. We pick the linux-clang job because it's relatively fast and the linux-gcc job already runs the testsuite twice. We want our tests to run as fast as possible, so we wouldn't want to add a third run to the linux-gcc job. To make sure we properly exercise the code, let's run the tests in the default mode (SHA-1) first and then run a second time with SHA-256. We explicitly specify SHA-1 for the first run so that if we change the default in the future, we make sure to test both cases. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Reviewed-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-23ci: use absolute PYTHON_PATH in the Linux jobsLibravatar SZEDER Gábor1-2/+2
In our test suite, when 'git p4' invokes a Git command as a subprocesses, then it should run the 'git' binary we are testing. Unfortunately, this is not the case in the 'linux-clang' and 'linux-gcc' jobs on Travis CI, where 'git p4' runs the system '/usr/bin/git' instead. Travis CI's default Linux image includes 'pyenv', and all Python invocations that involve PATH lookup go through 'pyenv', e.g. our 'PYTHON_PATH=$(which python3)' sets '/opt/pyenv/shims/python3' as PYTHON_PATH, which in turn will invoke '/usr/bin/python3'. Alas, the 'pyenv' version included in this image is buggy, and prepends the directory containing the Python binary to PATH even if that is a system directory already in PATH near the end. Consequently, 'git p4' in those jobs ends up with its PATH starting with '/usr/bin', and then runs '/usr/bin/git'. So use the absolute paths '/usr/bin/python{2,3}' explicitly when setting PYTHON_PATH in those Linux jobs to avoid the PATH lookup and thus the bogus 'pyenv' from interfering with our 'git p4' tests. Don't bother with special-casing Travis CI: while this issue doesn't affect the corresponding Linux jobs on GitHub Actions, both CI systems use Ubuntu LTS-based images, so we can safely rely on these Python paths. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-05-29Merge branch 'jk/ci-only-on-selected-branches'Libravatar Junio C Hamano1-2/+3
Dev support. * jk/ci-only-on-selected-branches: ci/config: correct instruction for CI preferences
2020-05-18ci/config: correct instruction for CI preferencesLibravatar Đoàn Trần Công Danh1-2/+3
From e76eec3554 (ci: allow per-branch config for GitHub Actions, 2020-05-07), we started to allow contributors decide which branch they want to build with GitHub Actions by checking for a file named "ci/config/allow-ref". In order to assist those contributors, we provided a sample in "ci/config/allow-refs.sample", and instructed them to drop the ".sample", then commit that file to their repository. We've misspelt the filename in that change. Let's fix the spelling. While we're at it, also instruct our contributors introduce that new file to Git before commit, in case of they've never told Git before. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-05-15Revert "ci: add a problem matcher for GitHub Actions"Libravatar Junio C Hamano2-21/+0
This reverts commit 676eb0c1ce0d380478eb16bdc5a3f2a7bc01c1d2; as we will be reverting the change to show these extra output tokens under bash, the pattern would not match anything. Helped-by: Carlo Marcelo Arenas Belón <carenas@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-05-07ci: allow per-branch config for GitHub ActionsLibravatar Jeff King1-0/+26
Depending on the workflows of individual developers, it can either be convenient or annoying that our GitHub Actions CI jobs are run on every branch. As an example of annoying: if you carry many half-finished work-in-progress branches and rebase them frequently against master, you'd get tons of failure reports that aren't interesting (not to mention the wasted CPU). This commit adds a new job which checks a special branch within the repository for CI config, and then runs a shell script it finds there to decide whether to skip the rest of the tests. The default will continue to run tests for all refs if that branch or script is missing. There have been a few alternatives discussed: One option is to carry information in the commit itself about whether it should be tested, either in the tree itself (changing the workflow YAML file) or in the commit message (a "[skip ci]" flag or similar). But these are frustrating and error-prone to use: - you have to manually apply them to each branch that you want to mark - it's easy for them to leak into other workflows, like emailing patches We could likewise try to get some information from the branch name. But that leads to debates about whether the default should be "off" or "on", and overriding still ends up somewhat awkward. If we default to "on", you have to remember to name your branches appropriately to skip CI. And if "off", you end up having to contort your branch names or duplicate your pushes with an extra refspec. By comparison, this commit's solution lets you specify your config once and forget about it, and all of the data is off in its own ref, where it can be changed by individual forks without touching the main tree. There were a few design decisions that came out of on-list discussion. I'll summarize here: - we could use GitHub's API to retrieve the config ref, rather than a real checkout (and then just operate on it via some javascript). We still have to spin up a VM and contact GitHub over the network from it either way, so it ends up not being much faster. I opted to go with shell to keep things similar to our other tools (and really could implement allow-refs in any language you want). This also makes it easy to test your script locally, and to modify it within the context of a normal git.git tree. - we could keep the well-known refname out of refs/heads/ to avoid cluttering the branch namespace. But that makes it awkward to manipulate. By contrast, you can just "git checkout ci-config" to make changes. - we could assume the ci-config ref has nothing in it except config (i.e., a branch unrelated to the rest of git.git). But dealing with orphan branches is awkward. Instead, we'll do our best to efficiently check out only the ci/config directory using a shallow partial clone, which allows your ci-config branch to be just a normal branch, with your config changes on top. - we could provide a simpler interface, like a static list of ref patterns. But we can't get out of spinning up a whole VM anyway, so we might as well use that feature to make the config as flexible as possible. If we add more config, we should be able to reuse our partial-clone to set more outputs. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-05-01Merge branch 'gs/commit-graph-path-filter'Libravatar Junio C Hamano1-0/+1
Introduce an extension to the commit-graph to make it efficient to check for the paths that were modified at each commit using Bloom filters. * gs/commit-graph-path-filter: bloom: ignore renames when computing changed paths commit-graph: add GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS test flag t4216: add end to end tests for git log with Bloom filters revision.c: add trace2 stats around Bloom filter usage revision.c: use Bloom filters to speed up path based revision walks commit-graph: add --changed-paths option to write subcommand commit-graph: reuse existing Bloom filters during write commit-graph: write Bloom filters to commit graph file commit-graph: examine commits by generation number commit-graph: examine changed-path objects in pack order commit-graph: compute Bloom filters for changed paths diff: halt tree-diff early after max_changes bloom.c: core Bloom filter implementation for changed paths. bloom.c: introduce core Bloom filter constructs bloom.c: add the murmur3 hash implementation commit-graph: define and use MAX_NUM_CHUNKS
2020-04-10ci: let GitHub Actions upload failed tests' directoriesLibravatar Johannes Schindelin1-0/+7
Arguably, CI builds' most important task is to not only identify regressions, but to make it as easy as possible to investigate what went wrong. In that light, we will want to provide users with a way to inspect the tests' output as well as the corresponding directories. This commit adds build steps that are only executed when tests failed, uploading the relevant information as build artifacts. These artifacts can then be downloaded by interested parties to diagnose the failures more efficiently. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-10ci: add a problem matcher for GitHub ActionsLibravatar Johannes Schindelin2-0/+21
With this patch, test failures will be annotated with a helpful, clickable message in GitHub Actions. For details, see https://github.com/actions/toolkit/blob/master/docs/problem-matchers.md Note: we need to set `TEST_SHELL_PATH` to Bash so that the problem matcher is fed a file and line number for each test failure. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-10ci: run gem with sudo to install asciidoctorLibravatar Johannes Schindelin1-1/+1
In a later patch, we will run Documentation job in GitHub Actions. The job will run without elevated permission. Run `gem` with `sudo` to elevate permission in order to be able to install to system location. This will also keep this installation in-line with other installation in our Linux system for CI. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> [Danh: reword commit message] Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-10ci: explicit install all required packagesLibravatar Đoàn Trần Công Danh1-3/+11
In a later patch, we will support GitHub Action. Explicitly install all of our build dependencies on Linux. Since GitHub Action's Linux VM hasn't installed our build dependencies. And there're no harm to reinstall them (in Travis) Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-07ci: fix the `jobname` of the `GETTEXT_POISON` jobLibravatar Johannes Schindelin1-1/+1
In 6cdccfce1e0f (i18n: make GETTEXT_POISON a runtime option, 2018-11-08), the `jobname` was adjusted to have the `GIT_TEST_` prefix, but that prefix makes no sense in this context. Co-authored-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-07ci/lib: set TERM environment variable if not existLibravatar Đoàn Trần Công Danh1-0/+3
GitHub Action doesn't set TERM environment variable, which is required by "tput". Fallback to dumb if it's not set. Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-07ci/lib: allow running in GitHub ActionsLibravatar Johannes Schindelin1-1/+19
For each CI system we support, we need a specific arm in that if/else construct in ci/lib.sh. Let's add one for GitHub Actions. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-07ci/lib: if CI type is unknown, show the environment variablesLibravatar Johannes Schindelin1-0/+1
This should help with adding new CI-specific if-else arms. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-07Merge branch 'dd/ci-musl-libc' into HEADLibravatar Junio C Hamano5-43/+100
* dd/ci-musl-libc: travis: build and test on Linux with musl libc and busybox ci/linux32: libify install-dependencies step ci: refactor docker runner script ci/linux32: parameterise command to switch arch ci/lib-docker: preserve required environment variables ci: make MAKEFLAGS available inside the Docker container in the Linux32 job
2020-04-06travis: build and test on Linux with musl libc and busyboxLibravatar Đoàn Trần Công Danh4-0/+16
Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-06ci/linux32: libify install-dependencies stepLibravatar Đoàn Trần Công Danh2-6/+15
In a later patch, we will add new Travis Job for linux-musl. Most of other code in this file could be reuse for that job. Move the code to install dependencies to a common script. Should we add new CI system that can run directly in container, we can reuse this script for installation step. Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-06ci: refactor docker runner scriptLibravatar Đoàn Trần Công Danh2-9/+19
We will support alpine check in docker later in this series. While we're at it, tell people to run as root in podman, if podman is used as drop-in replacement for docker, because podman will map host-user to container's root, therefore, mapping their permission. Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-06ci/linux32: parameterise command to switch archLibravatar Đoàn Trần Công Danh2-2/+13
In a later patch, the remaining of this command will be re-used for the CI job for linux with musl libc. Allow customisation of the emulator, now. Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-06ci/lib-docker: preserve required environment variablesLibravatar Đoàn Trần Công Danh1-3/+10
We're using "su -m" to preserve environment variables in the shell run by "su". But, that options will be ignored while "-l" (aka "--login") is specified in util-linux and busybox's su. In a later patch this script will be reused for checking Git for Linux with musl libc on Alpine Linux, Alpine Linux uses "su" from busybox. Since we don't have interest in all environment variables, pass only those necessary variables to the inner script. Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-06commit-graph: add GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS test flagLibravatar Garima Singh1-0/+1
Add GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS test flag to the test setup suite in order to toggle writing Bloom filters when running any of the git tests. If set to true, we will compute and write Bloom filters every time a test calls `git commit-graph write`, as if the `--changed-paths` option was passed in. The test suite passes when GIT_TEST_COMMIT_GRAPH and GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS are enabled. Helped-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Garima Singh <garima.singh@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-02ci: make MAKEFLAGS available inside the Docker container in the Linux32 jobLibravatar SZEDER Gábor2-0/+4
Once upon a time we ran 'make --jobs=2 ...' to build Git, its documentation, or to apply Coccinelle semantic patches. Then commit eaa62291ff (ci: inherit --jobs via MAKEFLAGS in run-build-and-tests, 2019-01-27) came along, and started using the MAKEFLAGS environment variable to centralize setting the number of parallel jobs in 'ci/libs.sh'. Alas, it forgot to update 'ci/run-linux32-docker.sh' to make MAKEFLAGS available inside the Docker container running the 32 bit Linux job, and, consequently, since then that job builds Git sequentially, and it ignores any Makefile knobs that we might set in MAKEFLAGS (though we don't set any for the 32 bit Linux job at the moment). So update the 'docker run' invocation in 'ci/run-linux32-docker.sh' to make MAKEFLAGS available inside the Docker container as well. Set CC=gcc for the 32 bit Linux job, because that's the compiler installed in the 32 bit Linux Docker image that we use (Travis CI nowadays sets CC=clang by default, but clang is not installed in this image). Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-03-25Merge branch 'yz/p4-py3'Libravatar Junio C Hamano1-0/+6
Update "git p4" to work with Python 3. * yz/p4-py3: ci: use python3 in linux-gcc and osx-gcc and python2 elsewhere git-p4: use python3's input() everywhere git-p4: simplify regex pattern generation for parsing diff-tree git-p4: use dict.items() iteration for python3 compatibility git-p4: use functools.reduce instead of reduce git-p4: fix freezing while waiting for fast-import progress git-p4: use marshal format version 2 when sending to p4 git-p4: open .gitp4-usercache.txt in text mode git-p4: convert path to unicode before processing them git-p4: encode/decode communication with git for python3 git-p4: encode/decode communication with p4 for python3 git-p4: remove string type aliasing git-p4: change the expansion test from basestring to list git-p4: make python2.7 the oldest supported version
2020-03-10ci: use python3 in linux-gcc and osx-gcc and python2 elsewhereLibravatar SZEDER Gábor1-0/+6
Python2 reached end of life, and we have been preparing our Python scripts to work with Python3. 'git p4', the main in-tree user of Python, has just received a number of compatibility updates. Our other notable Python script 'contrib/svn-fe/svnrdump_sim.py' is only used in 't9020-remote-svn.sh', and is apparently already compatible with both Python2 and 3. Our CI jobs currently only use Python2. We want to make sure that these Python scripts do indeed work with Python3, and we also want to make sure that these scripts keep working with Python2 as well, for the sake of some older LTS/Enterprise setups. Therefore, pick two jobs and use Python3 there, while leaving other jobs to still stick to Python2 for now. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-03-05Merge branch 'js/ci-windows-update'Libravatar Junio C Hamano1-2/+2
Updates to the CI settings. * js/ci-windows-update: Azure Pipeline: switch to the latest agent pools ci: prevent `perforce` from being quarantined t/lib-httpd: avoid using macOS' sed