summaryrefslogtreecommitdiff
path: root/ci
AgeCommit message (Collapse)AuthorFilesLines
2020-03-05Merge branch 'js/ci-windows-update'Libravatar Junio C Hamano1-2/+2
Updates to the CI settings. * js/ci-windows-update: Azure Pipeline: switch to the latest agent pools ci: prevent `perforce` from being quarantined t/lib-httpd: avoid using macOS' sed
2020-02-27ci: prevent `perforce` from being quarantinedLibravatar Johannes Schindelin1-2/+2
The most recent Azure Pipelines macOS agents enable what Apple calls "System Integrity Protection". This makes `p4d -V` hang: there is some sort of GUI dialog waiting for the user to acknowledge that the copied binaries are legit and may be executed, but on build agents, there is no user who could acknowledge that. Let's ask Homebrew specifically to _not_ quarantine the Perforce binaries. Helped-by: Aleksandr Chebotov Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-12Merge branch 'js/ci-squelch-doc-warning'Libravatar Junio C Hamano1-0/+1
Squelch unhelpful warning message during documentation build. * js/ci-squelch-doc-warning: ci: ignore rubygems warning in the "Documentation" job
2020-02-10ci: ignore rubygems warning in the "Documentation" jobLibravatar Johannes Schindelin1-0/+1
A recent update in the Linux VM images used by Azure Pipelines surfaced a new problem in the "Documentation" job. Apparently, this warning appears 396 times on `stderr` when running `make doc`: /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated This problem was already reported to the `rubygems` project via https://github.com/rubygems/rubygems/issues/3068. As there is nothing Git can do about this warning, and as the "Documentation" job reports this warning as a failure, let's just silence it and move on. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Acked-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-05Merge branch 'js/add-p-leftover-bits'Libravatar Junio C Hamano1-0/+1
The final leg of rewriting "add -i/-p" in C. * js/add-p-leftover-bits: ci: include the built-in `git add -i` in the `linux-gcc` job built-in add -p: handle Escape sequences more efficiently built-in add -p: handle Escape sequences in interactive.singlekey mode built-in add -p: respect the `interactive.singlekey` config setting terminal: add a new function to read a single keystroke terminal: accommodate Git for Windows' default terminal terminal: make the code of disable_echo() reusable built-in add -p: handle diff.algorithm built-in add -p: support interactive.diffFilter t3701: adjust difffilter test
2020-01-15ci: include the built-in `git add -i` in the `linux-gcc` jobLibravatar Johannes Schindelin1-0/+1
This job runs the test suite twice, once in regular mode, and once with a whole slew of `GIT_TEST_*` variables set. Now that the built-in version of `git add --interactive` is feature-complete, let's also throw `GIT_TEST_ADD_I_USE_BUILTIN` into that fray. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-06Merge branch 'sg/osx-force-gcc-9'Libravatar Junio C Hamano2-4/+5
TravisCI update. * sg/osx-force-gcc-9: ci: build Git with GCC 9 in the 'osx-gcc' build job
2019-12-05Merge branch 'sg/test-bool-env'Libravatar Junio C Hamano1-1/+1
Recently we have declared that GIT_TEST_* variables take the usual boolean values (it used to be that some used "non-empty means true" and taking GIT_TEST_VAR=YesPlease as true); make sure we notice and fail when non-bool strings are given to these variables. * sg/test-bool-env: t5608-clone-2gb.sh: turn GIT_TEST_CLONE_2GB into a bool tests: add 'test_bool_env' to catch non-bool GIT_TEST_* values
2019-12-05Merge branch 'jc/azure-ci-osx-fix-fix'Libravatar Junio C Hamano1-1/+2
CI jobs for macOS has been made less chatty when updating perforce package used during testing. * jc/azure-ci-osx-fix-fix: ci(osx): update homebrew-cask repository with less noise
2019-11-29ci: build Git with GCC 9 in the 'osx-gcc' build jobLibravatar SZEDER Gábor2-3/+5
Our 'osx-gcc' build job on Travis CI relied on GCC 8 being installed (but not linked) in the image we use [1]. Alas, since the last update of this image a few days ago this is not the case anymore, and now it contains GCC 9 (installed and linked) instead of GCC 8. The results are failed 'osx-gcc' jobs, because they can't find the 'gcc-8' command [2]. Let's move on to use GCC 9, with hopefully better error reporting and improved -Wfoo flags and what not. On Travis CI this has the benefit that we can spare a few seconds while installing dependencies, because it already comes pre-installed, at least for now. The Azure Pipelines OSX image doesn't include GCC, so we have to install it ourselves anyway, and then we might as well install the newer version. In a vain attempt I tried to future-proof this a bit: - Install 'gcc@9' specifically, so we'll still get what we want even after GCC 10 comes out, and the "plain" 'gcc' package starts to refer to 'gcc@10'. - Run both 'brew install gcc@9' and 'brew link gcc@9'. If 'gcc@9' is already installed and linked, then both commands are noop and exit with success. But as we saw in the past, sometimes the image contains the expected GCC package installed but not linked, so maybe it will happen again in the future as well. In that case 'brew install' is still a noop, and instructs the user to run 'brew link' instead, so that's what we'll do. And if 'gcc@9' is not installed, then 'brew install' will install it, and the subsequent 'brew link' becomes a noop. An additional benefit of this patch is that from now on we won't unnecessarily install GCC and its dependencies in the 'osx-clang' jobs on Azure Pipelines. [1] 7d4733c501 (ci: fix GCC install in the Travis CI GCC OSX job, 2019-10-24) [2] https://travis-ci.org/git/git/jobs/615442297#L333 Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-23t5608-clone-2gb.sh: turn GIT_TEST_CLONE_2GB into a boolLibravatar SZEDER Gábor1-1/+1
The GIT_TEST_CLONE_2GB environment variable is only ever checked with 'test -z' in 't5608-clone-2gb.sh', so any non-empty value is interpreted as "yes, run these expensive tests", even 'GIT_TEST_CLONE_2GB=NoThanks'. Similar GIT_TEST_* environment variables have already been turned into bools in 3b072c577b (tests: replace test_tristate with "git env--helper", 2019-06-21), so let's turn GIT_TEST_CLONE_2GB into a bool as well, to follow suit. Our CI builds set GIT_TEST_CLONE_2GB=YesPlease, so adjust them accordingly, thus removing the last 'YesPlease' from our CI scripts. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-20ci(osx): update homebrew-cask repository with less noiseLibravatar Junio C Hamano1-1/+2
The OSX CI build procedure updates the homebrew-cask repository before attempting to install perforce again, after seeing an installation failure. This involves a "git pull" that by default computes and outputs diffstat, which would only grow as the time goes by and the repository cast in stone in the CI build image becomes more and more stale relative to the upstream repository in the outside world. Suppress the diffstat to both save cycles to generate it, and strain on the eyeballs to skip it. Reported-by: SZEDER Gábor <szeder.dev@gmail.com> Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-24Merge branch 'sg/ci-osx-gcc8-fix'Libravatar Junio C Hamano1-0/+1
CI build fix. * sg/ci-osx-gcc8-fix: ci: fix GCC install in the Travis CI GCC OSX job
2019-10-24ci: fix GCC install in the Travis CI GCC OSX jobLibravatar SZEDER Gábor1-0/+1
A few days ago Travis CI updated their existing OSX images, including the Homebrew database in the xcode10.1 OSX image that we use. Since then installing dependencies in the 'osx-gcc' job fails when it tries to link gcc@8: + brew link gcc@8 Error: No such keg: /usr/local/Cellar/gcc@8 GCC8 is still installed but not linked to '/usr/local' in the updated image, as it was before this update, but now we have to link it by running 'brew link gcc'. So let's do that then, and fall back to linking gcc@8 if it doesn't, just to be sure. Our builds on Azure Pipelines are unaffected by this issue. The OSX image over there doesn't contain the gcc@8 package, so we have to 'brew install' it, which already takes care of linking it to '/usr/local'. After that the 'brew link gcc' command added by this patch fails, but the ||-chained fallback 'brew link gcc@8' command succeeds with an "already linked" warning. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-23ci(osx): use new location of the `perforce` caskLibravatar Johannes Schindelin1-0/+5
The Azure Pipelines builds are failing for macOS due to a change in the location of the perforce cask. The command outputs the following error: + brew install caskroom/cask/perforce Error: caskroom/cask was moved. Tap homebrew/cask-cask instead. So let's try to call `brew cask install perforce` first (which is what that error message suggests, in a most round-about way). Prior to 672f51cb we used to install the 'perforce' package with 'brew install perforce' (note: no 'cask' in there). The justification for 672f51cb was that the command 'brew install perforce' simply stopped working, after Homebrew folks decided that it's better to move the 'perforce' package to a "cask". Their justification for this move was that 'brew install perforce' "can fail due to a checksum mismatch ...", and casks can be installed without checksum verification. And indeed, both 'brew cask install perforce' and 'brew install caskroom/cask/perforce' printed something along the lines of: ==> No checksum defined for Cask perforce, skipping verification It is unclear why 672f51cb used 'brew install caskroom/cask/perforce' instead of 'brew cask install perforce'. It appears (by running both commands on old Travis CI macOS images) that both commands worked all the same already back then. In any case, as the error message at the top of this commit message shows, 'brew install caskroom/cask/perforce' has stopped working recently, but 'brew cask install perforce' still does, so let's use that. CI servers are typically fresh virtual machines, but not always. To accommodate for that, let's try harder if `brew cask install perforce` fails, by specifically pulling the latest `master` of the `homebrew-cask` repository. This will still fail, of course, when `homebrew-cask` falls behind Perforce's release schedule. But once it is updated, we can now simply re-run the failed jobs and they will pick up that update. As for updating `homebrew-cask`: the beginnings of automating this in https://dev.azure.com/gitgitgadget/git/_build?definitionId=11&_a=summary will be finished once the next Perforce upgrade comes around. Helped-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-07Merge branch 'dl/honor-cflags-in-hdr-check'Libravatar Junio C Hamano2-1/+5
Dev support. * dl/honor-cflags-in-hdr-check: ci: run `hdr-check` as part of the `Static Analysis` job Makefile: emulate compile in $(HCO) target better pack-bitmap.h: remove magic number promisor-remote.h: include missing header apply.h: include missing header
2019-10-07Merge branch 'sg/travis-help-debug'Libravatar Junio C Hamano1-0/+5
Dev support update. * sg/travis-help-debug: travis-ci: do not skip successfully tested trees in debug mode
2019-10-06Merge branch 'bc/doc-use-docbook-5'Libravatar Junio C Hamano2-1/+3
Start using DocBook 5 (instead of DocBook 4.5) as Asciidoctor 2.0 no longer works with the older one. * bc/doc-use-docbook-5: Documentation: fix build with Asciidoctor 2
2019-10-03ci: run `hdr-check` as part of the `Static Analysis` jobLibravatar Johannes Schindelin2-1/+5
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-09-28travis-ci: do not skip successfully tested trees in debug modeLibravatar SZEDER Gábor1-0/+5
Travis CI offers shell access to its virtual machine environment running the build jobs, called "debug mode" [1]. After restarting a build job in debug mode and logging in, the first thing I usually do is to install dependencies, i.e. run './ci/install-dependencies.sh'. This works just fine when I restarted a failed build job in debug mode. However, after restarting a successful build job in debug mode our CI scripts get all clever, and exit without doing anything useful, claiming that "This commit's tree has already been built and tested successfully" [2]. Our CI scripts are right, and we do want to skip building and testing already known good trees in "regular" CI builds. In debug mode, however, this is a nuisiance, because one has to delete the cache (or at least the 'good-trees' file in the cache) to proceed. Let's update our CI scripts, in particular the common 'ci/lib.sh', to not skip previously successfully built and tested trees in debug mode, so all those scripts will do what there were supposed to do even when a successful build job was restarted in debug mode. [1] https://docs.travis-ci.com/user/running-build-in-debug-mode/ [2] 9cc2c76f5e (travis-ci: record and skip successfully built trees, 2017-12-31) Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-09-16Documentation: fix build with Asciidoctor 2Libravatar brian m. carlson2-1/+3
Our documentation toolchain has traditionally been built around DocBook 4.5. This version of DocBook is the last DTD-based version of DocBook. In 2009, DocBook 5 was introduced using namespaces and its syntax is expressed in RELAX NG, which is more expressive and allows a wider variety of syntax forms. Asciidoctor, one of the alternatives for building our documentation, moved support for DocBook 4.5 out of core in its recent 2.0 release and now only supports DocBook 5 in the main release. The DocBoook 4.5 converter is still available as a separate component, but this is not available in most distro packages. This would not be a problem but for the fact that we use xmlto, which is still stuck in the DocBook 4.5 era. xmlto performs DTD validation as part of the build process. This is not problematic for DocBook 4.5, which has a valid DTD, but it clearly cannot work for DocBook 5, since no DTD can adequately express its full syntax. In addition, even if xmlto did support RELAX NG validation, that wouldn't be sufficient because it uses the libxml2-based xmllint to do so, which has known problems with validating interleaves in RELAX NG. Fortunately, there's an easy way forward: ask Asciidoctor to use its DocBook 5 backend and tell xmlto to skip validation. Asciidoctor has supported DocBook 5 since v0.1.4 in 2013 and xmlto has supported skipping validation for probably longer than that. We also need to teach xmlto how to use the namespaced DocBook XSLT stylesheets instead of the non-namespaced ones it usually uses. Normally these stylesheets are interchangeable, but the non-namespaced ones have a bug that causes them not to strip whitespace automatically from certain elements when namespaces are in use. This results in additional whitespace at the beginning of list elements, which is jarring and unsightly. We can do this by passing a custom stylesheet with the -x option that simply imports the namespaced stylesheets via a URL. Any system with support for XML catalogs will automatically look this URL up and reference a local copy instead without us having to know where this local copy is located. We know that anyone using xmlto will already have catalogs set up properly since the DocBook 4.5 DTD used during validation is also looked up via catalogs. All major Linux distributions distribute the necessary stylesheets and have built-in catalog support, and Homebrew does as well, albeit with a requirement to set an environment variable to enable catalog support. On the off chance that someone lacks support for catalogs, it is possible for xmlto (via xmllint) to download the stylesheets from the URLs in question, although this will likely perform poorly enough to attract attention. People still have the option of using the prebuilt documentation that we ship, so happily this should not be an impediment. Finally, we need to filter out some messages from other stylesheets that occur when invoking dblatex in the CI job. This tool strips namespaces much like the unnamespaced DocBook stylesheets and prints similar messages. If we permit these messages to be printed to standard error, our documentation CI job will fail because we check standard error for unexpected output. Due to dblatex's reliance on Python 2, we may need to revisit its use in the future, in which case this problem may go away, but this can be delayed until a future patch. The final message we filter is due to libxslt on modern Debian and Ubuntu. The patch which they use to implement reproducible ID generation also prints messages about the ID generation. While this doesn't affect our current CI images since they use Ubuntu 16.04 which lacks this patch, if we upgrade to Ubuntu 18.04 or a modern Debian, these messages will appear and, like the above messages, cause a CI failure. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-09-06ci: restore running httpd testsLibravatar SZEDER Gábor1-1/+1
Once upon a time GIT_TEST_HTTPD was a tristate variable and we exported 'GIT_TEST_HTTPD=YesPlease' in our CI scripts to make sure that we run the httpd tests in the Linux Clang and GCC build jobs, or error out if they can't be run for any reason [1]. Then 3b072c577b (tests: replace test_tristate with "git env--helper", 2019-06-21) came along, turned GIT_TEST_HTTPD into a bool, but forgot to update our CI scripts accordingly. So, since GIT_TEST_HTTPD is set explicitly, but its value is not one of the standardized true values, our CI jobs have been simply skipping the httpd tests in the last couple of weeks. Set 'GIT_TEST_HTTPD=true' to restore running httpd tests in our CI jobs. [1] a1157b76eb (travis-ci: set GIT_TEST_HTTPD in 'ci/lib-travisci.sh', 2017-12-12) Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-07-29Merge branch 'sg/travis-gcc-4.8'Libravatar Junio C Hamano1-4/+13
Add a job to build with a tad older GCC to make sure we are still buildable. * sg/travis-gcc-4.8: travis-ci: build with GCC 4.8 as well
2019-07-25Merge branch 'ab/test-env'Libravatar Junio C Hamano1-1/+1
Many GIT_TEST_* environment variables control various aspects of how our tests are run, but a few followed "non-empty is true, empty or unset is false" while others followed the usual "there are a few ways to spell true, like yes, on, etc., and also ways to spell false, like no, off, etc." convention. * ab/test-env: env--helper: mark a file-local symbol as static tests: make GIT_TEST_FAIL_PREREQS a boolean tests: replace test_tristate with "git env--helper" tests README: re-flow a previously changed paragraph tests: make GIT_TEST_GETTEXT_POISON a boolean t6040 test: stop using global "script" variable config.c: refactor die_bad_number() to not call gettext() early env--helper: new undocumented builtin wrapping git_env_*() config tests: simplify include cycle test
2019-07-19travis-ci: build with GCC 4.8 as wellLibravatar SZEDER Gábor1-4/+13
C99 'for' loop initial declaration, i.e. 'for (int i = 0; i < n; i++)', is not allowed in Git's codebase yet, to maintain compatibility with some older compilers. Our Travis CI builds used to catch 'for' loop initial declarations, because the GETTEXT_POISON job has always built Git with the default 'cc', which in Travis CI's previous default Linux image (based on Ubuntu 14.04 Trusty) is GCC 4.8, and that GCC version errors out on this construct (not only with DEVELOPER=1, but with our default CFLAGS as well). Alas, that's not the case anymore, becase after 14.04's EOL Travis CI's current default Linux image is based on Ubuntu 16.04 Xenial [1] and its default 'cc' is now GCC 5.4, which, just like all later GCC and Clang versions, simply accepts this construct, even if we don't explicitly specify '-std=c99'. Ideally we would adjust our CFLAGS used with DEVELOPER=1 to catch this undesired construct already when contributors build Git on their own machines. Unfortunately, however, there seems to be no compiler option that would catch only this particular construct without choking on many other things, e.g. while a later compiler with '-std=c90' and/or '-ansi' does catch this construct, it can't build Git because of several screenfulls of other errors. Add the 'linux-gcc-4.8' job to Travis CI, in order to build Git with GCC 4.8, and thus to timely catch any 'for' loop initial declarations. To catch those it's sufficient to only build Git with GCC 4.8, so don't run the test suite in this job, because 'make test' takes rather long [2], and it's already run five times in other jobs, so we wouldn't get our time's worth. [1] The Azure Pipelines builds have been using Ubuntu 16.04 images from the start, so I belive they never caught 'for' loop initial declarations. [2] On Travis CI 'make test' alone would take about 9 minutes in this new job (without running httpd, Subversion, and P4 tests). For comparison, starting the job and building Git with GCC 4.8 takes only about 2 minutes. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-07-08ci/lib.sh: update a comment about installed P4 and Git-LFS versionsLibravatar SZEDER Gábor1-2/+4
A comment in 'ci/lib.sh' claims that the "OS X build installs the latest available versions" of P4 and Git-LFS, but since f2f47150 ("ci: don't update Homebrew", 2019-07-03) that's no longer the case, as it will install the versions which were recorded in the image's Homebrew database when the image was created. Update this comment accordingly. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-07-03ci: disable Homebrew's auto cleanupLibravatar SZEDER Gábor1-1/+1
Lately Homebrew learned to automagically clean up information about outdated packages during other 'brew' commands, which might be useful for the avarage user, but is a waste of time in CI build jobs, because the next build jobs will start from the exact same image containing the same outdated packages anyway. Export HOMEBREW_NO_INSTALL_CLEANUP=1 to disable this auto cleanup feature, shaving off about 20-30s from the time needed to install dependencies in our macOS build jobs on Travis CI. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-07-03ci: don't update HomebrewLibravatar SZEDER Gábor1-1/+1
Lately our GCC macOS build job on Travis CI has been erroring out while installing dependencies with: +brew link gcc@8 Error: No such keg: /usr/local/Cellar/gcc@8 The command "ci/install-dependencies.sh" failed and exited with 1 during . Now, while gcc@8 is still pre-installed (but not linked) and would be perfectly usable in the Travis CI macOS image we use [1], it's at version 8.2. However, when installing dependencies we first explicitly run 'brew update', which spends over two minutes to update itself and information about the available packages, and it learns about GCC 8.3. After that point gcc@8 exclusively refers to v8.3, and, unfortunately, 'brew' is just too dumb to be able to do anything with the still installed 8.2 package, and the subsequent 'brew link gcc@8' fails. (Even 'brew uninstall gcc@8' fails with the same error!) Don't run 'brew update' to keep the already installed GCC 8.2 'brew link'-able. Note that in addition we have to 'export HOMEBREW_NO_AUTO_UPDATE=1' first, because 'brew' is so very helpful that it would implicitly run update for us on the next 'brew install <pkg>' otherwise. Disabling 'brew update' has additional benefits: - It shaves off 2-3mins from the ~4mins currently spent on installing dependencies, and the macOS build jobs have always been prone to exceeding the time limit on Travis CI. - Our builds won't suddenly break because of the occasional Homebrew breakages [2]. The drawback is that we'll be stuck with slightly older versions of the packages that we install via Homebrew (Git-LFS 2.5.2 and Perforce 2018.1; they are currently at 2.7.2 and 2019.1, respectively). We might want to reconsider this decision as time goes on and/or switch to a more recent macOS image as they become available. [1] 2000ac9fbf (travis-ci: switch to Xcode 10.1 macOS image, 2019-01-17) [2] See e.g. a1ccaedd62 (travis-ci: make the OSX build jobs' 'brew update' more quiet, 2019-02-02) or https://public-inbox.org/git/20180907032002.23366-1-szeder.dev@gmail.com/T/#+u Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-21tests: make GIT_TEST_GETTEXT_POISON a booleanLibravatar Ævar Arnfjörð Bjarmason1-1/+1
Change the GIT_TEST_GETTEXT_POISON variable from being "non-empty?" to being a more standard boolean variable. Since it needed to be checked in both C code and shellscript (via test -n) it was one of the remaining shellscript-like variables. Now that we have "env--helper" we can change that. There's a couple of tricky edge cases that arise because we're using git_env_bool() early, and the config-reading "env--helper". If GIT_TEST_GETTEXT_POISON is set to an invalid value die_bad_number() will die, but to do so it would usually call gettext(). Let's detect the special case of GIT_TEST_GETTEXT_POISON and always emit that message in the C locale, lest we infinitely loop. As seen in the updated tests in t0017-env-helper.sh there's also a caveat related to "env--helper" needing to read the config for trace2 purposes. Since the C_LOCALE_OUTPUT prerequisite is lazy and relies on "env--helper" we could get invalid results if we failed to read the config (e.g. because we'd loop on includes) when combined with e.g. "test_i18ngrep" wanting to check with "env--helper" if GIT_TEST_GETTEXT_POISON was true or not. I'm crossing my fingers and hoping that a test similar to the one I removed in the earlier "config tests: simplify include cycle test" change in this series won't happen again, and testing for this explicitly in "env--helper"'s own tests. This change breaks existing uses of e.g. GIT_TEST_GETTEXT_POISON=YesPlease, which we've documented in po/README and other places. As noted in [1] we might want to consider also accepting "YesPlease" in "env--helper" as a special-case. But as the lack of uproar over 6cdccfce1e ("i18n: make GETTEXT_POISON a runtime option", 2018-11-08) demonstrates the audience for this option is a really narrow set of git developers, who shouldn't have much trouble modifying their test scripts, so I think it's better to deal with that minor headache now and make all the relevant GIT_TEST_* variables boolean in the same way than carry the "YesPlease" special-case forward. 1. https://public-inbox.org/git/xmqqtvckm3h8.fsf@gitster-ct.c.googlers.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-05-19Merge branch 'sg/ci-libsvn-perl'Libravatar Junio C Hamano1-1/+1
To run tests for Git SVN, our scripts for CI used to install the git-svn package (in the hope that it would bring in the right dependencies). This has been updated to install the more direct dependency, namely, libsvn-perl. * sg/ci-libsvn-perl: ci: install 'libsvn-perl' instead of 'git-svn'
2019-05-07ci: install 'libsvn-perl' instead of 'git-svn'Libravatar SZEDER Gábor1-1/+1
Since e7e9f5e7a1 (travis-ci: enable Git SVN tests t91xx on Linux, 2016-05-19) some of our Travis CI build jobs install the 'git-svn' package, because it was a convenient way to install its dependencies, which are necessary to run our 'git-svn' tests (we don't actually need the 'git-svn' package itself). However, from those dependencies, namely the 'libsvn-perl', 'libyaml-perl', and 'libterm-readkey-perl' packages, only 'libsvn-perl' is necessary to run those tests, the others arent, not even to fulfill some prereqs. So update 'ci/install-dependencies.sh' to install only 'libsvn-perl' instead of 'git-svn' and its additional dependencies. Note that this change has more important implications than merely not installing three unnecessary packages, as it keeps our builds working with Travis CI's Xenial images. In our '.travis.yml' we never explicitly specified which Linux image we want to use to run our Linux build jobs, and so far they have been run on the default Ubuntu 14.04 Trusty image. However, 14.04 just reached its EOL, and Travis CI has already began the transition to use 16.04 Xenial as the default Linux build environment [1]. Alas, our Linux Clang and GCC build jobs can't simply 'apt-get install git-svn' in the current Xenial images [2], like they did in the Trusty images, and, consequently, fail. Installing only 'libsvn-perl' avoids this issue, while the 'git svn' tests are still run as they should. [1] https://blog.travis-ci.com/2019-04-15-xenial-default-build-environment [2] 'apt-get install git-svn' in the Xenial image fails with: The following packages have unmet dependencies: git-svn : Depends: git (< 1:2.7.4-.) E: Unable to correct problems, you have held broken packages. The reason is that both the Trusty and Xenial images contain the 'git' package installed from 'ppa:git-core/ppa', so it's considerably newer than the 'git' package in the corresponding standard Ubuntu package repositories. The difference is that the Trusty image still contains these third-party apt repositories, so the 'git-svn' package was installed from the same PPA, and its version matched the version of the already installed 'git' package. In the Xenial image, however, these third-party apt-repositories are removed (to reduce the risk of unrelated interference and faster 'apt-get update') [3], and the version of the 'git-svn' package coming from the standard Ubuntu package repositories doesn't match the much more recent version of the 'git' package installed from the PPA, resulting in this dependecy error. Adding back the 'ppa:git-core/ppa' package repository would solve this dependency issue as well, but since the troublesome package happens to be unnecessary, not installing it in the first place is better. [3] https://docs.travis-ci.com/user/reference/xenial/#third-party-apt-repositories-removed Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-05ci: fix AsciiDoc/Asciidoctor stderr check in the documentation build jobLibravatar SZEDER Gábor1-7/+16
In 'ci/test-documentation.sh' we save the standard error of 'make doc', and, in an attempt to make sure that neither AsciiDoc nor Asciidoctor printed any warnings, we check the emptiness of the resulting file with '! test -s stderr.log'. This check has never actually worked, because in our 'ci/*' build scripts we rely on 'set -e' aborting the build job when a command exits with error, and, unfortunately, the combination of the two doesn't work as intended. According to POSIX [1]: "The -e setting shall be ignored when executing [...] a pipeline beginning with the ! reserved word" [2] Watch and learn: $ echo unexpected >file $ ( set -e; ! test -s file ; echo "should not reach this" ) ; echo $? should not reach this 0 This is why we haven't noticed the warnings from Asciidoctor that were fixed in the first patches of this patch series, though some of them were already there in the build of v2.18.0-rc0 [3]. Check the emptiness of that file with 'test ! -s' instead, which works properly with 'set -e': $ ( set -e; test ! -s file ; echo "should not reach this" ) ; echo $? 1 Furthermore, dump the contents of that file to the log for our convenience, so if it were to unexpectedly end up being non-empty, then we wouldn't have to scroll through all that long build log looking for warnings, but could see them right away near the end of the log. Note that we are only really interested in the standard error of AsciiDoc and Asciidoctor, but by saving the stderr of 'make doc' we also save any error output from the make rules. Currently there is only one such line: we build the docs with Asciidoctor right after a 'make clean', meaning that 'make USE_ASCIIDOCTOR=1 doc' always starts with running 'GIT-VERSION-GEN', which in turn prints the version to stderr. A 'sed' command was supposed to remove this version line to prevent it from triggering that (previously defunct) emptiness check, but, unfortunately, this command doesn't work as intended, either, because it leaves the file to be checked intact, but that defunct emptiness check hid this issue, too... Furthermore, in the near future there will be an other line on stderr, because commit 9a71722b4d (Doc: auto-detect changed build flags, 2019-03-17) in the currently cooking branch 'ma/doc-diff-doc-vs-doctor-comparison' will print "* new asciidoc flags" at the beginning of both 'make doc' invokations. Extend that 'sed' command to remove this line, too, wrap it in a helper function so the output of both 'make doc' is filtered the same way, and change its invokation to actually write the logfile to be checked. [1] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#set [2] POSIX doesn't discuss the meaning of '! cmd' in case of simple commands, but it defines that "A pipeline is a sequence of one or more commands separated by the control operator '|'", so apparently a simple command is considered as pipeline as well. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_02 [3] https://travis-ci.org/git/git/jobs/385932007#L1463 Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-05ci: stick with Asciidoctor v1.5.8 for nowLibravatar SZEDER Gábor1-1/+1
The recent release of Asciidoctor v2.0.0 broke our documentation build job on Travis CI, where we 'gem install asciidoctor', which always brings us the latest and (supposedly) greatest. Alas, we are not ready for that just yet, because it removed support for DocBook 4.5, and we have been requiring that particular DocBook version to build 'user-manual.xml' with Asciidoctor, resulting in: ASCIIDOC user-manual.xml asciidoctor: FAILED: missing converter for backend 'docbook45'. Processing aborted. Use --trace for backtrace make[1]: *** [user-manual.xml] Error 1 Unfortunately, we can't simply switch to DocBook 5 right away, as doing so leads to validation errors from 'xmlto', and working around those leads to yet another errors... [1] So let's stick with Asciidoctor v1.5.8 (latest stable release before v2.0.0) in our documentation build job on Travis CI for now, until we figure out how to deal with the fallout from Asciidoctor v2.0.0. [1] https://public-inbox.org/git/20190324162131.GL4047@pobox.com/ Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-01ci: install Asciidoctor in 'ci/install-dependencies.sh'Libravatar SZEDER Gábor2-3/+3
When our '.travis.yml' was split into several 'ci/*' scripts [1], the installation of the 'asciidoctor' gem somehow ended up in 'ci/test-documentation.sh'. Install it in 'ci/install-dependencies.sh', where we install other dependencies of the Documentation build job as well (asciidoc, xmlto). [1] 657343a602 (travis-ci: move Travis CI code into dedicated scripts, 2017-09-10) Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-01travis: remove the hack to build the Windows job on Azure PipelinesLibravatar Johannes Schindelin1-103/+0
Since Travis did not support Windows (and now only supports very limited Windows jobs, too limited for our use, the test suite would time out *all* the time), we added a hack where a Travis job would trigger an Azure Pipeline (which back then was still called VSTS Build), wait for it to finish (or time out), and download the log (if available). Needless to say that it was a horrible hack, necessitated by a bad situation. Nowadays, however, we have Azure Pipelines support, and do not need that hack anymore. So let's retire it. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-07ci: clear and mark MAKEFLAGS exported just onceLibravatar Junio C Hamano1-3/+6
Clearing it once upfront, and turning all the assignment into appending, would future-proof the code even more, to prevent mistakes the previous one fixed from happening again. Also, mark the variable exported just once at the beginning. There is no point in marking it exported repeatedly. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-07ci: make sure we build Git parallelLibravatar SZEDER Gábor1-1/+1
Commit 2c8921db2b (travis-ci: build with the right compiler, 2019-01-17) started to use MAKEFLAGS to specify which compiler to use to build Git. A bit later, and in a different topic branch commit eaa62291ff (ci: inherit --jobs via MAKEFLAGS in run-build-and-tests, 2019-01-27) started to use MAKEFLAGS as well. Unfortunately, there is a semantic conflict between these two commits: both of them set MAKEFLAGS, and since the line adding CC from 2c8921db2b comes later in 'ci/lib.sh', it overwrites the number of parallel jobs added in eaa62291ff. Consequently, since both commits have been merged all our CI jobs have been building Git, building its documentation, and applying semantic patches sequentially, making all build jobs a bit slower. Running the test suite is unaffected, because the number of test jobs comes from GIT_PROVE_OPTS. Append to MAKEFLAGS when setting the compiler to use, to ensure that the number of parallel jobs to use is preserved. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-06Merge branch 'sg/travis-osx-brew-breakage-workaround'Libravatar Junio C Hamano1-1/+1
The way the OSX build jobs updates its build environment used the "--quiet" option to "brew update" command, but it wasn't all that quiet to be useful. The use of the option has been replaced with an explicit redirection to the /dev/null (which incidentally would have worked around a breakage by recent updates to homebrew, which has fixed itself already). * sg/travis-osx-brew-breakage-workaround: travis-ci: make the OSX build jobs' 'brew update' more quiet
2019-02-06Merge branch 'js/vsts-ci'Libravatar Junio C Hamano12-35/+157
Prepare to run test suite on Azure Pipeline. * js/vsts-ci: (22 commits) test-date: drop unused parameter to getnanos() ci: parallelize testing on Windows ci: speed up Windows phase tests: optionally skip bin-wrappers/ t0061: workaround issues with --with-dashes and RUNTIME_PREFIX tests: add t/helper/ to the PATH with --with-dashes mingw: try to work around issues with the test cleanup tests: include detailed trace logs with --write-junit-xml upon failure tests: avoid calling Perl just to determine file sizes README: add a build badge (status of the Azure Pipelines build) mingw: be more generous when wrapping up the setitimer() emulation ci: use git-sdk-64-minimal build artifact ci: add a Windows job to the Azure Pipelines definition Add a build definition for Azure DevOps ci/lib.sh: add support for Azure Pipelines tests: optionally write results as JUnit-style .xml test-date: add a subcommand to measure times in shell scripts ci: use a junction on Windows instead of a symlink ci: inherit --jobs via MAKEFLAGS in run-build-and-tests ci/lib.sh: encapsulate Travis-specific things ...
2019-02-04travis-ci: make the OSX build jobs' 'brew update' more quietLibravatar SZEDER Gábor1-1/+1
Before installing the necessary dependencies, our OSX build jobs run 'brew update --quiet'. This is problematic for two reasons: - This '--quiet' flag apparently broke overnight, resulting in errored builds: +brew update --quiet ==> Downloading https://homebrew.bintray.com/bottles-portable-ruby/portable-ruby-2.3.7.mavericks.bottle.tar.gz ######################################################################## 100.0% ==> Pouring portable-ruby-2.3.7.mavericks.bottle.tar.gz Usage: brew update_report [--preinstall] The Ruby implementation of brew update. Never called manually. --preinstall Run in 'auto-update' mode (faster, less output). -f, --force Override warnings and enable potentially unsafe operations. -d, --debug Display any debugging information. -v, --verbose Make some output more verbose. -h, --help Show this message. Error: invalid option: --quiet The command "ci/install-dependencies.sh" failed and exited with 1 during . I belive that this breakage will be noticed and fixed soon-ish, so we could probably just wait a bit for this issue to solve itself, but: - 'brew update --quiet' wasn't really quiet in the first place, as it listed over about 2000 lines worth of available packages that we absolutely don't care about, see e.g. one of the latest 'master' builds: https://travis-ci.org/git/git/jobs/486134962#L113 So drop this '--quiet' option and redirect 'brew update's standard output to /dev/null to make it really quiet, thereby making the OSX builds work again despite the above mentioned breakage. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29ci: parallelize testing on WindowsLibravatar Johannes Schindelin2-0/+29
The fact that Git's test suite is implemented in Unix shell script that is as portable as we can muster, combined with the fact that Unix shell scripting is foreign to Windows (and therefore has to be emulated), results in pretty abysmal speed of the test suite on that platform, for pretty much no other reason than that language choice. For comparison: while the Linux build & test is typically done within about 8 minutes, the Windows build & test typically lasts about 80 minutes in Azure Pipelines. To help with that, let's use the Azure Pipeline feature where you can parallelize jobs, make jobs depend on each other, and pass artifacts between them. The tests are distributed using the following heuristic: listing all test scripts ordered by size in descending order (as a cheap way to estimate the overall run time), every Nth script is run (where N is the total number of parallel jobs), starting at the index corresponding to the parallel job. This slicing is performed by a new function that is added to the `test-tool`. To optimize the overall runtime of the entire Pipeline, we need to move the Windows jobs to the beginning (otherwise there would be a very decent chance for the Pipeline to be run only the Windows build, while all the parallel Windows test jobs wait for this single one). We use Azure Pipelines Artifacts for both the minimal Git for Windows SDK as well as the built executables, as deduplication and caching close to the agents makes that really fast. For comparison: while downloading and unpacking the minimal Git for Windows SDK via PowerShell takes only one minute (down from anywhere between 2.5 to 7 when using a shallow clone), uploading it as Pipeline Artifact takes less than 30s and downloading and unpacking less than 20s (sometimes even as little as only twelve seconds). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29ci: speed up Windows phaseLibravatar Johannes Schindelin1-0/+2
As Unix shell scripting comes at a hefty price on Windows, we have to see where we can save some time to run the test suite. Let's skip the chain linting and the bin-wrappers/ redirection on Windows; this seems to shave of anywhere between 10-30% from the overall runtime. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29Add a build definition for Azure DevOpsLibravatar Johannes Schindelin1-0/+25
This commit adds an azure-pipelines.yml file which is Azure DevOps' equivalent to Travis CI's .travis.yml. The main idea is to replicate the Travis configuration as faithfully as possible, to make it easy to compare the Azure Pipeline builds to the Travis ones (spoiler: some parts, especially the macOS jobs, are way faster in Azure Pileines). Meaning: the number and the order of the jobs added in this commit faithfully replicates what we have in .travis.yml. Note: Our .travis.yml configuration has a Windows part that is *not* replicated in the Azure Pipelines definition. The reason is easy to see: As Travis cannot support our Windws needs (even with the preliminary Windows support that was recently added to Travis after waiting for *years* for that feature, our test suite would simply hit Travis' timeout every single time). To make things a bit easier to understand, we refrain from using the `matrix` feature here because (while it is powerful) it can be a bit confusing to users who are not familiar with CI setups. Therefore, we use a separate phase even for similar configurations (such as GCC vs Clang on Linux, GCC vs Clang on macOS). Also, we make use of the shiny new feature we just introduced where the test suite can output JUnit-style .xml files. This information is made available in a nice UI that allows the viewer to filter by phase and/or test number, and to see trends such as: number of (failing) tests, time spent running the test suite, etc. (While this seemingly contradicts the intention to replicate the Travis configuration as faithfully as possible, it is just too nice to show off that capability here already.) Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29ci/lib.sh: add support for Azure PipelinesLibravatar Johannes Schindelin2-0/+30
This patch introduces a conditional arm that defines some environment variables and a function that displays the URL given the job id (to identify previous runs for known-good trees). Because Azure Pipeline's macOS agents already have git-lfs and gettext installed, we can leave `BREW_INSTALL_PACKAGES` empty (unlike in Travis' case). Note: this patch does not introduce an Azure Pipelines definition yet; That is left for the next patch. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-28ci: use a junction on Windows instead of a symlinkLibravatar Johannes Schindelin1-1/+4
Symbolic links are still not quite as easy to use on Windows as on Linux (for example, on versions older than Windows 10, only administrators can create symlinks, and on Windows 10 you still need to be in developer mode for regular users to have permission), but NTFS junctions can give us a way out. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-28ci: inherit --jobs via MAKEFLAGS in run-build-and-testsLibravatar Johannes Schindelin5-5/+6
Let's not decide in the generic ci/ part how many jobs to run in parallel; different CI configurations would favor a different number of parallel jobs, and it is easy enough to hand that information down via the `MAKEFLAGS` variable. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-28ci/lib.sh: encapsulate Travis-specific thingsLibravatar Johannes Schindelin4-14/+43
The upcoming patches will allow building git.git via Azure Pipelines (i.e. Azure DevOps' Continuous Integration), where variable names and URLs look a bit different than in Travis CI. Also, the configurations of the available agents are different. For example, Travis' and Azure Pipelines' macOS agents are set up differently, so that on Travis, we have to install the git-lfs and gettext Homebrew packages, and on Azure Pipelines we do not need to. Likewise, Azure Pipelines' Ubuntu agents already have asciidoctor installed. Finally, on Azure Pipelines the natural way is not to base64-encode tar files of the trash directories of failed tests, but to publish build artifacts instead. Therefore, that code to log those base64-encoded tar files is guarded to be Travis-specific. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-28ci: rename the library of common functionsLibravatar Johannes Schindelin8-7/+7
The name is hard-coded to reflect that we use Travis CI for continuous testing. In the next commits, we will extend this to be able use Azure DevOps, too. So let's adjust the name to make it more generic. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-28travis: fix skipping tagged releasesLibravatar Johannes Schindelin1-11/+14
When building a PR, TRAVIS_BRANCH refers to the *target branch*. Therefore, if a PR targets `master`, and `master` happened to be tagged, we skipped the build by mistake. Fix this by using TRAVIS_PULL_REQUEST_BRANCH (i.e. the *source branch*) when available, falling back to TRAVIS_BRANCH (i.e. for CI builds, also known as "push builds"). Let's give it a new variable name, too: CI_BRANCH (as it is different from TRAVIS_BRANCH). This also prepares for the upcoming patches which will make our ci/* code a bit more independent from Travis and open it to other CI systems (in particular to Azure Pipelines). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-17travis-ci: build with the right compilerLibravatar SZEDER Gábor2-3/+17
Our 'Makefile' hardcodes the compiler to build Git as 'CC = cc'. This CC variable can be overridden from the command line, i.e. 'make CC=gcc-X.Y' will build with that particular GCC version, but not from the environment, i.e. 'CC=gcc-X.Y make' will still build with whatever 'cc' happens to be on the platform. Our build jobs on Travis CI are badly affected by this. In the build matrix we have dedicated build jobs to build Git with GCC and Clang both on Linux and macOS from the very beginning (522354d70f (Add Travis CI support, 2015-11-27)). Alas, this never really worked as supposed to, because Travis CI specifies the compiler for those build jobs as 'export CC=gcc' and 'export CC=clang' (which works fine for projects built with './configure && make'). Consequently, our 'linux-clang' build job has always used GCC, because that's where 'cc' points at in Travis CI's Linux images, while the 'osx-gcc' build job has always used Clang. Furthermore, 37fa4b3c78 (travis-ci: run gcc-8 on linux-gcc jobs, 2018-05-19) added an 'export CC=gcc-8' in an attempt to build with a more modern compiler, but to no avail. Set MAKEFLAGS with CC based on the $CC environment variable, so 'make' will run the "right" compiler. The Xcode 10.1 macOS image on Travis CI already contains the gcc@8 package from Homebrew, but we have to 'brew link' it first to be able to use it. So with this patch our build jobs will build Git with the following compiler versions: linux-clang: clang version 5.0.0 (tags/RELEASE_500/final) linux-gcc: gcc-8 (Ubuntu 8.1.0-5ubuntu1~14.04) 8.1.0 osx-clang: Apple LLVM version 10.0.0 (clang-1000.11.45.5) osx-gcc: gcc-8 (Homebrew GCC 8.2.0) 8.2.0 GETTEXT_POISON: gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4 Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>