summaryrefslogtreecommitdiff
path: root/t/perf/perf-lib.sh
AgeCommit message (Collapse)AuthorFilesLines
2019-03-18perf-lib.sh: rely on test-lib.sh for --tee handlingLibravatar Jeff King1-23/+11
Since its inception, the perf-lib.sh script has manually handled the "--tee" option (and other options which imply it, like "--valgrind") with a cut-and-pasted block from test-lib.sh. That block has grown stale over the years, and has at least three problems: 1. It uses $SHELL to re-exec the script, whereas the version in test-lib.sh learned to use $TEST_SHELL_PATH. 2. It does an ad-hoc search of the "$*" string, whereas test-lib.sh learned to carefully parse the arguments left to right. 3. It never learned about --verbose-log (which also implies --tee), so it would not trigger for that option. This last one was especially annoying, because t/perf/run uses the GIT_TEST_OPTS from your config.mak to run the perf scripts. So if you've set, say, "-x --verbose-log" there, it will be passed as part of most perf runs. And while this script doesn't recognize the option, the test-lib.sh that we source _does_, and the behavior ends up being much more annoying: - as the comment at the top of the block says, we have to run this tee code early, before we start munging variables (it says GIT_BUILD_DIR, but the problematic variable is actually GIT_TEST_INSTALLED). - since we don't recognize --verbose-log, we don't trigger the block. We go on to munge GIT_TEST_INSTALLED, converting it from a relative to an absolute path. - then we source test-lib.sh, which _does_ recognize --verbose-log. It re-execs the script, which runs again. But this time with an absolute version of GIT_TEST_INSTALLED. - As a result, we copy the absolute version of GIT_TEST_INSTALLED into perf_results_prefix. Instead of writing our results to the expected "test-results/build_1234abcd.p1234-whatever.times", we instead write them to "test-results/_full_path_to_repo_t_perf_build_1234...". The aggregate.perl script doesn't expect this, and so it prints "<missing>" for each result (even though it spent considerable time running the tests!). We can solve all of these in one blow by just deleting our custom handling, and relying on the inclusion of test-lib.sh to handle --tee, --verbose-log, etc. There's one catch, though. We want to handle GIT_TEST_INSTALLED after we've included test-lib.sh, since we want it un-munged in the re-exec'd version of the script. But if we want to convert it from a relative to an absolute path, we must do so before we load test-lib.sh, since it will change our working directory. So we compute the absolute directory first, store it away, then include test-lib.sh, and finally assign to GIT_TEST_INSTALLED as appropriate. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-20tests: send "bug in the test script" errors to the script's stderrLibravatar SZEDER Gábor1-2/+2
Some of the functions in our test library check that they were invoked properly with conditions like this: test "$#" = 2 || error "bug in the test script: not 2 parameters to test-expect-success" If this particular condition is triggered, then 'error' will abort the whole test script with a bold red error message [1] right away. However, under certain circumstances the test script will be aborted completely silently, namely if: - a similar condition in a test helper function like 'test_line_count' is triggered, - which is invoked from the test script's "main" shell [2], - and the test script is run manually (i.e. './t1234-foo.sh' as opposed to 'make t1234-foo.sh' or 'make test') [3] - and without the '--verbose' option, because the error message is printed from within 'test_eval_', where standard output is redirected either to /dev/null or to a log file. The only indication that something is wrong is that not all tests in the script are executed and at the end of the test script's output there is no "# passed all N tests" message, which are subtle and can easily go unnoticed, as I had to experience myself. Send these "bug in the test script" error messages directly to the test scripts standard error and thus to the terminal, so those bugs will be much harder to overlook. Instead of updating all ~20 such 'error' calls with a redirection, let's add a BUG() function to 'test-lib.sh', wrapping an 'error' call with the proper redirection and also including the common prefix of those error messages, and convert all those call sites [4] to use this new BUG() function instead. [1] That particular error message from 'test_expect_success' is printed in color only when running with or without '--verbose'; with '--tee' or '--verbose-log' the error is printed without color, but it is printed to the terminal nonetheless. [2] If such a condition is triggered in a subshell of a test, then 'error' won't be able to abort the whole test script, but only the subshell, which in turn causes the test to fail in the usual way, indicating loudly and clearly that something is wrong. [3] Well, 'error' aborts the test script the same way when run manually or by 'make' or 'prove', but both 'make' and 'prove' pay attention to the test script's exit status, and even a silently aborted test script would then trigger those tools' usual noticable error messages. [4] Strictly speaking, not all those 'error' calls need that redirection to send their output to the terminal, see e.g. 'test_expect_success' in the opening example, but I think it's better to be consistent. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-08-20t/perf: add infrastructure for measuring sizesLibravatar Jeff King1-0/+13
The main objective of scripts in the perf framework is to run "test_perf", which measures the time it takes to run some operation. However, it can also be interesting to see the change in the output size of certain operations. This patch introduces test_size, which records a single numeric output from the test and shows it in the aggregated output (with pretty printing and relative size comparison). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-08-20t/perf: factor boilerplate out of test_perfLibravatar Jeff King1-26/+35
About half of test_perf() is boilerplate preparing to run _any_ test, and the other half is specifically running a timing test. Let's split it into two functions, so that we can reuse the boilerplate in future commits. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-24perf: store subsection results in "test-results/$GIT_PERF_SUBSECTION/"Libravatar Christian Couder1-0/+1
When tests are run for a subsection defined in a config file, it is better if the results for the current subsection are not overwritting the results of a previous subsection. So let's store the results for a subsection in a subdirectory of "test-results/" with the subsection name. The aggregate.perl, when it is run for a subsection, should then aggregate the results found in "test-results/$GIT_PERF_SUBSECTION/". Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-24perf/run: add get_var_from_env_or_config()Libravatar Christian Couder1-3/+0
Add get_var_from_env_or_config() to easily set variables from a config file if they are defined there and not already set. This can also set them to a default value if one is provided. As an example, use this function to set GIT_PERF_REPEAT_COUNT from the perf.repeatCount config option or from the default value. Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-06-05perf: work around the tested repo having an index.lockLibravatar Ævar Arnfjörð Bjarmason1-1/+8
When the tested repo has an index.lock file it should be removed. This file may be present if e.g. git-status previously crashed in that repo, and it will make a lot of git commands fail. Let's try harder and remove the lock. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-12perf: add function to setup a fresh test repoLibravatar Ævar Arnfjörð Bjarmason1-4/+15
Add a function to setup a fresh test repo via 'git init' to compliment the existing functions to copy over a normal & large repo. Some performance tests don't need any existing repository data at all to be significant, e.g. tests which stress glob matches against single pathological revisions or files, which I'm about to add in a subsequent commit. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-03t/perf: use $MODERN_GIT for all repo-copying stepsLibravatar Jeff King1-2/+2
Since 1a0962dee (t/perf: fix regression in testing older versions of git, 2016-06-22), we point "$MODERN_GIT" to a copy of git that matches the t/perf script itself, and which can be used for tasks outside of the actual timings. This is needed because the setup done by perf scripts keeps moving forward in time, and may use features that the older versions of git we are testing do not have. That commit used $MODERN_GIT to fix a case where we relied on the relatively recent --git-path option. But if you go back further still, there are more problems. Since 7501b5921 (perf: make the tests work in worktrees, 2016-05-13), we use "git -C", but versions of git older than 44e1e4d67 (git: run in a directory given with -C option, 2013-09-09) don't know about "-C". So testing an old version of git with a new version of t/perf will fail the setup step. We can fix this by using $MODERN_GIT during the setup; there's no need to use the antique version, since it doesn't affect the timings. Likewise, we'll adjust the "init" invocation; antique versions of git called this "init-db". Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-11Merge branch 'jk/perf-any-version'Libravatar Junio C Hamano1-1/+4
Allow t/perf framework to use the features from the most recent version of Git even when testing an older installed version. * jk/perf-any-version: p4211: explicitly disable renames in no-rename test t/perf: fix regression in testing older versions of git
2016-06-22t/perf: fix regression in testing older versions of gitLibravatar Jeff King1-1/+4
Commit 7501b59 (perf: make the tests work in worktrees, 2016-05-13) introduced the use of "git rev-parse --git-path" in the perf-lib setup code. Because the to-be-tested version of git is at the front of the $PATH when this code runs, this means we cannot use modern versions of t/perf to test versions of git older than v2.5.0 (when that option was introduced). This is a symptom of a more general problem. The t/perf suite is essentially independent of git versions, and ideally we would be able to run the most modern and complete set of tests across many historical versions (to see how they compare). But any setup code they run is therefore required to use the lowest common denominator we expect to test. So let's introduce a new variable, $MODERN_GIT, that we can use both in perf-lib and in the test setup to get a reliable set of git features (we might change git and break some tests, of course, but $MODERN_GIT is tied to the same version of git as the t/perf scripts, so they can be fixed or adjusted together). This commit fixes the "--git-path" case, but does not mass-convert existing setup code to use $MODERN_GIT. Most setup code is fairly vanilla and will work with effectively all versions. But now the tool is there to fix any other issues we find going forward. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-06-21perf: accommodate for MacOSXLibravatar Johannes Schindelin1-1/+5
As this developer has no access to MacOSX developer setups anymore, Travis becomes the best bet to run performance tests on that OS. However, on MacOSX /usr/bin/time is that good old BSD executable that no Linux user cares about, as demonstrated by the perf-lib.sh's use of GNU-ish extensions. And by the hard-coded path. Let's just work around this issue by using gtime on MacOSX, the Homebrew-provided GNU implementation onto which pretty much every MacOSX power user falls back anyway. To help other developers use Travis to run performance tests on MacOSX, the .travis.yml file now sports a commented-out line that installs GNU time via Homebrew. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Reviewed-by: Lars Schneider <larsxschneider@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-05-31perf: make the tests work without a worktreeLibravatar René Scharfe1-1/+4
In regular repositories $source_git and $objects_dir contain relative paths based on $source. Go there to allow cp to resolve them. Signed-off-by: Rene Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-05-13perf: make the tests work in worktreesLibravatar Johannes Schindelin1-7/+7
This patch makes perf-lib.sh more robust so that it can run correctly even inside a worktree. For example, it assumed that $GIT_DIR/objects is the objects directory (which is not the case for worktrees) and it used the commondir file verbatim, even if it contained a relative path. Furthermore, the setup code expected `git rev-parse --git-dir` to spit out a relative path, which is also not true for worktrees. Let's just change the code to accept both relative and absolute paths, by avoiding the `cd` into the copied working directory. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-05-13perf: let's disable symlinks when they are not availableLibravatar Johannes Schindelin1-1/+4
We already have a perfectly fine prereq to tell us whether it is safe to use symlinks. So let's use it. This fixes the performance tests in Git for Windows' SDK, where symlinks are not really available ([*1*]). This is not an issue with Git for Windows itself because it configures core.symlinks=false in its system config. However, the system config is disabled for the performance tests, for obvious reasons: we want them to be independent of the vagaries of any local configuration. Footnote *1*: Windows has symbolic links. Git for Windows disables them by default, though (for example: in standard setups, non-admins lack the privilege to create symbolic links). For details, see https://github.com/git-for-windows/git/wiki/Symbolic-Links Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-25perf-lib: fix ignored exit code inside loopLibravatar Jeff King1-1/+1
When copying the test repository, we try to detect whether the copy succeeded. However, most of the heavy lifting is done inside a for loop, where our "break" will lose the exit code of the failing "cp". We can take advantage of the fact that we are in a subshell, and just "exit 1" to break out with a code. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-11-26test: replace shebangs with descriptions in shell librariesLibravatar Jonathan Nieder1-1/+3
A #! line in these files is misleading, since these scriptlets are meant to be sourced with '.' (using whatever shell sources them) instead of run directly using the interpreter named on the #! line. Removing the #! line shouldn't hurt syntax highlighting since these files have filenames ending with '.sh'. For documentation, add a brief description of how the files are meant to be used in place of the shebang line. Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-01Merge branch 'lf/echo-n-is-not-portable'Libravatar Junio C Hamano1-2/+2
* lf/echo-n-is-not-portable: Avoid using `echo -n` anywhere
2013-07-29Avoid using `echo -n` anywhereLibravatar Lukas Fleischer1-2/+2
`echo -n` is non-portable. The POSIX specification says: Conforming applications that wish to do prompting without <newline> characters or that could possibly be expecting to echo a -n, should use the printf utility derived from the Ninth Edition system. Since all of the affected shell scripts use a POSIX shell shebang, replace `echo -n` invocations with printf. Signed-off-by: Lukas Fleischer <git@cryptocrack.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-06-29perf-lib: fix start/stop of perf testsLibravatar Thomas Gummerer1-1/+2
ae75342 test-lib: rearrange start/end of test_expect_* and test_skip changed the way tests are started/stopped, but did not update the perf tests. They were therefore giving the wrong output, because of the wrong test count. Fix this by starting and stopping the tests correctly. Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com> Acked-by: Thomas Rast <trast@inf.ethz.ch> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-09-26MALLOC_CHECK: enable it, unless disabled explicitlyLibravatar René Scharfe1-1/+1
The malloc checks in tests are currently disabled. Actually evaluate the variable for turning them off and enable them if it's unset. Also use this opportunity to give it the more descriptive and consistent name TEST_NO_MALLOC_CHECK. Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-09-17MALLOC_CHECK: various clean-upsLibravatar Junio C Hamano1-0/+1
The most important in this change is to avoid affecting anything when test-lib is used from perf-lib. It also limits the effect of the MALLOC_CHECK only to what is run inside the actual test, and uses a fixed MALLOC_PERTURB_ in order to avoid hurting repeatability of the tests. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-08-04tests: Introduce test_seqLibravatar Michał Kiedrowicz1-1/+1
Jeff King wrote: The seq command is GNU-ism, and is missing at least in older BSD releases and their derivatives, not to mention antique commercial Unixes. We already purged it in b3431bc (Don't use seq in tests, not everyone has it, 2007-05-02), but a few new instances have crept in. They went unnoticed because they are in scripts that are not run by default. Replace them with test_seq that is implemented with a Perl snippet (proposed by Jeff). This is better than inlining this snippet everywhere it's needed because it's easier to read and it's easier to change the implementation (e.g. to C) if we ever decide to remove Perl from the test suite. Note that test_seq is not a complete replacement for seq(1). It just has what we need now, in addition that it makes it possible for us to do something like "test_seq a m" if we wanted to in the future. There are also many places that do `for i in 1 2 3 ...` but I'm not sure if it's worth converting them to test_seq. That would introduce running more processes of Perl. Signed-off-by: Michał Kiedrowicz <michal.kiedrowicz@gmail.com> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-03-08perf: export some important test-lib variablesLibravatar Thomas Rast1-0/+4
The only bug right now is that $GIT_TEST_CMP is needed for test_cmp to work. However, we also export the three most important paths for tests: TEST_DIRECTORY TRASH_DIRECTORY GIT_BUILD_DIR Since they are available within test_expect_success, a future test writer may expect them to also be defined in test_perf. Signed-off-by: Thomas Rast <trast@student.ethz.ch> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-03-08perf: load test-lib-functions from the correct directoryLibravatar Thomas Rast1-1/+1
Loading it in the subshells still referred to $TEST_DIRECTORY/.., which was only correct in preliminary versions of perf-lib.sh Signed-off-by: Thomas Rast <trast@student.ethz.ch> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-02-17Introduce a performance testing frameworkLibravatar Thomas Rast1-0/+198
This introduces a performance testing framework under t/perf/. It tries to be as close to the test-lib.sh infrastructure as possible, and thus should be easy to get used to for git developers. The following points were considered for the implementation: 1. You usually want to compare arbitrary revisions/build trees against each other. They may not have the performance test under consideration, or even the perf-lib.sh infrastructure. To cope with this, the 'run' script lets you specify arbitrary build dirs and revisions. It even automatically builds the revisions if it doesn't have them at hand yet. 2. Usually you would not want to run all tests. It would take too long anyway. The 'run' script lets you specify which tests to run; or you can also do it manually. There is a Makefile for discoverability and 'make clean', but it is not meant for real-world use. 3. Creating test repos from scratch in every test is extremely time-consuming, and shipping or downloading such large/weird repos is out of the question. We leave this decision to the user. Two different sizes of test repos can be configured, and the scripts just copy one or more of those (using hardlinks for the object store). By default it tries to use the build tree's git.git repository. This is fairly fast and versatile. Using a copy instead of a clone preserves many properties that the user may want to test for, such as lots of loose objects, unpacked refs, etc. Signed-off-by: Thomas Rast <trast@student.ethz.ch> Signed-off-by: Junio C Hamano <gitster@pobox.com>