summaryrefslogtreecommitdiff
path: root/t
AgeCommit message (Collapse)AuthorFilesLines
2018-11-12read-cache.c: mark more strings for translationLibravatar Nguyễn Thái Ngọc Duy1-1/+1
There are a couple other improvements on these strings as well: - add missing colon (as separator) - quote paths - provide more information on error messages - keep first word in lowercase Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-26Merge branch 'ld/p4-unshelve'Libravatar Junio C Hamano1-16/+59
"git p4 unshelve" improvements. * ld/p4-unshelve: git-p4: fully support unshelving changelists git-p4: unshelve into refs/remotes/p4-unshelved, not refs/remotes/p4/unshelved git-p4: do not fail in verbose mode for missing 'fileSize' key
2018-10-26Merge branch 'np/log-graph-octopus-fix'Libravatar Junio C Hamano1-0/+102
"git log --graph" showing an octopus merge sometimes miscounted the number of display columns it is consuming to show the merge and its parent commits, which has been corrected. * np/log-graph-octopus-fix: log: fix coloring of certain octopus merge shapes
2018-10-26Merge branch 'sg/split-index-racefix'Libravatar Junio C Hamano2-21/+242
The codepath to support the experimental split-index mode had remaining "racily clean" issues fixed. * sg/split-index-racefix: split-index: BUG() when cache entry refers to non-existing shared entry split-index: smudge and add racily clean cache entries to split index split-index: don't compare cached data of entries already marked for split index split-index: count the number of deleted entries t1700-split-index: date back files to avoid racy situations split-index: add tests to demonstrate the racy split index problem t1700-split-index: document why FSMONITOR is disabled in this test script
2018-10-26Merge branch 'ma/t7005-bash-workaround'Libravatar Junio C Hamano1-1/+1
Test fix. * ma/t7005-bash-workaround: t7005-editor: quote filename to fix whitespace-issue
2018-10-19Merge branch 'jt/cache-tree-allow-missing-object-in-partial-clone'Libravatar Junio C Hamano1-0/+33
In a partial clone that will lazily be hydrated from the originating repository, we generally want to avoid "does this object exist (locally)?" on objects that we deliberately omitted when we created the clone. The cache-tree codepath (which is used to write a tree object out of the index) however insisted that the object exists, even for paths that are outside of the partial checkout area. The code has been updated to avoid such a check. * jt/cache-tree-allow-missing-object-in-partial-clone: cache-tree: skip some blob checks in partial clone
2018-10-19Merge branch 'tb/filter-alternate-refs'Libravatar Junio C Hamano1-0/+41
When pushing into a repository that borrows its objects from an alternate object store, "git receive-pack" that responds to the push request on the other side lists the tips of refs in the alternate to reduce the amount of objects transferred. This sometimes is detrimental when the number of refs in the alternate is absurdly large, in which case the bandwidth saved in potentially fewer objects transferred is wasted in excessively large ref advertisement. The alternate refs that are advertised are now configurable with a pair of configuration variables. * tb/filter-alternate-refs: transport.c: introduce core.alternateRefsPrefixes transport.c: introduce core.alternateRefsCommand transport.c: extract 'fill_alternate_refs_command' transport: drop refnames from for_each_alternate_ref
2018-10-19Merge branch 'jt/avoid-ls-refs'Libravatar Junio C Hamano2-0/+32
Over some transports, fetching objects with an exact commit object name can be done without first seeing the ref advertisements. The code has been optimized to exploit this. * jt/avoid-ls-refs: fetch: do not list refs if fetching only hashes transport: list refs before fetch if necessary transport: do not list refs if possible transport: allow skipping of ref listing
2018-10-19Merge branch 'jt/non-blob-lazy-fetch'Libravatar Junio C Hamano1-0/+41
A partial clone that is configured to lazily fetch missing objects will on-demand issue a "git fetch" request to the originating repository to fill not-yet-obtained objects. The request has been optimized for requesting a tree object (and not the leaf blob objects contained in it) by telling the originating repository that no blobs are needed. * jt/non-blob-lazy-fetch: fetch-pack: exclude blobs when lazy-fetching trees fetch-pack: avoid object flags if no_dependents
2018-10-19Merge branch 'rs/grep-no-recursive'Libravatar Junio C Hamano1-0/+12
Unlike "grep", "git grep" by default recurses to the whole tree. The command learned "git grep --recursive" option, so that "git grep --no-recursive" can serve as a synonym to setting the max-depth to 0. * rs/grep-no-recursive: grep: add -r/--[no-]recursive
2018-10-19Merge branch 'nd/help-commands-verbose-by-default'Libravatar Junio C Hamano1-2/+2
"git help -a" and "git help -av" give different pieces of information, and generally the "verbose" version is more friendly to the new users. "git help -a" by default now uses the more verbose output (with "--no-verbose", you can go back to the original). Also "git help -av" now lists aliases and external commands, which it did not used to. * nd/help-commands-verbose-by-default: help -a: improve and make --verbose default
2018-10-19Merge branch 'sm/show-superproject-while-conflicted'Libravatar Junio C Hamano1-0/+16
A corner-case bugfix. * sm/show-superproject-while-conflicted: rev-parse: --show-superproject-working-tree should work during a merge
2018-10-19Merge branch 'jt/fetch-tips-in-partial-clone'Libravatar Junio C Hamano1-0/+17
"git fetch $repo $object" in a partial clone did not correctly fetch the asked-for object that is referenced by an object in promisor packfile, which has been fixed. * jt/fetch-tips-in-partial-clone: fetch: in partial clone, check presence of targets connected: document connectivity in partial clones
2018-10-19Merge branch 'bp/read-cache-parallel'Libravatar Junio C Hamano2-4/+14
A new extension to the index file has been introduced, which allows the file to be read in parallel. * bp/read-cache-parallel: read-cache: load cache entries on worker threads ieot: add Index Entry Offset Table (IEOT) extension read-cache: load cache extensions on a worker thread config: add new index.threads config setting eoie: add End of Index Entry (EOIE) extension read-cache: clean up casting and byte decoding read-cache.c: optimize reading index format v4
2018-10-19Merge branch 'bp/rename-test-env-var'Libravatar Junio C Hamano5-13/+52
Some environment variables that control the runtime options of Git used during tests are getting renamed for consistency. * bp/rename-test-env-var: t0000: do not get self-test disrupted by environment warnings preload-index: update GIT_FORCE_PRELOAD_TEST support read-cache: update TEST_GIT_INDEX_VERSION support fsmonitor: update GIT_TEST_FSMONITOR support preload-index: use git_env_bool() not getenv() for customization t/README: correct spelling of "uncommon"
2018-10-19Merge branch 'ss/wt-status-committable'Libravatar Junio C Hamano1-2/+8
Code clean-up in the internal machinery used by "git status" and "git commit --dry-run". * ss/wt-status-committable: roll wt_status_state into wt_status and populate in the collect phase wt-status.c: set the committable flag in the collect phase t7501: add test of "commit --dry-run --short" wt-status: rename commitable to committable wt-status.c: move has_unmerged earlier in the file
2018-10-19Merge branch 'nd/the-index'Libravatar Junio C Hamano1-1/+1
Various codepaths in the core-ish part learn to work on an arbitrary in-core index structure, not necessarily the default instance "the_index". * nd/the-index: (23 commits) revision.c: reduce implicit dependency the_repository revision.c: remove implicit dependency on the_index ws.c: remove implicit dependency on the_index tree-diff.c: remove implicit dependency on the_index submodule.c: remove implicit dependency on the_index line-range.c: remove implicit dependency on the_index userdiff.c: remove implicit dependency on the_index rerere.c: remove implicit dependency on the_index sha1-file.c: remove implicit dependency on the_index patch-ids.c: remove implicit dependency on the_index merge.c: remove implicit dependency on the_index merge-blobs.c: remove implicit dependency on the_index ll-merge.c: remove implicit dependency on the_index diff-lib.c: remove implicit dependency on the_index read-cache.c: remove implicit dependency on the_index diff.c: remove implicit dependency on the_index grep.c: remove implicit dependency on the_index diff.c: remove the_index dependency in textconv() functions blame.c: rename "repo" argument to "r" combine-diff.c: remove implicit dependency on the_index ...
2018-10-16Merge branch 'ma/t1400-undebug-test'Libravatar Junio C Hamano1-1/+1
Test fix. * ma/t1400-undebug-test: t1400: drop debug `echo` to actually execute `test`
2018-10-16Merge branch 'en/status-multiple-renames-to-the-same-target-fix'Libravatar Junio C Hamano1-0/+23
The code in "git status" sometimes hit an assertion failure. This was caused by a structure that was reused without cleaning the data used for the first run, which has been corrected. * en/status-multiple-renames-to-the-same-target-fix: commit: fix erroneous BUG, 'multiple renames on the same target? how?'
2018-10-16Merge branch 'jn/gc-auto'Libravatar Junio C Hamano1-3/+3
"gc --auto" ended up calling exit(-1) upon error, which has been corrected to use exit(1). Also the error reporting behaviour when daemonized has been updated to exit with zero status when stopping due to a previously discovered error (which implies there is no point running gc to improve the situation); we used to exit with failure in such a case. * jn/gc-auto: gc: do not return error for prior errors in daemonized mode
2018-10-16Merge branch 'jn/gc-auto-prep'Libravatar Junio C Hamano1-1/+1
Code clean-up. * jn/gc-auto-prep: gc: exit with status 128 on failure gc: improve handling of errors reading gc.log
2018-10-16Merge branch 'md/test-cleanup'Libravatar Junio C Hamano38-399/+550
Various test scripts have been updated for style and also correct handling of exit status of various commands. * md/test-cleanup: tests: order arguments to git-rev-list properly t9109: don't swallow Git errors upstream of pipes tests: don't swallow Git errors upstream of pipes t/*: fix ordering of expected/observed arguments tests: standardize pipe placement Documentation: add shell guidelines t/README: reformat Do, Don't, Keep in mind lists
2018-10-16Merge branch 'tg/t5551-with-curl-7.61.1'Libravatar Junio C Hamano1-34/+34
Test update. * tg/t5551-with-curl-7.61.1: t5551: compare sorted cookies files t5551: move setup code inside test_expect blocks
2018-10-16Merge branch 'ts/alias-of-alias'Libravatar Junio C Hamano1-0/+40
An alias that expands to another alias has so far been forbidden, but now it is allowed to create such an alias. * ts/alias-of-alias: t0014: introduce an alias testing suite alias: show the call history when an alias is looping alias: add support for aliases of an alias
2018-10-16Merge branch 'ds/commit-graph-with-grafts'Libravatar Junio C Hamano2-2/+68
The recently introduced commit-graph auxiliary data is incompatible with mechanisms such as replace & grafts that "breaks" immutable nature of the object reference relationship. Disable optimizations based on its use (and updating existing commit-graph) when these incompatible features are in use in the repository. * ds/commit-graph-with-grafts: commit-graph: close_commit_graph before shallow walk commit-graph: not compatible with uninitialized repo commit-graph: not compatible with grafts commit-graph: not compatible with replace objects test-repository: properly init repo commit-graph: update design document refs.c: upgrade for_each_replace_ref to be a each_repo_ref_fn callback refs.c: migrate internal ref iteration to pass thru repository argument
2018-10-16Merge branch 'ab/commit-graph-progress'Libravatar Junio C Hamano1-0/+21
Generation of (experimental) commit-graph files have so far been fairly silent, even though it takes noticeable amount of time in a meaningfully large repository. The users will now see progress output. * ab/commit-graph-progress: gc: fix regression in 7b0f229222 impacting --quiet commit-graph verify: add progress output commit-graph write: add progress output
2018-10-16git-p4: fully support unshelving changelistsLibravatar Luke Diamand1-13/+56
The previous git-p4 unshelve support would check for changes in Perforce to the files being unshelved since the original shelve, and would complain if any were found. This was to ensure that the user wouldn't end up with both the shelved change delta, and some deltas from other changes in their git commit. e.g. given fileA: the quick brown fox change1: s/the/The/ <- p4 shelve this change change2: s/fox/Fox/ <- p4 submit this change git p4 unshelve 1 <- FAIL This change teaches the P4Unshelve class to always create a parent commit which matches the P4 tree (for the files being unshelved) at the point prior to the P4 shelve being created (which is reported in the p4 description for a shelved changelist). That then means git-p4 can always create a git commit matching the P4 shelve that was originally created, without any extra deltas. The user might still need to use the --origin option though - there is no way for git-p4 to work out the versions of all of the other *unchanged* files in the shelve, since this information is not recorded by Perforce. Additionally this fixes handling of shelved 'move' operations. Signed-off-by: Luke Diamand <luke@diamand.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-16git-p4: unshelve into refs/remotes/p4-unshelved, not refs/remotes/p4/unshelvedLibravatar Luke Diamand1-3/+3
The branch detection code looks for branches under refs/remotes/p4/... and can end up getting confused if there are unshelved changes in there as well. This happens in the function p4BranchesInGit(). Instead, put the unshelved changes into refs/remotes/p4-unshelved/<N>. Signed-off-by: Luke Diamand <luke@diamand.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-12log: fix coloring of certain octopus merge shapesLibravatar Noam Postavsky1-0/+102
For octopus merges where the first parent edge immediately merges into the next column to the left, the number of columns should be one less than the usual case. First parent to the left case: | *-. | |\ \ |/ / / The usual case: | *-. | |\ \ | | | * Also refactor the code to iterate over columns rather than dashes, building from an initial patch suggested by Jeff King. Signed-off-by: Noam Postavsky <npostavs@users.sourceforge.net> Reviewed-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-12split-index: smudge and add racily clean cache entries to split indexLibravatar SZEDER Gábor1-6/+2
Ever since the split index feature was introduced [1], refreshing a split index is prone to a variant of the classic racy git problem. Consider the following sequence of commands updating the split index when the shared index contains a racily clean cache entry, i.e. an entry whose cached stat data matches with the corresponding file in the worktree and the cached mtime matches that of the index: echo "cached content" >file git update-index --split-index --add file echo "dirty worktree" >file # size stays the same! # ... wait ... git update-index --add other-file Normally, when a non-split index is updated, then do_write_index() (the function responsible for writing all kinds of indexes, "regular", split, and shared) recognizes racily clean cache entries, and writes them with smudged stat data, i.e. with file size set to 0. When subsequent git commands read the index, they will notice that the smudged stat data doesn't match with the file in the worktree, and then go on to check the file's content and notice its dirtiness. In the above example, however, in the second 'git update-index' prepare_to_write_split_index() decides which cache entries stored only in the shared index should be replaced in the new split index. Alas, this function never looks out for racily clean cache entries, and since the file's stat data in the worktree hasn't changed since the shared index was written, it won't be replaced in the new split index. Consequently, do_write_index() doesn't even get this racily clean cache entry, and can't smudge its stat data. Subsequent git commands will then see that the index has more recent mtime than the file and that the (not smudged) cached stat data still matches with the file in the worktree, and, ultimately, will erroneously consider the file clean. Modify prepare_to_write_split_index() to recognize racily clean cache entries, and mark them to be added to the split index. Note that there are two places where it should check raciness: first those cache entries that are only stored in the shared index, and then those that have been copied by unpack_trees() from the shared index while it constructed a new index. This way do_write_index() will get these racily clean cache entries as well, and will then write them with smudged stat data to the new split index. This change makes all tests in 't1701-racy-split-index.sh' pass, so flip the two 'test_expect_failure' tests to success. Also add the '#' (as in nr. of trial) to those tests' description that were omitted when the tests expected failure. Note that after this change if the index is split when it contains a racily clean cache entry, then a smudged cache entry will be written both to the new shared and to the new split indexes. This doesn't affect regular git commands: as far as they are concerned this is just an entry in the split index replacing an outdated entry in the shared index. It did affect a few tests in 't1700-split-index.sh', though, because they actually check which entries are stored in the split index; a previous patch in this series has already made the necessary adjustments in 't1700'. And racily clean cache entries and index splitting are rare enough to not worry about the resulting duplicated smudged cache entries, and the additional complexity required to prevent them is not worth it. Several tests failed occasionally when the test suite was run with 'GIT_TEST_SPLIT_INDEX=yes'. Here are those that I managed to trace back to this racy split index problem, starting with those failing more frequently, with a link to a failing Travis CI build job for each. The highlighted line [2] shows when the racy file was written, which is not always in the failing test but in a preceeding setup test. t3903-stash.sh: https://travis-ci.org/git/git/jobs/385542084#L5858 t4024-diff-optimize-common.sh: https://travis-ci.org/git/git/jobs/386531969#L3174 t4015-diff-whitespace.sh: https://travis-ci.org/git/git/jobs/360797600#L8215 t2200-add-update.sh: https://travis-ci.org/git/git/jobs/382543426#L3051 t0090-cache-tree.sh: https://travis-ci.org/git/git/jobs/416583010#L3679 There might be others, e.g. perhaps 't1000-read-tree-m-3way.sh' and others using 'lib-read-tree-m-3way.sh', but I couldn't confirm yet. [1] In the branch leading to the merge commit v2.1.0-rc0~45 (Merge branch 'nd/split-index', 2014-07-16). [2] Note that those highlighted lines are in the 'after failure' fold, and your browser might unhelpfully fold it up before you could take a good look. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-12t1700-split-index: date back files to avoid racy situationsLibravatar SZEDER Gábor1-21/+28
't1700-split-index.sh' checks that the index was split correctly under various circumstances and that all the different ways to turn the split index feature on and off work correctly. To do so, most of its tests use 'test-tool dump-split-index' to see which files have their cache entries in the split index. All these tests assume that all cache entries are written to the shared index (called "base" throughout these tests) when a new shared index is created. This is an implementation detail: most git commands (basically all except 'git update-index') don't care or know at all about split index or whether a cache entry is stored in the split or shared index. As demonstrated in the previous patch, refreshing a split index is prone to a variant of the classic racy git issue. The next patch will fix this issue, but while doing so it will also slightly change this behaviour: only cache entries with mtime in the past will be written only to the newly created shared index, but racily clean cache entries will be written to the new split index (with smudged stat data). While this upcoming change won't at all affect any git commands, it will violate the above mentioned assumption of 't1700's tests. Since these tests create or modify files and create or refresh the split index in rapid succession, there are plenty of racily clean cache entries to be dealt with, which will then be written to the new split indexes, and, ultimately, will cause several tests in 't1700' to fail. Let's prepare 't1700-split-index.sh' for this upcoming change and modify its tests to avoid racily clean files by backdating the mtime of any file modifications (and since a lot of tests create or modify files, encapsulate it into a helper function). Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-12split-index: add tests to demonstrate the racy split index problemLibravatar SZEDER Gábor1-0/+218
Ever since the split index feature was introduced [1], refreshing a split index is prone to a variant of the classic racy git problem. There are a couple of unrelated tests in the test suite that occasionally fail when run with 'GIT_TEST_SPLIT_INDEX=yes', but 't1700-split-index.sh', the only test script focusing solely on split index, has never noticed this issue, because it only cares about how the index is split under various circumstances and all the different ways to turn the split index feature on and off. Add a dedicated test script 't1701-racy-split-index.sh' to exercise the split index feature in racy situations as well; kind of a "t0010-racy-git.sh for split index" but with modern style (the tests do everything in &&-chained list of commands in 'test_expect_...' blocks, and use 'test_cmp' for more informative output on failure). The tests cover the following sequences of index splitting, updating, and racy file modifications, with the last two cases demonstrating the racy split index problem: 1. Split the index while adding a racily clean file: echo "cached content" >file git update-index --split-index --add file echo "dirty worktree" >file # size stays the same This case already works properly. Even though the cache entry's stat data matches with the modifid file in the worktree, subsequent git commands will notice that the (split) index and the file have the same mtime, and then will go on to check the file's content and notice its dirtiness. 2. Add a racily clean file to an already split index: git update-index --split-index echo "cached content" >file git update-index --add file echo "dirty worktree" >file This case already works properly. After the second 'git update-index' writes the newly added file's cache entry to the new split index, it basically works in the same way as case #1. 3. Split the index when it (i.e. the not yet splitted index) contains a racily clean cache entry, i.e. an entry whose cached stat data matches with the corresponding file in the worktree and the cached mtime matches that of the index: echo "cached content" >file git update-index --add file echo "dirty worktree" >file # ... wait ... git update-index --split-index --add other-file This case already works properly. The shared index is written by do_write_index(), i.e. the same function that is responsible for writing "regular" and split indexes as well. This function cleverly notices the racily clean cache entry, and writes the entry to the new shared index with smudged stat data, i.e. file size set to 0. When subsequent git commands read the index, they will notice that the smudged stat data doesn't match with the file in the worktree, and then go on to check the file's content and notice its dirtiness. 4. Update the split index when it contains a racily clean cache entry: git update-index --split-index echo "cached content" >file git update-index --add file echo "dirty worktree" >file # ... wait ... git update-index --add other-file This case already works properly. After the second 'git update-index' the newly added file's cache entry is only stored in the split index. If a cache entry is present in the split index (even if it is a replacement of an outdated entry in the shared index), then it will always be included in the new split index on subsequent split index updates (until the file is removed or a new shared index is written), independently from whether the entry is racily clean or not. When do_write_index() writes the new split index, it notices the racily clean cache entry, and smudges its stat date. Subsequent git commands reading the index will notice the smudged stat data and then go on to check the file's content and notice its dirtiness. 5. Update the split index when a racily clean cache entry is stored only in the shared index: echo "cached content" >file git update-index --split-index --add file echo "dirty worktree" >file # ... wait ... git update-index --add other-file This case fails due to the racy split index problem. In the second 'git update-index' prepare_to_write_split_index() decides, among other things, which cache entries stored only in the shared index should be replaced in the new split index. Alas, this function never looks out for racily clean cache entries, and since the file's stat data in the worktree hasn't changed since the shared index was written, the entry won't be replaced in the new split index. Consequently, do_write_index() doesn't even get this racily clean cache entry, and can't smudge its stat data. Subsequent git commands will then see that the index has more recent mtime than the file and that the (not smudged) cached stat data still matches with the file in the worktree, and, ultimately, will erroneously consider the file clean. 6. Update the split index after unpack_trees() copied a racily clean cache entry from the shared index: echo "cached content" >file git update-index --split-index --add file echo "dirty worktree" >file # ... wait ... git read-tree -m HEAD This case fails due to the racy split index problem. This basically fails for the same reason as case #5 above, but there is one important difference, which warrants the dedicated test. While that second 'git update-index' in case #5 updates index_state in place, in this case 'git read-tree -m' calls unpack_trees(), which throws out the entire index, and constructs a new one from the (potentially updated) copies of the original's cache entries. Consequently, when prepare_to_write_split_index() gets to work on this reconstructed index, it takes a different code path than in case #5 when deciding which cache entries in the shared index should be replaced. The result is the same, though: the racily clean cache entry goes unnoticed, it isn't added to the split index with smudged stat data, and subsequent git commands will then erroneously consider the file clean. Note that in the last two 'test_expect_failure' cases I omitted the '#' (as in nr. of trial) from the tests' description on purpose for now, as it breakes the TAP output [2]; it will be added at the end of the series, when those two tests will be flipped to 'test_expect_success'. [1] In the branch leading to the merge commit v2.1.0-rc0~45 (Merge branch 'nd/split-index', 2014-07-16). [2] In the TAP output a '#' should separate the test's description from the TODO directive emitted by 'test_expect_failure'. The additional '#' in "#$trial" interferes with this, the test harness won't recognize the TODO directive, and will report that those tests failed unexpectedly. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-11config: add new index.threads config settingLibravatar Ben Peart2-0/+10
Add support for a new index.threads config setting which will be used to control the threading code in do_read_index(). A value of 0 will tell the index code to automatically determine the correct number of threads to use. A value of 1 will make the code single threaded. A value greater than 1 will set the maximum number of threads to use. For testing purposes, this setting can be overwritten by setting the GIT_TEST_INDEX_THREADS=<n> environment variable to a value greater than 0. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-11eoie: add End of Index Entry (EOIE) extensionLibravatar Ben Peart1-4/+4
The End of Index Entry (EOIE) is used to locate the end of the variable length index entries and the beginning of the extensions. Code can take advantage of this to quickly locate the index extensions without having to parse through all of the index entries. The EOIE extension is always written out to the index file including to the shared index when using the split index feature. Because it is always written out, the SHA checksums in t/t1700-split-index.sh were updated to reflect its inclusion. It is written as an optional extension to ensure compatibility with other git implementations that do not yet support it. It is always written out to ensure it is available as often as possible to speed up index operations. Because it must be able to be loaded before the variable length cache entries and other index extensions, this extension must be written last. The signature for this extension is { 'E', 'O', 'I', 'E' }. The extension consists of: - 32-bit offset to the end of the index entries - 160-bit SHA-1 over the extension types and their sizes (but not their contents). E.g. if we have "TREE" extension that is N-bytes long, "REUC" extension that is M-bytes long, followed by "EOIE", then the hash would be: SHA-1("TREE" + <binary representation of N> + "REUC" + <binary representation of M>) Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-10Merge branch 'ab/fsck-skiplist'Libravatar Junio C Hamano3-4/+121
Update fsck.skipList implementation and documentation. * ab/fsck-skiplist: fsck: support comments & empty lines in skipList fsck: use oidset instead of oid_array for skipList fsck: use strbuf_getline() to read skiplist file fsck: add a performance test for skipList fsck: add a performance test fsck: document that skipList input must be unabbreviated fsck: document and test commented & empty line skipList input fsck: document and test sorted skipList input fsck tests: add a test for no skipList input fsck tests: setup of bogus commit object
2018-10-10Merge branch 'ds/multi-pack-verify'Libravatar Junio C Hamano1-1/+135
"git multi-pack-index" learned to detect corruption in the .midx file it uses, and this feature has been integrated into "git fsck". * ds/multi-pack-verify: fsck: verify multi-pack-index multi-pack-index: report progress during 'verify' multi-pack-index: verify object offsets multi-pack-index: fix 32-bit vs 64-bit size check multi-pack-index: verify oid lookup order multi-pack-index: verify oid fanout order multi-pack-index: verify missing pack multi-pack-index: verify packname order multi-pack-index: verify corrupt chunk lookup table multi-pack-index: verify bad header multi-pack-index: add 'verify' verb
2018-10-10Merge branch 'bc/hash-independent-tests'Libravatar Junio C Hamano14-113/+350
Various tests have been updated to make it easier to swap the hash function used for object identification. * bc/hash-independent-tests: t5318: use test_oid for HASH_LEN t1407: make hash size independent t1406: make hash-size independent t1405: make hash size independent t1400: switch hard-coded object ID to variable t1006: make hash size independent t0064: make hash size independent t0002: abstract away SHA-1 specific constants t0000: update tests for SHA-256 t0000: use hash translation table t: add test functions to translate hash-related values
2018-10-10Merge branch 'nd/test-tool'Libravatar Junio C Hamano12-109/+121
Test helper binaries clean-up. * nd/test-tool: Makefile: add a hint about TEST_BUILTINS_OBJS t/helper: merge test-dump-fsmonitor into test-tool t/helper: merge test-parse-options into test-tool t/helper: merge test-pkt-line into test-tool t/helper: merge test-dump-untracked-cache into test-tool t/helper: keep test-tool command list sorted
2018-10-10cache-tree: skip some blob checks in partial cloneLibravatar Jonathan Tan1-0/+33
In a partial clone, whenever a sparse checkout occurs, the existence of all blobs in the index is verified, whether they are included or excluded by the .git/info/sparse-checkout specification. This significantly degrades performance because a lazy fetch occurs whenever the existence of a missing blob is checked. This is because cache_tree_update() checks the existence of all objects in the index, whether or not CE_SKIP_WORKTREE is set on them. Teach cache_tree_update() to skip checking CE_SKIP_WORKTREE objects when the repository is a partial clone. This improves performance for sparse checkout and also other operations that use cache_tree_update(). Instead of completely removing the check, an argument could be made that the check should instead be replaced by a check that the blob is promised, but for performance reasons, I decided not to do this. If the user needs to verify the repository, it can be done using fsck (which will notify if a tree points to a missing and non-promised blob, whether the blob is included or excluded by the sparse-checkout specification). Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-09transport.c: introduce core.alternateRefsPrefixesLibravatar Taylor Blau1-0/+8
The recently-introduced "core.alternateRefsCommand" allows callers to specify with high flexibility the tips that they wish to advertise from alternates. This flexibility comes at the cost of some inconvenience when the caller only wishes to limit the advertisement to one or more prefixes. For example, to advertise only tags, a caller using 'core.alternateRefsCommand' would have to do: $ git config core.alternateRefsCommand ' \ f() { git -C "$1" for-each-ref \ refs/tags --format="%(objectname)" }; f "$@"' The above is cumbersome to write, so let's introduce a "core.alternateRefsPrefixes" to address this common case. Instead, the caller can run: $ git config core.alternateRefsPrefixes 'refs/tags' Which will behave identically to the longer example using "core.alternateRefsCommand". Since the value of "core.alternateRefsPrefixes" is appended to 'git for-each-ref' and then executed, include a "--" before taking the configured value to avoid misinterpreting arguments as flags to 'git for-each-ref'. In the case that the caller wishes to specify multiple prefixes, they may separate them by whitespace. If "core.alternateRefsCommand" is set, it will take precedence over "core.alternateRefsPrefixes". Signed-off-by: Taylor Blau <me@ttaylorr.com> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-09transport.c: introduce core.alternateRefsCommandLibravatar Taylor Blau1-0/+33
When in a repository containing one or more alternates, Git would sometimes like to list references from those alternates. For example, 'git receive-pack' lists the "tips" pointed to by references in those alternates as special ".have" references. Listing ".have" references is designed to make pushing changes from upstream to a fork a lightweight operation, by advertising to the pusher that the fork already has the objects (via its alternate). Thus, the client can avoid sending them. However, when the alternate (upstream, in the previous example) has a pathologically large number of references, the initial advertisement is too expensive. In fact, it can dominate any such optimization where the pusher avoids sending certain objects. Introduce "core.alternateRefsCommand" in order to provide a facility to limit or filter alternate references. This can be used, for example, to filter out references the alternate does not wish to send (for space concerns, or otherwise) during the initial advertisement. Let the repository that has alternates configure this command to avoid trusting the alternate to provide us a safe command to run in the shell. To find the alternate, pass its absolute path as the first argument. Signed-off-by: Taylor Blau <me@ttaylorr.com> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-07fetch: do not list refs if fetching only hashesLibravatar Jonathan Tan2-0/+28
If only hash literals are given on a "git fetch" command-line, tag following is not requested, and the fetch is done using protocol v2, a list of refs is not required from the remote. Therefore, optimize by invoking transport_get_remote_refs() only if we need the refs. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-07transport: do not list refs if possibleLibravatar Jonathan Tan1-0/+4
When all refs to be fetched are exact OIDs, it is possible to perform a fetch without requiring the remote to list refs if protocol v2 is used. Teach Git to do this. This currently has an effect only for lazy fetches done from partial clones. The change necessary to likewise optimize "git fetch <remote> <sha-1>" will be done in a subsequent patch. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-07tests: order arguments to git-rev-list properlyLibravatar Matthew DeVore3-29/+44
It is a common mistake to put positional arguments before flags when invoking git-rev-list. Order the positional arguments last. This patch skips git-rev-list invocations which include the --not flag, since the ordering of flags and positional arguments affects the behavior. This patch also skips invocations of git-rev-list that occur in command substitution in which the exit code is discarded, since fixing those properly will require a more involved cleanup. Signed-off-by: Matthew DeVore <matvore@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-07t9109: don't swallow Git errors upstream of pipesLibravatar Matthew DeVore1-13/+21
'git ... | foo' will mask any errors or crashes in git, so split up such pipes in this file. One testcase uses several separate pipe sequences in a row which are awkward to split up. Wrap the split-up pipe in a function so the awkwardness is not repeated. Also change that testcase's surrounding quotes from double to single to avoid premature string interpolation. Signed-off-by: Matthew DeVore <matvore@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-07tests: don't swallow Git errors upstream of pipesLibravatar Matthew DeVore3-132/+141
Some pipes in tests lose the exit code of git processes, which can mask unexpected behavior like crashes. Split these pipes up so that git commands are only at the end of pipes rather than the beginning or middle. The violations fixed in this patch were found in the process of fixing pipe placement in a prior patch. Signed-off-by: Matthew DeVore <matvore@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-07t/*: fix ordering of expected/observed argumentsLibravatar Matthew DeVore32-112/+112
Fix various places where the ordering was obviously wrong, meaning it was easy to find with grep. Signed-off-by: Matthew DeVore <matvore@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-07tests: standardize pipe placementLibravatar Matthew DeVore7-250/+344
Instead of using a line-continuation and pipe on the second line, take advantage of the shell's implicit line continuation after a pipe character. So for example, instead of some long line \ | next line use some long line | next line And add a blank line before and after the pipe where it aids readability (it usually does). This better matches the coding style documented in Documentation/CodingGuidelines and used in shell scripts elsewhere in the tree. Signed-off-by: Matthew DeVore <matvore@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-07Documentation: add shell guidelinesLibravatar Matthew DeVore1-0/+27
Add the following guideline to Documentation/CodingGuidelines: Break overlong lines after "&&", "||", and "|", not before them; that way the command can continue to subsequent lines without backslash at the end. And the following to t/README (since it is specific to writing tests): Pipes and $(git ...) should be avoided when they swallow exit codes of Git processes Signed-off-by: Matthew DeVore <matvore@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-07t/README: reformat Do, Don't, Keep in mind listsLibravatar Matthew DeVore1-22/+20
The list of Don'ts for test writing has grown large such that it is hard to see at a glance which section an item is in. In other words, if I ignore a little bit of surrounding context, the "don'ts" look like "do's." To make the list more readable, prefix "Don't" in front of every first sentence in the items. Also, the "Keep in mind" list is out of place and awkward, because it was a very short "list" beneath two very long ones, and it seemed easy to miss under the list of "don'ts," and it only had one item. So move this item to the list of "do's" and phrase as "Remember..." Signed-off-by: Matthew DeVore <matvore@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>