summaryrefslogtreecommitdiff
path: root/t/t9300-fast-import.sh
AgeCommit message (Collapse)AuthorFilesLines
2019-12-06Sync with 2.23.1Libravatar Johannes Schindelin1-14/+44
* maint-2.23: (44 commits) Git 2.23.1 Git 2.22.2 Git 2.21.1 mingw: sh arguments need quoting in more circumstances mingw: fix quoting of empty arguments for `sh` mingw: use MSYS2 quoting even when spawning shell scripts mingw: detect when MSYS2's sh is to be spawned more robustly t7415: drop v2.20.x-specific work-around Git 2.20.2 t7415: adjust test for dubiously-nested submodule gitdirs for v2.20.x Git 2.19.3 Git 2.18.2 Git 2.17.3 Git 2.16.6 test-drop-caches: use `has_dos_drive_prefix()` Git 2.15.4 Git 2.14.6 mingw: handle `subst`-ed "DOS drives" mingw: refuse to access paths with trailing spaces or periods mingw: refuse to access paths with illegal characters ...
2019-12-06Sync with 2.22.2Libravatar Johannes Schindelin1-14/+44
* maint-2.22: (43 commits) Git 2.22.2 Git 2.21.1 mingw: sh arguments need quoting in more circumstances mingw: fix quoting of empty arguments for `sh` mingw: use MSYS2 quoting even when spawning shell scripts mingw: detect when MSYS2's sh is to be spawned more robustly t7415: drop v2.20.x-specific work-around Git 2.20.2 t7415: adjust test for dubiously-nested submodule gitdirs for v2.20.x Git 2.19.3 Git 2.18.2 Git 2.17.3 Git 2.16.6 test-drop-caches: use `has_dos_drive_prefix()` Git 2.15.4 Git 2.14.6 mingw: handle `subst`-ed "DOS drives" mingw: refuse to access paths with trailing spaces or periods mingw: refuse to access paths with illegal characters unpack-trees: let merged_entry() pass through do_add_entry()'s errors ...
2019-12-06Sync with 2.21.1Libravatar Johannes Schindelin1-14/+44
* maint-2.21: (42 commits) Git 2.21.1 mingw: sh arguments need quoting in more circumstances mingw: fix quoting of empty arguments for `sh` mingw: use MSYS2 quoting even when spawning shell scripts mingw: detect when MSYS2's sh is to be spawned more robustly t7415: drop v2.20.x-specific work-around Git 2.20.2 t7415: adjust test for dubiously-nested submodule gitdirs for v2.20.x Git 2.19.3 Git 2.18.2 Git 2.17.3 Git 2.16.6 test-drop-caches: use `has_dos_drive_prefix()` Git 2.15.4 Git 2.14.6 mingw: handle `subst`-ed "DOS drives" mingw: refuse to access paths with trailing spaces or periods mingw: refuse to access paths with illegal characters unpack-trees: let merged_entry() pass through do_add_entry()'s errors quote-stress-test: offer to test quoting arguments for MSYS2 sh ...
2019-12-06Sync with 2.19.3Libravatar Johannes Schindelin1-14/+44
* maint-2.19: (34 commits) Git 2.19.3 Git 2.18.2 Git 2.17.3 Git 2.16.6 test-drop-caches: use `has_dos_drive_prefix()` Git 2.15.4 Git 2.14.6 mingw: handle `subst`-ed "DOS drives" mingw: refuse to access paths with trailing spaces or periods mingw: refuse to access paths with illegal characters unpack-trees: let merged_entry() pass through do_add_entry()'s errors quote-stress-test: offer to test quoting arguments for MSYS2 sh t6130/t9350: prepare for stringent Win32 path validation quote-stress-test: allow skipping some trials quote-stress-test: accept arguments to test via the command-line tests: add a helper to stress test argument quoting mingw: fix quoting of arguments Disallow dubiously-nested submodule git directories protect_ntfs: turn on NTFS protection by default path: also guard `.gitmodules` against NTFS Alternate Data Streams ...
2019-12-06Sync with 2.18.2Libravatar Johannes Schindelin1-14/+44
* maint-2.18: (33 commits) Git 2.18.2 Git 2.17.3 Git 2.16.6 test-drop-caches: use `has_dos_drive_prefix()` Git 2.15.4 Git 2.14.6 mingw: handle `subst`-ed "DOS drives" mingw: refuse to access paths with trailing spaces or periods mingw: refuse to access paths with illegal characters unpack-trees: let merged_entry() pass through do_add_entry()'s errors quote-stress-test: offer to test quoting arguments for MSYS2 sh t6130/t9350: prepare for stringent Win32 path validation quote-stress-test: allow skipping some trials quote-stress-test: accept arguments to test via the command-line tests: add a helper to stress test argument quoting mingw: fix quoting of arguments Disallow dubiously-nested submodule git directories protect_ntfs: turn on NTFS protection by default path: also guard `.gitmodules` against NTFS Alternate Data Streams is_ntfs_dotgit(): speed it up ...
2019-12-06Sync with 2.17.3Libravatar Johannes Schindelin1-14/+44
* maint-2.17: (32 commits) Git 2.17.3 Git 2.16.6 test-drop-caches: use `has_dos_drive_prefix()` Git 2.15.4 Git 2.14.6 mingw: handle `subst`-ed "DOS drives" mingw: refuse to access paths with trailing spaces or periods mingw: refuse to access paths with illegal characters unpack-trees: let merged_entry() pass through do_add_entry()'s errors quote-stress-test: offer to test quoting arguments for MSYS2 sh t6130/t9350: prepare for stringent Win32 path validation quote-stress-test: allow skipping some trials quote-stress-test: accept arguments to test via the command-line tests: add a helper to stress test argument quoting mingw: fix quoting of arguments Disallow dubiously-nested submodule git directories protect_ntfs: turn on NTFS protection by default path: also guard `.gitmodules` against NTFS Alternate Data Streams is_ntfs_dotgit(): speed it up mingw: disallow backslash characters in tree objects' file names ...
2019-12-06Sync with 2.15.4Libravatar Johannes Schindelin1-14/+44
* maint-2.15: (29 commits) Git 2.15.4 Git 2.14.6 mingw: handle `subst`-ed "DOS drives" mingw: refuse to access paths with trailing spaces or periods mingw: refuse to access paths with illegal characters unpack-trees: let merged_entry() pass through do_add_entry()'s errors quote-stress-test: offer to test quoting arguments for MSYS2 sh t6130/t9350: prepare for stringent Win32 path validation quote-stress-test: allow skipping some trials quote-stress-test: accept arguments to test via the command-line tests: add a helper to stress test argument quoting mingw: fix quoting of arguments Disallow dubiously-nested submodule git directories protect_ntfs: turn on NTFS protection by default path: also guard `.gitmodules` against NTFS Alternate Data Streams is_ntfs_dotgit(): speed it up mingw: disallow backslash characters in tree objects' file names path: safeguard `.git` against NTFS Alternate Streams Accesses clone --recurse-submodules: prevent name squatting on Windows is_ntfs_dotgit(): only verify the leading segment ...
2019-12-04fast-import: disallow "feature import-marks" by defaultLibravatar Jeff King1-5/+17
As with export-marks in the previous commit, import-marks can access the filesystem. This is significantly less dangerous than export-marks because it only involves reading from arbitrary paths, rather than writing them. However, it could still be surprising and have security implications (e.g., exfiltrating data from a service that accepts fast-import streams). Let's lump it (and its "if-exists" counterpart) in with export-marks, and enable the in-stream version only if --allow-unsafe-features is set. Signed-off-by: Jeff King <peff@peff.net>
2019-12-04fast-import: disallow "feature export-marks" by defaultLibravatar Jeff King1-8/+15
The fast-import stream command "feature export-marks=<path>" lets the stream write marks to an arbitrary path. This may be surprising if you are running fast-import against an untrusted input (which otherwise cannot do anything except update Git objects and refs). Let's disallow the use of this feature by default, and provide a command-line option to re-enable it (you can always just use the command-line --export-marks as well, but the in-stream version provides an easy way for exporters to control the process). This is a backwards-incompatible change, since the default is flipping to the new, safer behavior. However, since the main users of the in-stream versions would be import/export-based remote helpers, and since we trust remote helpers already (which are already running arbitrary code), we'll pass the new option by default when reading a remote helper's stream. This should minimize the impact. Note that the implementation isn't totally simple, as we have to work around the fact that fast-import doesn't parse its command-line options until after it has read any "feature" lines from the stream. This is how it lets command-line options override in-stream. But in our case, it's important to parse the new --allow-unsafe-features first. There are three options for resolving this: 1. Do a separate "early" pass over the options. This is easy for us to do because there are no command-line options that allow the "unstuck" form (so there's no chance of us mistaking an argument for an option), though it does introduce a risk of incorrect parsing later (e.g,. if we convert to parse-options). 2. Move the option parsing phase back to the start of the program, but teach the stream-reading code never to override an existing value. This is tricky, because stream "feature" lines override each other (meaning we'd have to start tracking the source for every option). 3. Accept that we might parse a "feature export-marks" line that is forbidden, as long we don't _act_ on it until after we've parsed the command line options. This would, in fact, work with the current code, but only because the previous patch fixed the export-marks parser to avoid touching the filesystem. So while it works, it does carry risk of somebody getting it wrong in the future in a rather subtle and unsafe way. I've gone with option (1) here as simple, safe, and unlikely to cause regressions. This fixes CVE-2019-1348. Signed-off-by: Jeff King <peff@peff.net>
2019-12-04fast-import: delay creating leading directories for export-marksLibravatar Jeff King1-2/+11
When we parse the --export-marks option, we don't immediately open the file, but we do create any leading directories. This can be especially confusing when a command-line option overrides an in-stream one, in which case we'd create the leading directory for the in-stream file, even though we never actually write the file. Let's instead create the directories just before opening the file, which means we'll create only useful directories. Note that this could change the handling of relative paths if we chdir() in between, but we don't actually do so; the only permanent chdir is from setup_git_directory() which runs before either code path (potentially we should take the pre-setup dir into account to avoid surprising the user, but that's an orthogonal change). The test just adapts the existing "override" test to use paths with leading directories. This checks both that the correct directory is created (which worked before but was not tested), and that the overridden one is not (our new fix here). While we're here, let's also check the error result of safe_create_leading_directories(). We'd presumably notice any failure immediately after when we try to open the file itself, but we can give a more specific error message in this case. Signed-off-by: Jeff King <peff@peff.net>
2019-12-04t9300: create marks files for double-import-marks testLibravatar Jeff King1-0/+2
Our tests confirm that providing two "import-marks" options in a fast-import stream is an error. However, the invoked command would fail even without covering this case, because the marks files themselves do not actually exist. Let's create the files to make sure we fail for the right reason (we actually do, because the option parsing happens before we open anything, but this future-proofs our test). Signed-off-by: Jeff King <peff@peff.net>
2019-12-04t9300: drop some useless uses of catLibravatar Jeff King1-5/+5
These waste a process, and make the line longer than it needs to be. Signed-off-by: Jeff King <peff@peff.net>
2019-10-15Merge branch 'en/fast-imexport-nested-tags'Libravatar Junio C Hamano1-0/+37
Updates to fast-import/export. * en/fast-imexport-nested-tags: fast-export: handle nested tags t9350: add tests for tags of things other than a commit fast-export: allow user to request tags be marked with --mark-tags fast-export: add support for --import-marks-if-exists fast-import: add support for new 'alias' command fast-import: allow tags to be identified by mark labels fast-import: fix handling of deleted tags fast-export: fix exporting a tag and nothing else
2019-10-07tests: remove "cat foo" before "test_i18ngrep bar foo"Libravatar René Scharfe1-13/+0
Some tests print a file before searching for a pattern using test_i18ngrep. This is useful when debugging tests with --verbose when the pattern is not found as expected. Since 63b1a175ee (t: make 'test_i18ngrep' more informative on failure, 2018-02-08) test_i18ngrep already shows the contents of a file that doesn't match the expected pattern, though. So don't bother doing the same unconditionally up-front. The contents are not interesting if the expected pattern is found, and showing it twice if it doesn't match is of no use. Signed-off-by: René Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-04fast-import: add support for new 'alias' commandLibravatar Elijah Newren1-0/+5
fast-export and fast-import have nice --import-marks flags which allow for incremental migrations. However, if there is a mark in fast-export's file of marks without a corresponding mark in the one for fast-import, then we run the risk that fast-export tries to send new objects relative to the mark it knows which fast-import does not, causing fast-import to fail. This arises in practice when there is a filter of some sort running between the fast-export and fast-import processes which prunes some commits programmatically. Provide such a filter with the ability to alias pruned commits to their most recent non-pruned ancestor. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-04fast-import: allow tags to be identified by mark labelsLibravatar Elijah Newren1-0/+19
Mark identifiers are used in fast-export and fast-import to provide a label to refer to earlier content. Blobs are given labels because they need to be referenced in the commits where they first appear with a given filename, and commits are given labels because they can be the parents of other commits. Tags were never given labels, probably because they were viewed as unnecessary, but that presents two problems: 1. It leaves us without a way of referring to previous tags if we want to create a tag of a tag (or higher nestings). 2. It leaves us with no way of recording that a tag has already been imported when using --export-marks and --import-marks. Fix these problems by allowing an optional mark label for tags. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-04fast-import: fix handling of deleted tagsLibravatar Elijah Newren1-0/+13
If our input stream includes a tag which is later deleted, we were not properly deleting it. We did have a step which would delete it, but we left a tag in the tag list noting that it needed to be updated, and the updating of annotated tags occurred AFTER ref deletion. So, when we record that a tag needs to be deleted, also remove it from the list of annotated tags to update. While this has likely been something that has not happened in practice, it will come up more in order to support nested tags. For nested tags, we either need to give temporary names to the intermediate tags and then delete them, or else we need to use the final name for the intermediate tags. If we use the final name for the intermediate tags, then in order to keep the sanity check that someone doesn't try to update the same tag twice, we need to delete the ref after creating the intermediate tag. So, either way nested tags imply the need to delete temporary inner tag references. Helped-by: René Scharfe <l.s.r@web.de> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-05-14fast-import: support 'encoding' commit headerLibravatar Elijah Newren1-0/+20
Since git supports commit messages with an encoding other than UTF-8, allow fast-import to import such commits. This may be useful for folks who do not want to reencode commit messages from an external system, and may also be useful to achieve reversible history rewrites (e.g. sha1sum <-> sha256sum transitions or subtree work) with git repositories that have used specialized encodings in their commit history. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-01fast-import: fix erroneous handling of get-mark with empty orphan commitsLibravatar Elijah Newren1-2/+2
When get-mark was introduced in commit 28c7b1f7b7b7 ("fast-import: add a get-mark command", 2015-07-01), it followed the precedent of the cat-blob command to be allowed on any line other than in the middle of a data directive; see commit 777f80d7429b ("fast-import: Allow cat-blob requests at arbitrary points in stream", 2010-11-28). It was useful to allow cat-blob directives in the middle of a commit to get more data that would be used in writing the current commit object. get-mark is not similarly useful since fast-import can already use either object id or mark. Further, trying to allow this command anywhere caused parsing bugs. Fix the parsing problems by only allowing get-mark commands to appear when other commands have completed. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-01t9300: demonstrate bug with get-mark and empty orphan commitsLibravatar Elijah Newren1-0/+37
Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-22multi-pack-index: define GIT_TEST_MULTI_PACK_INDEXLibravatar Derrick Stolee1-1/+1
The multi-pack-index feature is tested in isolation by t5319-multi-pack-index.sh, but there are many more interesting scenarios in the test suite surrounding pack-file data shapes and interactions. Since the multi-pack-index is an optional data structure, it does not make sense to include it by default in those tests. Instead, add a new GIT_TEST_MULTI_PACK_INDEX environment variable that enables core.multiPackIndex and writes a multi-pack-index after each 'git repack' command. This adds extra test coverage when needed. There are a few spots in the test suite that need to react to this change: * t5319-multi-pack-index.sh: there is a test that checks that 'git repack' deletes the multi-pack-index. Disable the environment variable to ensure this still happens. * t5310-pack-bitmaps.sh: One test moves a pack-file from the object directory to an alternate. This breaks the multi-pack-index, so delete the multi-pack-index at this point, if it exists. * t9300-fast-import.sh: One test verifies the number of files in the .git/objects/pack directory is exactly 8. Exclude the multi-pack-index from this count so it is still 8 in all cases. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-08-20Merge branch 'ab/test-must-be-empty-for-master'Libravatar Junio C Hamano1-5/+2
Test updates. * ab/test-must-be-empty-for-master: tests: make use of the test_must_be_empty function
2018-08-15Merge branch 'sg/fast-import-dump-refs-on-checkpoint-fix'Libravatar Junio C Hamano1-1/+4
Test update. * sg/fast-import-dump-refs-on-checkpoint-fix: t9300: wait for background fast-import process to die after killing it
2018-07-30tests: make use of the test_must_be_empty functionLibravatar Ævar Arnfjörð Bjarmason1-5/+2
Change various tests that use an idiom of the form: >expect && test_cmp expect actual To instead use: test_must_be_empty actual The test_must_be_empty() wrapper was introduced in ca8d148daf ("test: test_must_be_empty helper", 2013-06-09). Many of these tests have been added after that time. This was mostly found with, and manually pruned from: git grep '^\s+>.*expect.* &&$' t Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-20t9300: wait for background fast-import process to die after killing itLibravatar SZEDER Gábor1-1/+4
The five new tests added to 't9300-fast-import.sh' in 30e215a65c (fast-import: checkpoint: dump branches/tags/marks even if object_count==0, 2017-09-28), all with the prefix "V:" in their test description, run 'git fast-import' in the background and then 'kill' it as part of a 'test_when_finished' cleanup command. When this test script is executed with Bash, some or even all of these tests tend to pollute the test script's stderr, and messages about terminated processes end up on the terminal: $ bash ./t9300-fast-import.sh <... snip ...> ok 179 - V: checkpoint helper does not get stuck with extra output /<...>/test-lib-functions.sh: line 388: 28383 Terminated git fast-import $options 0<&8 1>&9 ok 180 - V: checkpoint updates refs after reset ./t9300-fast-import.sh: line 3210: 28401 Terminated git fast-import $options 0<&8 1>&9 ok 181 - V: checkpoint updates refs and marks after commit ok 182 - V: checkpoint updates refs and marks after commit (no new objects) ./test-lib.sh: line 634: line 3250: 28485 Terminated git fast-import $options 0<&8 1>&9 ok 183 - V: checkpoint updates tags after tag ./t9300-fast-import.sh: line 3264: 28510 Terminated git fast-import $options 0<&8 1>&9 After a background child process terminates, its parent Bash process always outputs a message like those above to stderr, even when in non-interactive mode. But how do some of these messages end up on the test script's stderr, why don't we get them from all five tests, and why do they come from different file/line locations? Well, after sending the TERM signal to the background child process, it takes a little while until that process receives the signal and terminates, and then it takes another while until the parent process notices it. During this time the parent Bash process is continuing execution, and by the time it notices that its child terminated it might have already left 'test_eval_inner_' and its stderr is not redirected to /dev/null anymore. That's why such a message can appear on the test script's stderr, while other times, when the child terminates fast and/or the parent shell is slow enough, the message ends up in /dev/null, just like any other output of the test does. Bash always adds the file name and line number of the code location it was about to execute when it notices the termination of its child process as a prefix to that message, hence the varying and sometimes totally unrelated location prefixes in those messages (e.g. line 388 in 'test-lib-functions.sh' is 'test_verify_prereq', and I saw such a message pointing to 'say_color' as well). Prevent these messages from appearing on the test script's stderr by 'wait'-ing on the pid of the background 'git fast-import' process after sending it the TERM signal. This ensures that the executing shell's stderr is still redirected when the shell notices the termination of its child process in the background, and that these messages get a consistent file/line location prefix. Note that this is not an issue when the test script is run with Bash and '-v', because then these messages are supposed to go to the test script's stderr anyway, and indeed all of them do; though the sometimes seemingly random file/line prefixes could be confusing still. Similarly, it's not an issue with Bash and '--verbose-log' either, because then all messages go to the log file as they should. Finally, it's not an issue with some other shells (I tried dash, ksh, ksh93 and mksh) even without any of the verbose options, because they don't print messages like these in non-interactive mode in the first place. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-05-14t: switch $_x40 to $OID_REGEXLibravatar brian m. carlson1-3/+3
Switch all uses of $_x40 to $OID_REGEX so that they work correctly with larger hashes. This commit was created by using the following sed command to modify all files in the t directory except t/test-lib.sh: sed -i 's/\$_x40/$OID_REGEX/g' Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-27t/helper: merge test-genrandom into test-toolLibravatar Nguyễn Thái Ngọc Duy1-1/+1
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-12-04print_sha1_ellipsis: introduce helperLibravatar Ann T Ropea1-1/+1
Introduce a helper print_sha1_ellipsis() that pays attention to the GIT_PRINT_SHA1_ELLIPSIS environment variable, and prepare the tests to unconditionally set it for the test pieces that will be broken once the code stops showing the extra dots by default. The removal of these dots is merely a plan at this step and has not happened yet but soon will. Document GIT_PRINT_SHA1_ELLIPSIS. Signed-off-by: Ann T Ropea <bedhanger@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-29fast-import: checkpoint: dump branches/tags/marks even if object_count==0Libravatar Eric Rannaud1-0/+142
The checkpoint command cycles packfiles if object_count != 0, a sensible test or there would be no pack files to write. Since 820b931012, the command also dumps branches, tags and marks, but still conditionally. However, it is possible for a command stream to modify refs or create marks without creating any new objects. For example, reset a branch (and keep fast-import running): $ git fast-import reset refs/heads/master from refs/heads/master^ checkpoint but refs/heads/master remains unchanged. Other example: a commit command that re-creates an object that already exists in the object database. The man page also states that checkpoint "updates the refs" and that "placing a progress command immediately after a checkpoint will inform the reader when the checkpoint has been completed and it can safely access the refs that fast-import updated". This wasn't always true without this patch. This fix unconditionally calls dump_{branches,tags,marks}() for all checkpoint commands. dump_branches() and dump_tags() are cheap to call in the case of a no-op. Add tests to t9300 that observe the (non-packfiles) effects of checkpoint. Signed-off-by: Eric Rannaud <e@nanocritical.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-06-27Spelling fixesLibravatar Ville Skyttä1-1/+1
Signed-off-by: Ville Skyttä <ville.skytta@iki.fi> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-13Merge branch 'jk/big-and-future-archive-tar'Libravatar Junio C Hamano1-20/+3
"git archive" learned to handle files that are larger than 8GB and commits far in the future than expressible by the traditional US-TAR format. * jk/big-and-future-archive-tar: archive-tar: drop return value archive-tar: write extended headers for far-future mtime archive-tar: write extended headers for file sizes >= 8GB t5000: test tar files that overflow ustar headers t9300: factor out portable "head -c" replacement
2016-07-01t9300: factor out portable "head -c" replacementLibravatar Jeff King1-20/+3
It is sometimes useful to be able to read exactly N bytes from a pipe. Doing this portably turns out to be surprisingly difficult in shell scripts. We want a solution that: - is portable - never reads more than N bytes due to buffering (which would mean those bytes are not available to the next program to read from the same pipe) - handles partial reads by looping until N bytes are read (or we see EOF) - is resilient to stray signals giving us EINTR while trying to read (even though we don't send them, things like SIGWINCH could cause apparently-random failures) Some possible solutions are: - "head -c" is not portable, and implementations may buffer (though GNU head does not) - "read -N" is a bash-ism, and thus not portable - "dd bs=$n count=1" does not handle partial reads. GNU dd has iflags=fullblock, but that is not portable - "dd bs=1 count=$n" fixes the partial read problem (all reads are 1-byte, so there can be no partial response). It does make a lot of write() calls, but for our tests that's unlikely to matter. It's fairly portable. We already use it in our tests, and it's unlikely that implementations would screw up any of our criteria. The most unknown one would be signal handling. - perl can do a sysread() loop pretty easily. On my Linux system, at least, it seems to restart the read() call automatically. If that turns out not to be portable, though, it would be easy for us to handle it. That makes the perl solution the least bad (because we conveniently omitted "length of code" as a criterion). It's also what t9300 is currently using, so we can just pull the implementation from there. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-06-20Merge branch 'ew/fast-import-unpack-limit'Libravatar Junio C Hamano1-0/+2
"git fast-import" learned the same performance trick to avoid creating too small a packfile as "git fetch" and "git push" have, using *.unpackLimit configuration. * ew/fast-import-unpack-limit: fast-import: invalidate pack_id references after loosening fast-import: implement unpack limit
2016-05-17fast-import: do not truncate exported marks fileLibravatar Felipe Contreras1-0/+15
Certain lines of the marks file might be corrupted (or the objects missing due to a garbage collection), but that's no reason to truncate the file and essentially destroy the rest of it. Ideally missing objects should not cause a crash, we could just skip them, but that's another patch. Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-05-11fast-import: implement unpack limitLibravatar Eric Wong1-0/+2
With many incremental imports, small packs become highly inefficient due to the need to readdir scan and load many indices to locate even a single object. Frequent repacking and consolidation may be prohibitively expensive in terms of disk I/O, especially in large repositories where the initial packs were aggressively optimized and marked with .keep files. In those cases, users may be better served with loose objects and relying on "git gc --auto". This changes the default behavior of fast-import for small imports found in test cases, so adjustments to t9300 were necessary. Signed-off-by: Eric Wong <normalperson@yhbt.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-04-14Merge branch 'jk/getwholeline-getdelim-empty' into maintLibravatar Junio C Hamano1-0/+4
strbuf_getwholeline() did not NUL-terminate the buffer on certain corner cases in its error codepath. * jk/getwholeline-getdelim-empty: strbuf_getwholeline: NUL-terminate getdelim buffer on error
2016-03-05strbuf_getwholeline: NUL-terminate getdelim buffer on errorLibravatar Jeff King1-0/+4
Commit 0cc30e0 (strbuf_getwholeline: use getdelim if it is available, 2015-04-16) tries to clean up after getdelim() returns EOF, but gets one case wrong, which can lead in some obscure cases to us reading uninitialized memory. After getdelim() returns -1, we re-initialize the strbuf only if sb->buf is NULL. The thinking was that either: 1. We fed an existing allocated buffer to getdelim(), and at most it would have realloc'd, leaving our NUL in place. 2. We didn't have a buffer to feed, so we gave getdelim() NULL; sb->buf will remain NULL, and we just want to restore the empty slopbuf. But that second case isn't quite right. getdelim() may allocate a buffer, write nothing into it, and then return EOF. The resulting strbuf rightfully has sb->len set to "0", but is missing the NUL terminator in the first byte. Most call-sites are fine with this. They see the EOF and don't bother looking at the strbuf. Or they notice that sb->len is empty, and don't look at the contents. But there's at least one case that does neither, and relies on parsing the resulting (possibly zero-length) string: fast-import. You can see this in action with the new test (though we probably only notice failure there when run with --valgrind or ASAN). We can fix this by unconditionally resetting the strbuf when we have a buffer after getdelim(). That fixes case 2 above. Case 1 is probably already fine in practice, but it does not hurt for us to re-assert our invariants (especially because we are relying on whatever getdelim() happens to do, which may vary from platform to platform). Our fix covers that case, too. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-01-12t9300-fast-import.sh: use the $( ... ) construct for command substitutionLibravatar Elia Pinto1-34/+34
The Git CodingGuidelines prefer the $(...) construct for command substitution instead of using the backquotes `...`. The backquoted form is the traditional method for command substitution, and is supported by POSIX. However, all but the simplest uses become complicated quickly. In particular, embedded command substitutions and/or the use of double quotes require careful escaping with the backslash character. The patch was generated by: for _f in $(find . -name "*.sh") do perl -i -pe 'BEGIN{undef $/;} s/`(.+?)`/\$(\1)/smg' "${_f}" done and then carefully proof-read. Signed-off-by: Elia Pinto <gitter.spiros@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-11-20modernize t9300: move test preparations into test_expect_successLibravatar Johannes Sixt1-1250/+1264
Our usual style these days is to execute everything inside test_expect_success. Make it so. Signed-off-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Jeff King <peff@peff.net>
2015-11-20modernize t9300: mark here-doc words to ignore tab indentationLibravatar Johannes Sixt1-100/+100
In the next commit, we will indent test case preparations. This will require that here-documents ignore the tab indentation. Prepare for this change by marking the here-doc words accordingly. This does not have an effect now, but will remove some noise from the git diff -b output of the next commit. The change here is entirely automated with this perl command: perl -i -lpe 's/(cat.*<<) *((EOF|(EXPECT|INPUT)_END).*$)/$1-$2 &&/' t/t9300-fast-import.sh i.e., inserts a dash between << and the EOF word (and removes blanks that our style guide abhors) and appends the && that will become necessary. Signed-off-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Jeff King <peff@peff.net>
2015-11-20modernize t9300: use test_when_finished for clean-upLibravatar Johannes Sixt1-20/+16
A number of clean-ups of test cases are performed outside of test_expect_success. Replace these cases by using test_when_finished. Signed-off-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Jeff King <peff@peff.net>
2015-11-20modernize t9300: wrap lines after &&Libravatar Johannes Sixt1-16/+32
It is customary to have each command in test snippets on its own line. Fix those instances that do not follow this guideline. Signed-off-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Jeff King <peff@peff.net>
2015-11-20modernize t9300: use test_must_be_emptyLibravatar Johannes Sixt1-5/+3
Instead of comparing actual output to an empty file, use test_must_be_empty. In addition to the better error message provided by the helper, allocation of an empty file during the setup sequence can be avoided. Signed-off-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Jeff King <peff@peff.net>
2015-11-20modernize t9300: use test_must_failLibravatar Johannes Sixt1-14/+3
One test case open-codes a test for an expected failure. Replace it by test_must_fail. Signed-off-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Jeff King <peff@peff.net>
2015-11-20modernize t9300: single-quote placement and indentationLibravatar Johannes Sixt1-435/+437
Many test cases do not follow our modern style that places the single-quotes that surround the shell code snippets before and after the shell code. Make it so. Many of the lines changed in this way are indented other than by a single tab. Change them (and some additional lines) to be indented with a tab. Signed-off-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Jeff King <peff@peff.net>
2015-07-01fast-import: add a get-mark commandLibravatar Michael Haggerty1-0/+13
It is sometimes useful for importers to be able to read the SHA-1 corresponding to a mark that they have created via fast-import. For example, they might want to embed the SHA-1 into the commit message of a later commit. Or it might be useful for internal bookkeeping uses, or for logging. Add a "get-mark" command to "git fast-import" that allows the importer to ask for the value of a mark that has been created earlier. Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-20t: use test_must_fail instead of hand-rolled blocksLibravatar Jeff King1-5/+2
These test scripts likely predate test_must_fail, and can be made simpler by using it (in addition to making them pass --chain-lint). The case in t6036 loses some verbosity in the failure case, but it is so tied to a specific failure mode that it is not worth keeping around (and the outcome of the test is not affected at all). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-20t: fix trivial &&-chain breakageLibravatar Jeff King1-1/+1
These are tests which are missing a link in their &&-chain, but during a setup phase. We may fail to notice failure in commands that build the test environment, but these are typically not expected to fail at all (but it's still good to double-check that our test environment is what we expect). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-20t: fix severe &&-chain breakageLibravatar Jeff King1-1/+1
These are tests which are missing a link in their &&-chain, in a location which causes a significant portion of the test to be missed (e.g., the test effectively does nothing, or consists of a long string of actions and output comparisons, and we throw away the exit code of at least one part of the string). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-05Merge branch 'jc/diff-test-updates'Libravatar Junio C Hamano1-2/+2
Test clean-up. * jc/diff-test-updates: test_ln_s_add: refresh stat info of fake symbolic links t4008: modernise style t/diff-lib: check exact object names in compare_diff_raw tests: do not borrow from COPYING and README from the real source t4010: correct expected object names t9300: correct expected object names t4008: correct stale comments