summaryrefslogtreecommitdiff
path: root/t/t0021-conversion.sh
AgeCommit message (Collapse)AuthorFilesLines
2016-07-22diff: do not reuse worktree files that need "clean" conversionLibravatar Jeff King1-0/+11
When accessing a blob for a diff, we may try to reuse file contents in the working tree, under the theory that it is faster to mmap those file contents than it would be to extract the content from the object database. When we have to filter those contents, though, that assumption does not hold. Even for our internal conversions like CRLF, we have to allocate and fill a new buffer anyway. But much worse, for external clean filters we have to exec an arbitrary script, and we have no idea how expensive it may be to run. So let's skip this optimization when conversion into git's "clean" form is required. This applies whenever the "want_file" flag is false. When it's true, the caller actually wants the smudged worktree contents, which the reused file by definition already has (in fact, this is a key optimization going the other direction, since reusing the worktree file there lets us skip smudge filters). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-01-29convert: treat an empty string for clean/smudge filters as "cat"Libravatar Lars Schneider1-0/+16
Once a lower-priority configuration file defines a clean or smudge filter, there is no convenient way to override it to produce as-is output. Even though the configuration mechanism implements "the last one wins" semantics, you cannot set them to an empty string and expect them to work, as apply_filter() would try to run the empty string as an external command and fail. The conversion is not done, but the function would still report a failure to convert. Even though resetting the variable to "cat" (i.e. pass the data back as-is and report success) is an obvious and a viable way to solve this, it is wasteful to spawn an external process just as a workaround. Instead, teach apply_filter() to treat an empty string as a no-op filter that always returns successfully its input as-is without conversion. Signed-off-by: Lars Schneider <larsxschneider@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-06-01Merge branch 'jh/filter-empty-contents'Libravatar Junio C Hamano1-0/+26
The clean/smudge interface did not work well when filtering an empty contents (failed and then passed the empty input through). It can be argued that a filter that produces anything but empty for an empty input is nonsense, but if the user wants to do strange things, then why not? * jh/filter-empty-contents: sha1_file: pass empty buffer to index empty file
2015-05-20filter_buffer_or_fd(): ignore EPIPELibravatar Junio C Hamano1-0/+10
We are explicitly ignoring SIGPIPE, as we fully expect that the filter program may not read our output fully. Ignore EPIPE that may come from writing to it as well. A new test was stolen from Jeff's suggestion. Helped-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-05-18sha1_file: pass empty buffer to index empty fileLibravatar Jim Hill1-0/+26
`git add` of an empty file with a filter pops complaints from `copy_fd` about a bad file descriptor. This traces back to these lines in sha1_file.c:index_core: if (!size) { ret = index_mem(sha1, NULL, size, type, path, flags); The problem here is that content to be added to the index can be supplied from an fd, or from a memory buffer, or from a pathname. This call is supplying a NULL buffer pointer and a zero size. Downstream logic takes the complete absence of a buffer to mean the data is to be found elsewhere -- for instance, these, from convert.c: if (params->src) { write_err = (write_in_full(child_process.in, params->src, params->size) < 0); } else { write_err = copy_fd(params->fd, child_process.in); } ~If there's a buffer, write from that, otherwise the data must be coming from an open fd.~ Perfectly reasonable logic in a routine that's going to write from either a buffer or an fd. So change `index_core` to supply an empty buffer when indexing an empty file. There's a patch out there that instead changes the logic quoted above to take a `-1` fd to mean "use the buffer", but it seems to me that the distinction between a missing buffer and an empty one carries intrinsic semantics, where the logic change is adapting the code to handle incorrect arguments. Signed-off-by: Jim Hill <gjthill@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-08-28convert: stream from fd to required clean filter to reduce used address spaceLibravatar Steffen Prohaska1-5/+19
The data is streamed to the filter process anyway. Better avoid mapping the file if possible. This is especially useful if a clean filter reduces the size, for example if it computes a sha1 for binary data, like git media. The file size that the previous implementation could handle was limited by the available address space; large files for example could not be handled with (32-bit) msysgit. The new implementation can filter files of any size as long as the filter output is small enough. The new code path is only taken if the filter is required. The filter consumes data directly from the fd. If it fails, the original data is not immediately available. The condition can easily be handled as a fatal error, which is expected for a required filter anyway. If the filter was not required, the condition would need to be handled in a different way, like seeking to 0 and reading the data. But this would require more restructuring of the code and is probably not worth it. The obvious approach of falling back to reading all data would not help achieving the main purpose of this patch, which is to handle large files with limited address space. If reading all data is an option, we can simply take the old code path right away and mmap the entire file. The environment variable GIT_MMAP_LIMIT, which has been introduced in a previous commit is used to test that the expected code path is taken. A related test that exercises required filters is modified to verify that the data actually has been modified on its way from the file system to the object store. Signed-off-by: Steffen Prohaska <prohaska@zib.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-09test: turn EXPENSIVE into a lazy prerequisiteLibravatar Junio C Hamano1-2/+0
Two test scripts (t0021 and t5551) had copy & paste code to set EXPENSIVE prerequisite. Use the test_lazy_prereq helper to define them in the common t/test-lib.sh. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-20xread, xwrite: limit size of IO to 8MBLibravatar Steffen Prohaska1-0/+14
Checking out 2GB or more through an external filter (see test) fails on Mac OS X 10.8.4 (12E55) for a 64-bit executable with: error: read from external filter cat failed error: cannot feed the input to external filter cat error: cat died of signal 13 error: external filter cat failed 141 error: external filter cat failed The reason is that read() immediately returns with EINVAL when asked to read more than 2GB. According to POSIX [1], if the value of nbyte passed to read() is greater than SSIZE_MAX, the result is implementation-defined. The write function has the same restriction [2]. Since OS X still supports running 32-bit executables, the 32-bit limit (SSIZE_MAX = INT_MAX = 2GB - 1) seems to be also imposed on 64-bit executables under certain conditions. For write, the problem has been addressed earlier [6c642a]. Address the problem for read() and write() differently, by limiting size of IO chunks unconditionally on all platforms in xread() and xwrite(). Large chunks only cause problems, like causing latencies when killing the process, even if OS X was not buggy. Doing IO in reasonably sized smaller chunks should have no negative impact on performance. The compat wrapper clipped_write() introduced earlier [6c642a] is not needed anymore. It will be reverted in a separate commit. The new test catches read and write problems. Note that 'git add' exits with 0 even if it prints filtering errors to stderr. The test, therefore, checks stderr. 'git add' should probably be changed (sometime in another commit) to exit with nonzero if filtering fails. The test could then be changed to use test_must_fail. Thanks to the following people for suggestions and testing: Johannes Sixt <j6t@kdbg.org> John Keeping <john@keeping.me.uk> Jonathan Nieder <jrnieder@gmail.com> Kyle J. McKay <mackyle@gmail.com> Linus Torvalds <torvalds@linux-foundation.org> Torsten Bögershausen <tboegi@web.de> [1] http://pubs.opengroup.org/onlinepubs/009695399/functions/read.html [2] http://pubs.opengroup.org/onlinepubs/009695399/functions/write.html [6c642a] commit 6c642a878688adf46b226903858b53e2d31ac5c3 compate/clipped-write.c: large write(2) fails on Mac OS X/XNU Signed-off-by: Steffen Prohaska <prohaska@zib.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-02-17Add a setting to require a filter to be successfulLibravatar Jehan Bing1-0/+37
By default, a missing filter driver or a failure from the filter driver is not an error, but merely makes the filter operation a no-op pass through. This is useful to massage the content into a shape that is more convenient for the platform, filesystem, and the user to use, and the content filter mechanism is not used to turn something unusable into usable. However, we could also use of the content filtering mechanism and store the content that cannot be directly used in the repository (e.g. a UUID that refers to the true content stored outside git, or an encrypted content) and turn it into a usable form upon checkout (e.g. download the external content, or decrypt the encrypted content). For such a use case, the content cannot be used when filter driver fails, and we need a way to tell Git to abort the whole operation for such a failing or missing filter driver. Add a new "filter.<driver>.required" configuration variable to mark the second use case. When it is set, git will abort the operation when the filter driver does not exist or exits with a non-zero status code. Signed-off-by: Jehan Bing <jehan@orb.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-05-26t0021: test application of both crlf and identLibravatar Junio C Hamano1-8/+24
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-05-26t0021-conversion.sh: fix NoTerminatingSymbolAtEOF testLibravatar René Scharfe1-11/+12
The last line of the test file "expanded-keywords" ended in a newline, which is a valid terminator for ident. Use printf instead of echo to omit it and thus really test if a file that ends unexpectedly in the middle of an ident tag is handled properly. Also take the oppertunity to calculate the expected ID dynamically instead of hardcoding it into the test script. This should make future changes easier. Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-12-22t0021: avoid getting filter killed with SIGPIPELibravatar Junio C Hamano1-0/+1
The fake filter did not read from the standard input at all, which caused the calling side to die with SIGPIPE, depending on the timing. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-12-22convert filter: supply path to external driverLibravatar Pete Wyckoff1-0/+42
Filtering to support keyword expansion may need the name of the file being filtered. In particular, to support p4 keywords like $File: //depot/product/dir/script.sh $ the smudge filter needs to know the name of the file it is smudging. Allow "%f" in the custom filter command line specified in the configuration. This will be substituted by the filename inside a single-quote pair to be passed to the shell. Signed-off-by: Pete Wyckoff <pw@padd.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-10convert: Keep foreign $Id$ on checkout.Libravatar Henrik Grubbström1-1/+1
If there are foreign $Id$ keywords in the repository, they are most likely there for a reason. Let's keep them on checkout (which is also what the documentation indicates). Foreign $Id$ keywords are now recognized by there being multiple space separated fields in $Id:xxxxx$. Signed-off-by: Henrik Grubbström <grubba@grubba.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-10convert: Safer handling of $Id$ contraction.Libravatar Henrik Grubbström1-6/+10
The code to contract $Id:xxxxx$ strings could eat an arbitrary amount of source text if the terminating $ was lost. It now refuses to contract $Id:xxxxx$ strings spanning multiple lines. Signed-off-by: Henrik Grubbström <grubba@grubba.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-01-05t0021: use $SHELL_PATH for the filter scriptLibravatar Johannes Sixt1-1/+2
On Windows, we need the shbang line to correctly invoke shell scripts via a POSIX shell, except when the script is invoked via 'sh -c' because sh (a bash) does "the right thing". But the clean and smudge filters will not always be invoked via 'sh -c'; to futureproof, we should mark the the one in t0021-conversion with #!$SHELL_PATH. Signed-off-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-03-11t0021: tr portability fix for SolarisLibravatar Jeff King1-1/+3
Solaris' /usr/bin/tr doesn't seem to like multiple character ranges in brackets (it simply prints "Bad string"). Instead, let's just enumerate the transformation we want. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-10-21t0021-conversion.sh: Test that the clean filter really cleans content.Libravatar Johannes Sixt1-1/+6
This test uses a rot13 filter, which is its own inverse. It tested only that the content was the same as the original after both the 'clean' and the 'smudge' filter were applied. This way it would not detect whether any filter was run at all. Hence, here we add another test that checks that the repository contained content that was processed by the 'clean' filter. Signed-off-by: Johannes Sixt <johannes.sixt@telecom.at> Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2007-05-28Add test case for $Id$ expanded in the repositoryLibravatar Andy Parkins1-0/+36
This test case would have caught the bug fixed by revision c23290d5. It puts various forms of $Id$ into a file in the repository, without allowing git to collapse them to uniformity. Then enables the $Id$ expansion on checkout, and checks that what is checked out has coped with the various forms. Signed-off-by: Andy Parkins <andyparkins@gmail.com> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-14Use $Id$ as the ident attribute keyword rather than $ident$ to be consistent ↵Libravatar Andy Parkins1-2/+2
with other VCSs $Id$ is present already in SVN and CVS; it would mean that people converting their existing repositories won't have to make any changes to the source files should they want to make use of the ident attribute. Given that it's a feature that's meant to calm those very people, it seems obtuse to make them edit every file just to make use of it. I think that bzr uses $Id$; Mercurial has examples hooks for $Id$; monotone has $Id$ on its wishlist. I can't think of a good reason not to stick with the de-facto standard and call ours $Id$ instead of $ident$. Signed-off-by: Andy Parkins <andyparkins@gmail.com> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-24Add 'filter' attribute and external filter driver definition.Libravatar Junio C Hamano1-0/+9
The interface is similar to the custom low-level merge drivers. First you configure your filter driver by defining 'filter.<name>.*' variables in the configuration. filter.<name>.clean filter command to run upon checkin filter.<name>.smudge filter command to run upon checkout Then you assign filter attribute to each path, whose name matches the custom filter driver's name. Example: (in .gitattributes) *.c filter=indent (in config) [filter "indent"] clean = indent smudge = cat Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-24Add 'ident' conversion.Libravatar Junio C Hamano1-0/+39
The 'ident' attribute set to path squashes "$ident:<any bytes except dollor sign>$" to "$ident$" upon checkin, and expands it to "$ident: <blob SHA-1> $" upon checkout. As we have two conversions that affect checkin/checkout paths, clarify how they interact with each other. Signed-off-by: Junio C Hamano <junkio@cox.net>