summaryrefslogtreecommitdiff
path: root/sha1_file.c
AgeCommit message (Collapse)AuthorFilesLines
2011-03-26Merge branch 'jc/maint-rerere-in-workdir'Libravatar Junio C Hamano1-0/+29
* jc/maint-rerere-in-workdir: rerere: make sure it works even in a workdir attached to a young repository
2011-03-23rerere: make sure it works even in a workdir attached to a young repositoryLibravatar Junio C Hamano1-0/+29
The git-new-workdir script in contrib/ makes a new work tree by sharing many subdirectories of the .git directory with the original repository. When rerere.enabled is set in the original repository, but the user has not encountered any conflicts yet, the original repository may not yet have .git/rr-cache directory. When rerere wants to run in a new work tree created from such a young original repository, it fails to mkdir(2) .git/rr-cache that is a symlink to a yet-to-be-created directory. There are three possible approaches to this: - A naive solution is not to create a symlink in the git-new-workdir script to a directory the original does not have (yet). This is not a solution, as we tend to lazily create subdirectories of .git/, and having rerere.enabled configuration set is a strong indication that the user _wants_ to have this lazy creation to happen; - We could always create .git/rr-cache upon repository creation. This is tempting but will not help people with existing repositories. - Detect this case by seeing that mkdir(2) failed with EEXIST, checking that the path is a symlink, and try running mkdir(2) on the link target. This patch solves the issue by doing the third one. Strictly speaking, this is incomplete. It does not attempt to handle relative symbolic link that points into the original repository, but this is good enough to help people who use contrib/workdir/git-new-workdir script. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-03-23Merge branch 'jn/maint-c99-format'Libravatar Junio C Hamano1-7/+2
* jn/maint-c99-format: unbreak and eliminate NO_C99_FORMAT mktag: avoid %td in format string
2011-03-17unbreak and eliminate NO_C99_FORMATLibravatar Jonathan Nieder1-7/+2
In the spirit of v1.5.0.2~21 (Check for PRIuMAX rather than NO_C99_FORMAT in fast-import.c, 2007-02-20), use PRIuMAX from git-compat-util.h on all platforms instead of C99-specific formats like %zu with dangerous fallbacks to %u or %lu. So now C99-challenged platforms can build git without provoking warnings or errors from printf, even if pointers do not have the same size as an int or long. The need for a fallback PRIuMAX is detected in git-compat-util.h with "#ifndef PRIuMAX". So while at it, simplify the Makefile and configure script by eliminating the NO_C99_FORMAT knob altogether. Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-03-15Merge branch 'sp/maint-fd-limit'Libravatar Junio C Hamano1-18/+66
* sp/maint-fd-limit: sha1_file.c: Don't retain open fds on small packs mingw: add minimum getrlimit() compatibility stub Limit file descriptors used by packs
2011-03-02sha1_file.c: Don't retain open fds on small packsLibravatar Shawn O. Pearce1-5/+36
If a pack file is small enough that its entire contents fits within one mmap window, mmap the file and then immediately close its file descriptor. This reduces the number of file descriptors that are needed to read from repositories with many tiny pack files, such as one that has received 1000 pushes (and created 1000 small pack files) since its last repack. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-02-28Limit file descriptors used by packsLibravatar Shawn O. Pearce1-13/+30
Rather than using 'errno == EMFILE' after a failed open() call to indicate the process is out of file descriptors and an LRU pack window should be closed, place a hard upper limit on the number of open packs based on the actual rlimit of the process. By using a hard upper limit that is below the rlimit of the current process it is not necessary to check for EMFILE on every single fd-allocating system call. Instead reserving 25 file descriptors makes it safe to assume the system call won't fail due to being over the filedescriptor limit. Here 25 is chosen as a WAG, but considers 3 for stdin/stdout/stderr, and at least a few for other Git code to operate on temporary files. An additional 20 is reserved as it is not known what the C library needs to perform other services on Git's behalf, such as nsswitch or name resolution. This fixes a case where running `git gc --auto` in a repository with more than 1024 packs (but an rlimit of 1024 open fds) fails due to the temporary output file not being able to allocate a file descriptor. The output file is opened by pack-objects after object enumeration and delta compression are done, both of which have already opened all of the packs and fully populated the file descriptor table. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-02-27Merge branch 'nd/hash-object-sanity'Libravatar Junio C Hamano1-7/+47
* nd/hash-object-sanity: Make hash-object more robust against malformed objects Conflicts: cache.h
2011-02-14correct type of EMPTY_TREE_SHA1_BINLibravatar Jonathan Nieder1-1/+1
Functions such as hashcmp that expect a binary SHA-1 value take parameters of type "unsigned char *" to avoid accepting a textual SHA-1 passed by mistake. Unfortunately, this means passing the string literal EMPTY_TREE_SHA1_BIN requires an ugly cast. Tweak the definition of EMPTY_TREE_SHA1_BIN to produce a value of more convenient type. In the future the definition might change to extern const unsigned char empty_tree_sha1_bin[20]; #define EMPTY_TREE_SHA1_BIN empty_tree_sha1_bin Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-02-07sha1_object_info: examine cached_object store tooLibravatar Nguyễn Thái Ngọc Duy1-0/+8
Cached object store was added in d66b37b (Add pretend_sha1_file() interface. - 2007-02-04) as a way to temporarily inject some objects to object store. But only read_sha1_file() knows about this store. While it will return an object from this store, sha1_object_info() will happily say "object not found". Teach sha1_object_info() about the cached store for consistency. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-02-07sha1_file.c: move find_cached_object up so sha1_object_info can use itLibravatar Nguyễn Thái Ngọc Duy1-35/+35
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-02-07Make hash-object more robust against malformed objectsLibravatar Nguyễn Thái Ngọc Duy1-7/+47
Commits, trees and tags have structure. Don't let users feed git with malformed ones. Sooner or later git will die() when encountering them. Note that this patch does not check semantics. A tree that points to non-existent objects is perfectly OK (and should be so, users may choose to add commit first, then its associated tree for example). Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-01-20Correctly report corrupted objectsLibravatar Björn Steinbrink1-1/+1
The errno check added in commit 3ba7a06 "A loose object is not corrupt if it cannot be read due to EMFILE" only checked for whether errno is not ENOENT and thus incorrectly treated "no error" as an error condition. Because of that, it never reached the code path that would report that the object is corrupted and instead caused funny errors like: fatal: failed to read object 333c4768ce595793fdab1ef3a036413e2a883853: Success So we have to extend the check to cover the case in which the object file was successfully read, but its contents are corrupted. Reported-by: Will Palmer <wmpalmer@gmail.com> Signed-off-by: Björn Steinbrink <B.Steinbrink@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-12-03Merge branch 'jn/thinner-wrapper'Libravatar Junio C Hamano1-0/+26
* jn/thinner-wrapper: Remove pack file handling dependency from wrapper.o pack-objects: mark file-local variable static wrapper: give zlib wrappers their own translation unit strbuf: move strbuf_branchname to sha1_name.c path helpers: move git_mkstemp* to wrapper.c wrapper: move odb_* to environment.c wrapper: move xmmap() to sha1_file.c
2010-11-10Remove pack file handling dependency from wrapper.oLibravatar Jonathan Nieder1-0/+11
As v1.7.0-rc0~43 (slim down "git show-index", 2010-01-21) explains, use of xmalloc() brings in a dependency on zlib, the sha1 lib, and the rest of git's object file access machinery via try_to_free_pack_memory. That is overkill when xmalloc is just being used as a convenience wrapper to exit when no memory is available. So defer setting try_to_free_pack_memory as try_to_free_routine until the first packfile is opened in add_packed_git(). After this change, a simple program using xmalloc() and no other functions will not pull in any code from libgit.a aside from wrapper.o and usage.o. Improved-by: René Scharfe <rene.scharfe@lsrfire.ath.cx> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-11-10wrapper: move xmmap() to sha1_file.cLibravatar Jonathan Nieder1-0/+15
wrapper.o depends on sha1_file.o for a number of reasons. One is release_pack_memory(). xmmap function calls mmap, discarding unused pack windows when necessary to relieve memory pressure. Simple git programs using wrapper.o as a friendly libc do not need this functionality. So move xmmap to sha1_file.o, where release_pack_memory() is. Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-11-03Work around EMFILE when there are too many pack filesLibravatar Shawn O. Pearce1-16/+27
When opening any files in the object database, release unused pack windows if the open(2) syscall fails due to EMFILE (too many open files in this process). This allows Git to degrade gracefully on a repository with thousands of pack files, and a commit stored in a loose object in the middle of the history. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-11-03Use git_open_noatime when accessing pack dataLibravatar Shawn O. Pearce1-4/+6
This utility function avoids an unnecessary update of the access time for a loose object file. Just as the atime isn't useful on a loose object, its not useful on the pack or the corresonding idx file. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-11-03A loose object is not corrupt if it cannot be read due to EMFILELibravatar Junio C Hamano1-1/+6
"git fsck" bails out with a claim that a loose object that cannot be read but exists on the filesystem to be corrupt, which is wrong when read_object() failed due to e.g. EMFILE. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-11-03read_sha1_file(): report correct name of packfile with a corrupt objectLibravatar Junio C Hamano1-17/+24
Clarify the error reporting logic by moving the normal codepath (i.e. we read the object we wanted to read correctly) up and return early. The logic to report the name of the packfile with a corrupt object, introduced by e8b15e6 (sha1_file: Show the the type and path to corrupt objects, 2010-06-10), was totally bogus. The function that knows which bad object came from what packfile is has_packed_and_bad(); make it report which packfile the problem was found. "Corrupt" is already an adjective, e.g. an object is "corrupt"; we do not have to say "corrupted object". Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-07-14sha1_file: Show the the type and path to corrupt objectsLibravatar Ævar Arnfjörð Bjarmason1-2/+11
Change the error message that's displayed when we encounter corrupt objects to be more specific. We now print the type (loose or packed) of corrupted objects, along with the full path to the file in question. Before: $ git cat-file blob 909ef997367880aaf2133bafa1f1a71aa28e09df fatal: object 909ef997367880aaf2133bafa1f1a71aa28e09df is corrupted After: $ git cat-file blob 909ef997367880aaf2133bafa1f1a71aa28e09df fatal: loose object 909ef997367880aaf2133bafa1f1a71aa28e09df (stored in .git/objects/90/9ef997367880aaf2133bafa1f1a71aa28e09df) is corrupted Knowing the path helps to quickly analyze what's wrong: $ file .git/objects/90/9ef997367880aaf2133bafa1f1a71aa28e09df .git/objects/90/9ef997367880aaf2133bafa1f1a71aa28e09df: empty Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-06-13Merge branch 'jk/maint-sha1-file-name-fix'Libravatar Junio C Hamano1-13/+15
* jk/maint-sha1-file-name-fix: remove over-eager caching in sha1_file_name
2010-05-25remove over-eager caching in sha1_file_nameLibravatar Jeff King1-13/+15
This function takes a sha1 and produces a loose object filename. It caches the location of the object directory so that it can fill the sha1 information directly without allocating a new buffer (and in its original incarnation, without calling getenv(), though these days we cache that with the code in environment.c). This cached base directory can become stale, however, if in a single process git changes the location of the object directory (e.g., by running setup_work_tree, which will chdir to the new worktree). In most cases this isn't a problem, because we tend to set up the git repository location and do any chdir()s before actually looking up any objects, so the first lookup will cache the correct location. In the case of reset --hard, however, we do something like: 1. look up the commit object 2. notice we are doing --hard, run setup_work_tree 3. look up the tree object to reset Step (3) fails because our cache object directory value is bogus. This patch simply removes the caching. We use a static buffer instead of allocating one each time (the original version treated the malloc'd buffer as a static, so there is no change in calling semantics). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-05-21Merge branch 'sp/maint-dumb-http-pack-reidx'Libravatar Junio C Hamano1-4/+10
* sp/maint-dumb-http-pack-reidx: http.c::new_http_pack_request: do away with the temp variable filename http-fetch: Use temporary files for pack-*.idx until verified http-fetch: Use index-pack rather than verify-pack to check packs Allow parse_pack_index on temporary files Extract verify_pack_index for reuse from verify_pack Introduce close_pack_index to permit replacement http.c: Remove unnecessary strdup of sha1_to_hex result http.c: Don't store destination name in request structures http.c: Drop useless != NULL test in finish_http_pack_request http.c: Tiny refactoring of finish_http_pack_request t5550-http-fetch: Use subshell for repository operations http.c: Remove bad free of static block
2010-05-18Merge branch 'maint'Libravatar Junio C Hamano1-3/+4
* maint: Documentation/gitdiffcore: fix order in pickaxe description Documentation: fix minor inconsistency Documentation: rebase -i ignores options passed to "git am" hash_object: correction for zero length file
2010-05-18hash_object: correction for zero length fileLibravatar Dmitry Potapov1-3/+4
The check whether size is zero was done after if size <= SMALL_FILE_SIZE, as result, zero size case was never triggered. Instead zero length file was treated as any other small file. This did not caused any problem, but if we have a special case for size equal to zero, it is better to make it work and avoid redundant malloc(). Signed-off-by: Dmitry Potapov <dpotapov@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-19Allow parse_pack_index on temporary filesLibravatar Shawn O. Pearce1-2/+1
The easiest way to verify a pack index is to open it through the standard parse_pack_index function, permitting the header check to happen when the file is mapped. However, the dumb HTTP client needs to verify a pack index before its moved into its proper file name within the objects/pack directory, to prevent a corrupt index from being made available. So permit the caller to specify the exact path of the index file. For now we're still using the final destination name within the sole call site in http.c, but eventually we will start to parse the temporary path instead. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-19Introduce close_pack_index to permit replacementLibravatar Shawn O. Pearce1-2/+9
By closing the pack index, a caller can later overwrite the index with an updated index file, possibly after converting from v1 to the v2 format. Because p->index_data is NULL after close, on the next access the index will be opened again and the other members will be updated with new data. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-01make commit_tree a library functionLibravatar Jeff King1-0/+10
Until now, this has been part of the commit-tree builtin. However, it is already used by other builtins (like commit, merge, and notes), and it would be useful to access it from library code. The check_valid helper has to come along, too, but is given a more library-ish name of "assert_sha1_type". Otherwise, the code is unchanged. There are still a few rough edges for a library function, like printing the utf8 warning to stderr, but we can address those if and when they come up as inappropriate. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-01fix const-correctness of write_sha1_fileLibravatar Jeff King1-3/+3
These should take const buffers as input data, but zlib's next_in pointer is not const-correct. Let's fix it at the zlib level, though, so the cast happens in one obvious place. This should be safe, as a similar cast is used in zlib's example code for a const array. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-03-08Merge branch 'mm/mkstemps-mode-for-packfiles' into maintLibravatar Junio C Hamano1-3/+3
* mm/mkstemps-mode-for-packfiles: Use git_mkstemp_mode instead of plain mkstemp to create object files git_mkstemps_mode: don't set errno to EINVAL on exit. Use git_mkstemp_mode and xmkstemp_mode in odb_mkstemp, not chmod later. git_mkstemp_mode, xmkstemp_mode: variants of gitmkstemps with mode argument. Move gitmkstemps to path.c Add a testcase for ACL with restrictive umask.
2010-03-07Merge branch 'nd/root-git'Libravatar Junio C Hamano1-7/+0
* nd/root-git: Add test for using Git at root of file system Support working directory located at root Move offset_1st_component() to path.c init-db, rev-parse --git-dir: do not append redundant slash make_absolute_path(): Do not append redundant slash Conflicts: setup.c sha1_file.c
2010-03-07Merge branch 'mm/mkstemps-mode-for-packfiles'Libravatar Junio C Hamano1-3/+3
* mm/mkstemps-mode-for-packfiles: Use git_mkstemp_mode instead of plain mkstemp to create object files git_mkstemps_mode: don't set errno to EINVAL on exit. Use git_mkstemp_mode and xmkstemp_mode in odb_mkstemp, not chmod later. git_mkstemp_mode, xmkstemp_mode: variants of gitmkstemps with mode argument. Move gitmkstemps to path.c Add a testcase for ACL with restrictive umask.
2010-03-04Merge branch 'dp/read-not-mmap-small-loose-object' into maintLibravatar Junio C Hamano1-0/+10
* dp/read-not-mmap-small-loose-object: hash-object: don't use mmap() for small files
2010-03-02Merge branch 'np/compress-loose-object-memsave'Libravatar Junio C Hamano1-14/+19
* np/compress-loose-object-memsave: sha1_file: be paranoid when creating loose objects sha1_file: don't malloc the whole compressed result when writing out objects
2010-02-22Use git_mkstemp_mode instead of plain mkstemp to create object filesLibravatar Matthieu Moy1-3/+3
We used to unnecessarily give the read permission to group and others, regardless of the umask, which isn't serious because the objects are still protected by their containing directory, but isn't necessary either. Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-02-21sha1_file: be paranoid when creating loose objectsLibravatar Nicolas Pitre1-0/+9
We don't want the data being deflated and stored into loose objects to be different from what we expect. While the deflated data is protected by a CRC which is good enough for safe data retrieval operations, we still want to be doubly sure that the source data used at object creation time is still what we expected once that data has been deflated and its CRC32 computed. The most plausible data corruption may occur if the source file is modified while Git is deflating and writing it out in a loose object. Or Git itself could have a bug causing memory corruption. Or even bad RAM could cause trouble. So it is best to make sure everything is coherent and checksum protected from beginning to end. To do so we compute the SHA1 of the data being deflated _after_ the deflate operation has consumed that data, and make sure it matches with the expected SHA1. This way we can rely on the CRC32 checked by the inflate operation to provide a good indication that the data is still coherent with its SHA1 hash. One pathological case we ignore is when the data is modified before (or during) deflate call, but changed back before it is hashed. There is some overhead of course. Using 'git add' on a set of large files: Before: real 0m25.210s user 0m23.783s sys 0m1.408s After: real 0m26.537s user 0m25.175s sys 0m1.358s The overhead is around 5% for full data coherency guarantee. Signed-off-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-02-21hash-object: don't use mmap() for small filesLibravatar Dmitry Potapov1-0/+10
Using read() instead of mmap() can be 39% speed up for 1Kb files and is 1% speed up 1Mb files. For larger files, it is better to use mmap(), because the difference between is not significant, and when there is not enough memory, mmap() performs much better, because it avoids swapping. Signed-off-by: Dmitry Potapov <dpotapov@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-02-21sha1_file: don't malloc the whole compressed result when writing out objectsLibravatar Nicolas Pitre1-14/+10
There is no real advantage to malloc the whole output buffer and deflate the data in a single pass when writing loose objects. That is like only 1% faster while using more memory, especially with large files where memory usage is far more. It is best to deflate and write the data out in small chunks reusing the same memory instead. For example, using 'git add' on a few large files averaging 40 MB ... Before: 21.45user 1.10system 0:22.57elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+828040outputs (0major+142640minor)pagefaults 0swaps After: 21.50user 1.25system 0:22.76elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+828040outputs (0major+104408minor)pagefaults 0swaps While the runtime stayed relatively the same, the number of minor page faults went down significantly. Signed-off-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-02-16Move offset_1st_component() to path.cLibravatar Nguyễn Thái Ngọc Duy1-7/+0
The implementation is also lightly modified to use is_dir_sep() instead of hardcoding '/'. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-01-27Merge branch 'il/maint-xmallocz'Libravatar Junio C Hamano1-4/+2
* il/maint-xmallocz: Fix integer overflow in unpack_compressed_entry() Fix integer overflow in unpack_sha1_rest() Fix integer overflow in patch_delta() Add xmallocz()
2010-01-26Fix integer overflow in unpack_compressed_entry()Libravatar Ilari Liusvaara1-2/+1
Signed-off-by: Ilari Liusvaara <ilari.liusvaara@elisanet.fi> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-01-26Fix integer overflow in unpack_sha1_rest()Libravatar Ilari Liusvaara1-2/+1
[jc: later NUL termination by the caller becomes unnecessary] Signed-off-by: Ilari Liusvaara <ilari.liusvaara@elisanet.fi> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-01-21slim down "git show-index"Libravatar Linus Torvalds1-66/+0
As the documentation says, this is primarily for debugging, and in the longer term we should rename it to test-show-index or something. In the meantime, just avoid xmalloc (which slurps in the rest of git), and separating out the trivial hex functions into "hex.o". This results in [torvalds@nehalem git]$ size git-show-index text data bss dec hex filename 222818 2276 112688 337782 52776 git-show-index (before) 5696 624 1264 7584 1da0 git-show-index (after) which is a whole lot better. Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-01-12sha1_file.c: remove unused functionLibravatar Junio C Hamano1-8/+0
has_pack_file() is not used anywhere. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-10-21Fix incorrect error check while reading deflated pack dataLibravatar Junio C Hamano1-5/+3
The loop in get_size_from_delta() feeds a deflated delta data from the pack stream _until_ we get inflated result of 20 bytes[*] or we reach the end of stream. Side note. This magic number 20 does not have anything to do with the size of the hash we use, but comes from 1a3b55c (reduce delta head inflated size, 2006-10-18). The loop reads like this: do { in = use_pack(); stream.next_in = in; st = git_inflate(&stream, Z_FINISH); curpos += stream.next_in - in; } while ((st == Z_OK || st == Z_BUF_ERROR) && stream.total_out < sizeof(delta_head)); This git_inflate() can return: - Z_STREAM_END, if use_pack() fed it enough input and the delta itself was smaller than 20 bytes; - Z_OK, when some progress has been made; - Z_BUF_ERROR, if no progress is possible, because we either ran out of input (due to corrupt pack), or we ran out of output before we saw the end of the stream. The fix b3118bd (sha1_file: Fix infinite loop when pack is corrupted, 2009-10-14) attempted was against a corruption that appears to be a valid stream that produces a result larger than the output buffer, but we are not even trying to read the stream to the end in this loop. If avail_out becomes zero, total_out will be the same as sizeof(delta_head) so the loop will terminate without the "fix". There is no fix from b3118bd needed for this loop, in other words. The loop in unpack_compressed_entry() is quite a different story. It feeds a deflated stream (either delta or base) and allows the stream to produce output up to what we expect but no more. do { in = use_pack(); stream.next_in = in; st = git_inflate(&stream, Z_FINISH); curpos += stream.next_in - in; } while (st == Z_OK || st == Z_BUF_ERROR) This _does_ risk falling into an endless interation, as we can exhaust avail_out if the length we expect is smaller than what the stream wants to produce (due to pack corruption). In such a case, avail_out will become zero and inflate() will return Z_BUF_ERROR, while avail_in may (or may not) be zero. But this is not a right fix: do { in = use_pack(); stream.next_in = in; st = git_inflate(&stream, Z_FINISH); + if (st == Z_BUF_ERROR && (stream.avail_in || !stream.avail_out) + break; /* wants more input??? */ curpos += stream.next_in - in; } while (st == Z_OK || st == Z_BUF_ERROR) as Z_BUF_ERROR from inflate() may be telling us that avail_in has also run out before reading the end of stream marker. In such a case, both avail_in and avail_out would be zero, and the loop should iterate to allow the end of stream marker to be seen by inflate from the input stream. The right fix for this loop is likely to be to increment the initial avail_out by one (we allocate one extra byte to terminate it with NUL anyway, so there is no risk to overrun the buffer), and break out if we see that avail_out has become zero, in order to detect that the stream wants to produce more than what we expect. After the loop, we have a check that exactly tests this condition: if ((st != Z_STREAM_END) || stream.total_out != size) { free(buffer); return NULL; } So here is a patch (without my previous botched attempts) to fix this issue. The first hunk reverts the corresponding hunk from b3118bd, and the second hunk is the same fix proposed earlier. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-10-14sha1_file: Fix infinite loop when pack is corruptedLibravatar Shawn O. Pearce1-0/+4
Some types of corruption to a pack may confuse the deflate stream which stores an object. In Andy's reported case a 36 byte region of the pack was overwritten, leading to what appeared to be a valid deflate stream that was trying to produce a result larger than our allocated output buffer could accept. Z_BUF_ERROR is returned from inflate() if either the input buffer needs more input bytes, or the output buffer has run out of space. Previously we only considered the former case, as it meant we needed to move the stream's input buffer to the next window in the pack. We now abort the loop if inflate() returns Z_BUF_ERROR without consuming the entire input buffer it was given, or has filled the entire output buffer but has not yet returned Z_STREAM_END. Either state is a clear indicator that this loop is not working as expected, and should not continue. This problem cannot occur with loose objects as we open the entire loose object as a single buffer and treat Z_BUF_ERROR as an error. Reported-by: Andy Isaacson <adi@hexapodia.org> Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-08-21Merge branch 'cc/replace'Libravatar Junio C Hamano1-5/+18
* cc/replace: t6050: check pushing something based on a replaced commit Documentation: add documentation for "git replace" Add git-replace to .gitignore builtin-replace: use "usage_msg_opt" to give better error messages parse-options: add new function "usage_msg_opt" builtin-replace: teach "git replace" to actually replace Add new "git replace" command environment: add global variable to disable replacement mktag: call "check_sha1_signature" with the replacement sha1 replace_object: add a test case object: call "check_sha1_signature" with the replacement sha1 sha1_file: add a "read_sha1_file_repl" function replace_object: add mechanism to replace objects found in "refs/replace/" refs: add a "for_each_replace_ref" function
2009-07-22refactor: use bitsizeof() instead of 8 * sizeof()Libravatar Pierre Habouzit1-1/+1
Signed-off-by: Pierre Habouzit <madcoder@debian.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-07-06Merge branch 'tr/die_errno'Libravatar Junio C Hamano1-1/+1
* tr/die_errno: Use die_errno() instead of die() when checking syscalls Convert existing die(..., strerror(errno)) to die_errno() die_errno(): double % in strerror() output just in case Introduce die_errno() that appends strerror(errno) to die()