summaryrefslogtreecommitdiff
path: root/pack-write.c
AgeCommit message (Collapse)AuthorFilesLines
2014-01-10Merge branch 'jk/sha1write-void'Libravatar Junio C Hamano1-2/+1
Code clean-up. * jk/sha1write-void: do not pretend sha1write returns errors
2013-12-26do not pretend sha1write returns errorsLibravatar Jeff King1-2/+1
The sha1write function returns an int, but it will always be "0". The failure-prone parts of the function happen in the "flush" callback, which cannot pass an error back to us. So we just end up calling die() during the flush. Let's just drop the return value altogether, as it only confuses callers into thinking that it might be useful. Only one call site actually checked the return value. We can drop that check, since it just led to a die() anyway. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-12-05pack-objects: name pack files after trailer hashLibravatar Jeff King1-7/+1
Our current scheme for naming packfiles is to calculate the sha1 hash of the sorted list of objects contained in the packfile. This gives us a unique name, so we are reasonably sure that two packs with the same name will contain the same objects. It does not, however, tell us that two such packs have the exact same bytes. This makes things awkward if we repack the same set of objects. Due to run-to-run variations, the bytes may not be identical (e.g., changed zlib or git versions, different source object reuse due to new packs in the repository, or even different deltas due to races during a multi-threaded delta search). In theory, this could be helpful to a program that cares that the packfile contains a certain set of objects, but does not care about the particular representation. In practice, no part of git makes use of that, and in many cases it is potentially harmful. For example, if a dumb http client fetches the .idx file, it must be sure to get the exact .pack that matches it. Similarly, a partial transfer of a .pack file cannot be safely resumed, as the actual bytes may have changed. This could also affect a local client which opened the .idx and .pack files, closes the .pack file (due to memory or file descriptor limits), and then re-opens a changed packfile. In all of these cases, git can detect the problem, as we have the sha1 of the bytes themselves in the pack trailer (which we verify on transfer), and the .idx file references the trailer from the matching packfile. But it would be simpler and more efficient to actually get the correct bytes, rather than noticing the problem and having to restart the operation. This patch simply uses the pack trailer sha1 as the pack name. It should be similarly unique, but covers the exact representation of the objects. Other parts of git should not care, as the pack name is returned by pack-objects and is essentially opaque. One test needs to be updated, because it actually corrupts a pack and expects that re-packing the corrupted bytes will use the same name. It won't anymore, but we can easily just use the name that pack-objects hands back. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-12-21Appease Sun Studio by renaming "tmpfile"Libravatar Ævar Arnfjörð Bjarmason1-3/+3
On Solaris the system headers define the "tmpfile" name, which'll cause Git compiled with Sun Studio 12 Update 1 to whine about us redefining the name: "pack-write.c", line 76: warning: name redefined by pragma redefine_extname declared static: tmpfile (E_PRAGMA_REDEFINE_STATIC) "sha1_file.c", line 2455: warning: name redefined by pragma redefine_extname declared static: tmpfile (E_PRAGMA_REDEFINE_STATIC) "fast-import.c", line 858: warning: name redefined by pragma redefine_extname declared static: tmpfile (E_PRAGMA_REDEFINE_STATIC) "builtin/index-pack.c", line 175: warning: name redefined by pragma redefine_extname declared static: tmpfile (E_PRAGMA_REDEFINE_STATIC) Just renaming the "tmpfile" variable to "tmp_file" in the relevant places is the easiest way to fix this. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-12-16Merge branch 'jc/stream-to-pack'Libravatar Junio C Hamano1-0/+53
* jc/stream-to-pack: bulk-checkin: replace fast-import based implementation csum-file: introduce sha1file_checkpoint finish_tmp_packfile(): a helper function create_tmp_packfile(): a helper function write_pack_header(): a helper function Conflicts: pack.h
2011-11-16receive-pack, fetch-pack: reject bogus pack that records objects twiceLibravatar Junio C Hamano1-0/+4
When receive-pack & fetch-pack are run and store the pack obtained over the wire to a local repository, they internally run the index-pack command with the --strict option. Make sure that we reject incoming packfile that records objects twice to avoid spreading such a damage. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-10-28finish_tmp_packfile(): a helper functionLibravatar Junio C Hamano1-0/+31
Factor out a small logic out of the private write_pack_file() function in builtin/pack-objects.c. This changes the order of finishing multi-pack generation slightly. The code used to - adjust shared perm of temporary packfile - rename temporary packfile to the final name - update mtime of the packfile under the final name - adjust shared perm of temporary idxfile - rename temporary idxfile to the final name but because the helper does not want to do the mtime thing, the updated code does that step first and then all the rest. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-10-28create_tmp_packfile(): a helper functionLibravatar Junio C Hamano1-0/+10
Factor out a small logic out of the private write_pack_file() function in builtin/pack-objects.c Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-10-28write_pack_header(): a helper functionLibravatar Junio C Hamano1-0/+12
Factor out a small logic out of the private write_pack_file() function in builtin/pack-objects.c Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-02-27index-pack --verify: read anomalous offsets from v2 idx fileLibravatar Junio C Hamano1-1/+17
A pack v2 .idx file usually records offset using 64-bit representation only when the offset does not fit within 31-bit, but you can handcraft your .idx file to record smaller offset using 64-bit, storing all zero in the upper 4-byte. By inspecting the original idx file when running index-pack --verify, encode such low offsets that do not need to be in 64-bit but are encoded using 64-bit just like the original idx file so that we can still validate the pack/idx pair by comparing the idx file recomputed with the original. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-02-27write_idx_file: need_large_offset() helper functionLibravatar Junio C Hamano1-10/+19
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-02-27index-pack: --verifyLibravatar Junio C Hamano1-10/+16
Given an existing .pack file and the .idx file that describes it, this new mode of operation reads and re-index the packfile and makes sure the existing .idx file matches the result byte-for-byte. All the objects in the .pack file are validated during this operation as well. Unlike verify-pack, which visits each object described in the .idx file in the SHA-1 order, index-pack efficiently exploits the delta-chain to avoid rebuilding the objects that are used as the base of deltified objects over and over again while validating the objects, resulting in much quicker verification of the .pack file and its .idx file. This version however cannot verify a .pack/.idx pair with a handcrafted v2 index that uses 64-bit offset representation for offsets that would fit within 31-bit. You can create such an .idx file by giving a custom offset to --index-version option to the command. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-02-27write_idx_file: introduce a struct to hold idx customization optionsLibravatar Junio C Hamano1-6/+11
Remove two globals, pack_idx_default version and pack_idx_off32_limit, and place them in a pack_idx_option structure. Allow callers to pass it to write_idx_file() as a parameter. Adjust all callers to the API change. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-02-23move encode_in_pack_object_header() to a better placeLibravatar Nicolas Pitre1-0/+27
Commit 1b22b6c897 made duplicated versions of encode_header() into a common version called encode_in_pack_object_header(). There is however a better location that sha1_file.c for such a function though, as sha1_file.c contains nothing related to the creation of packs, and it is quite populated already. Also the comment that was moved to the header file should really remain near the function as it covers implementation details and provides no information about the actual function interface. Signed-off-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-01-22make "index-pack" a built-inLibravatar Linus Torvalds1-2/+2
This required some fairly trivial packfile function 'const' cleanup, since the builtin commands get a const char *argv[] array. Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-06-27Convert existing die(..., strerror(errno)) to die_errno()Libravatar Thomas Rast1-5/+5
Change calls to die(..., strerror(errno)) to use the new die_errno(). In the process, also make slight style adjustments: at least state _something_ about the function that failed (instead of just printing the pathname), and put paths in single quotes. Signed-off-by: Thomas Rast <trast@student.ethz.ch> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-02-25Merge branch 'jc/maint-1.6.0-pack-directory'Libravatar Junio C Hamano1-4/+2
* jc/maint-1.6.0-pack-directory: Make sure objects/pack exists before creating a new pack
2009-02-25Make sure objects/pack exists before creating a new packLibravatar Junio C Hamano1-4/+2
In a repository created with git older than f49fb35 (git-init-db: create "pack" subdirectory under objects, 2005-06-27), objects/pack/ directory is not created upon initialization. It was Ok because subdirectories are created as needed inside directories init-db creates, and back then, packfiles were recent invention. After the said commit, new codepaths started relying on the presense of objects/pack/ directory in the repository. This was exacerbated with 8b4eb6b (Do not perform cross-directory renames when creating packs, 2008-09-22) that moved the location temporary pack files are created from objects/ directory to objects/pack/ directory, because moving temporary to the final location was done carefully with lazy leading directory creation. Many packfile related operations in such an old repository can fail mysteriously because of this. This commit introduces two helper functions to make things work better. - odb_mkstemp() is a specialized version of mkstemp() to refactor the code and teach it to create leading directories as needed; - odb_pack_keep() refactors the code to create a ".keep" file while create leading directories as needed. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-10-02fix openssl headers conflicting with custom SHA1 implementationsLibravatar Nicolas Pitre1-15/+15
On ARM I have the following compilation errors: CC fast-import.o In file included from cache.h:8, from builtin.h:6, from fast-import.c:142: arm/sha1.h:14: error: conflicting types for 'SHA_CTX' /usr/include/openssl/sha.h:105: error: previous declaration of 'SHA_CTX' was here arm/sha1.h:16: error: conflicting types for 'SHA1_Init' /usr/include/openssl/sha.h:115: error: previous declaration of 'SHA1_Init' was here arm/sha1.h:17: error: conflicting types for 'SHA1_Update' /usr/include/openssl/sha.h:116: error: previous declaration of 'SHA1_Update' was here arm/sha1.h:18: error: conflicting types for 'SHA1_Final' /usr/include/openssl/sha.h:117: error: previous declaration of 'SHA1_Final' was here make: *** [fast-import.o] Error 1 This is because openssl header files are always included in git-compat-util.h since commit 684ec6c63c whenever NO_OPENSSL is not set, which somehow brings in <openssl/sha1.h> clashing with the custom ARM version. Compilation of git is probably broken on PPC too for the same reason. Turns out that the only file requiring openssl/ssl.h and openssl/err.h is imap-send.c. But only moving those problematic includes there doesn't solve the issue as it also includes cache.h which brings in the conflicting local SHA1 header file. As suggested by Jeff King, the best solution is to rename our references to SHA1 functions and structure to something git specific, and define those according to the implementation used. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2008-09-22Do not perform cross-directory renames when creating packsLibravatar Petr Baudis1-1/+1
A comment on top of create_tmpfile() describes caveats ('can have problems on various systems (FAT, NFS, Coda)') that should apply in this situation as well. This in the end did not end up solving any of my personal problems, but it might be a useful cleanup patch nevertheless. Signed-off-by: Petr Baudis <pasky@suse.cz> Acked-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-08-29fixup_pack_header_footer(): use nicely aligned buffer sizesLibravatar Nicolas Pitre1-3/+8
It should be more efficient to use nicely aligned buffer sizes, either for filesystem operations or SHA1 checksums. Also, using a relatively small nominal size might allow for the data to remain in L1 cache between both SHA1_Update() calls. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-08-29improve reliability of fixup_pack_header_footer()Libravatar Nicolas Pitre1-12/+59
Currently, this function has the potential to read corrupted pack data from disk and give it a valid SHA1 checksum. Let's add the ability to validate SHA1 checksum of existing data along the way, including before and after any arbitrary point in the pack. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-08-27index-pack: be careful after fixing up the header/footerLibravatar Linus Torvalds1-0/+1
The index-pack command, when processing a thin pack, fixed up the pack after-the-fact. It forgets to fsync the result, because it only did that in one path rather in all cases of fixup. This moves the fsync_or_die() to the fix-up routine itself, rather than doing it in one of the callers, so that all cases are covered. Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-06-25pack.indexversion config option now defaults to 2Libravatar Nicolas Pitre1-1/+1
As announced for 1.6.0. Git older than version 1.5.2 (or any other git version with this option set to 1) may revert to version 1 of the pack index by manually deleting all .idx files and recreating them using 'git index-pack'. Communication over the git native protocol is unaffected since the pack index is never transferred. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-05-31Make pack creation always fsync() the resultLibravatar Linus Torvalds1-1/+1
This means that we can depend on packs always being stable on disk, simplifying a lot of the object serialization worries. And unlike loose objects, serializing pack creation IO isn't going to be a performance killer. Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-05-03Cleanup xread() loops to use read_in_full()Libravatar Heikki Orsila1-6/+2
Signed-off-by: Heikki Orsila <heikki.orsila@iki.fi> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-11-02Merge branch 'np/progress'Libravatar Junio C Hamano1-1/+2
* np/progress: Show total transferred as part of throughput progress make sure throughput display gets updated even if progress doesn't move return the prune-packed progress display to the inner loop add throughput display to git-push add some copyright notice to the progress display code add throughput display to index-pack add throughput to progress display relax usage of the progress API make struct progress an opaque type prune-packed: don't call display_progress() for every file Stop displaying "Pack pack-$ID created." during git-gc Teach prune-packed to use the standard progress meter Change 'Deltifying objects' to 'Compressing objects' fix for more minor memory leaks fix const issues with some functions pack-objects.c: fix some global variable abuse and memory leaks pack-objects: no delta possible with only one object in the list cope with multiple line breaks within sideband progress messages more compact progress display
2007-10-17fix const issues with some functionsLibravatar Nicolas Pitre1-1/+2
Two functions, namely write_idx_file() and open_pack_file(), currently return a const pointer. However that pointer is either a copy of the first argument, or set to a malloc'd buffer when that first argument is null. In the later case it is wrong to qualify that pointer as const since ownership of the buffer is transferred to the caller to dispose of, and obviously the free() function is not meant to be passed const pointers. Making the return pointer not const causes a warning when the first argument is returned since that argument is also marked const. The correct thing to do is therefore to remove the const qualifiers, avoiding the need for ugly casts only to silence some warnings. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2007-09-19Refactor index-pack "keep $sha1" handling for reuseLibravatar Shawn O. Pearce1-0/+26
There is a subtle (but important) linkage between receive-pack and index-pack that allows index-pack to create a packfile but protect it from being deleted by a concurrent `git repack -a -d` operation. The linkage works by having index-pack mark the newly created pack with a ".keep" file and then it passes the SHA-1 name of that new packfile to receive-pack along its stdout channel. The receive-pack process must unkeep the packfile by deleting the .keep file, but can it can only do so after all elgible refs have been updated in the receiving repository. This ensures that the packfile is either kept or its objects are reachable, preventing a concurrent repacker from deleting the packfile before it can determine that its objects are actually needed by the repository. The new builtin-fetch code needs to perform the same actions if it choose to run index-pack rather than unpack-objects, so I am moving this code out to its own function where both receive-pack and fetch-pack are able to invoke it when necessary. The caller is responsible for deleting the returned ".keep" and freeing the path if the returned path is not NULL. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-08-14Use xmkstemp() instead of mkstemp()Libravatar Luiz Fernando N. Capitulino1-1/+1
xmkstemp() performs error checking and prints a standard error message when an error occur. Signed-off-by: Luiz Fernando N. Capitulino <lcapitulino@mandriva.com.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-06-02Unify write_index_file functionsLibravatar Geert Bosch1-0/+142
This patch unifies the write_index_file functions in builtin-pack-objects.c and index-pack.c. As the name "index" is overloaded in git, move in the direction of using "idx" and "pack idx" when refering to the pack index. There should be no change in functionality. Signed-off-by: Geert Bosch <bosch@gnat.com> Acked-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-15Fix signedness on return value from xread()Libravatar Johan Herland1-1/+1
The return value from xread() is ssize_t. Paolo Teti <paolo.teti@gmail.com> pointed out that in this case, the signed return value was assigned to an unsigned type (size_t). This patch fixes that. Signed-off-by: Johan Herland <johan@herland.net> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-02Create pack-write.c for common pack writing codeLibravatar Dana L. How1-0/+39
Include a generalized fixup_pack_header_footer() in this new file. Needed by git-repack --max-pack-size feature in a later patchset. [sp: Moved close(pack_fd) to callers, to support index-pack, and changed name to better indicate it is for packfiles.] Signed-off-by: Dana L. How <danahow@gmail.com> Signed-off-by: Shawn O. Pearce <spearce@spearce.org>