summaryrefslogtreecommitdiff
path: root/sha1_file.c
AgeCommit message (Collapse)AuthorFilesLines
2007-03-10Merge branch 'jc/fsck'Libravatar Junio C Hamano1-8/+19
* jc/fsck: fsck: exit with non-zero status upon errors unpack_sha1_file(): detect corrupt loose object files. fsck: fix broken loose object check.
2007-03-07Cast 64 bit off_t to 32 bit size_tLibravatar Shawn O. Pearce1-17/+23
Some systems have sizeof(off_t) == 8 while sizeof(size_t) == 4. This implies that we are able to access and work on files whose maximum length is around 2^63-1 bytes, but we can only malloc or mmap somewhat less than 2^32-1 bytes of memory. On such a system an implicit conversion of off_t to size_t can cause the size_t to wrap, resulting in unexpected and exciting behavior. Right now we are working around all gcc warnings generated by the -Wshorten-64-to-32 option by passing the off_t through xsize_t(). In the future we should make xsize_t on such problematic platforms detect the wrapping and die if such a file is accessed. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-07Use off_t when we really mean a file offset.Libravatar Shawn O. Pearce1-25/+29
Not all platforms have declared 'unsigned long' to be a 64 bit value, but we want to support a 64 bit packfile (or close enough anyway) in the near future as some projects are getting large enough that their packed size exceeds 4 GiB. By using off_t, the POSIX type that is declared to mean an offset within a file, we support whatever maximum file size the underlying operating system will handle. For most modern systems this is up around 2^60 or higher. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-07Use uint32_t for all packed object counts.Libravatar Shawn O. Pearce1-5/+5
As we permit up to 2^32-1 objects in a single packfile we cannot use a signed int to represent the object offset within a packfile, after 2^31-1 objects we will start seeing negative indexes and error out or compute bad addresses within the mmap'd index. This is a minor cleanup that does not introduce any significant logic changes. It is roach free. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-07General const correctness fixesLibravatar Shawn O. Pearce1-2/+1
We shouldn't attempt to assign constant strings into char*, as the string is not writable at runtime. Likewise we should always be treating unsigned values as unsigned values, not as signed values. Most of these are very straightforward. The only exception is the (unnecessary) xstrdup/free in builtin-branch.c for the detached head case. Since this is a user-level interactive type program and that particular code path is executed no more than once, I feel that the extra xstrdup call is well worth the easy elimination of this warning. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-07Fix mmap leak caused by reading bad indexes.Libravatar Shawn O. Pearce1-9/+16
If an index is corrupt, or is simply too new for us to understand, we were leaking the mmap that held the entire content of the index. This could be a considerable size on large projects, given that the index is at least 24 bytes * nr_objects. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-07Display the null SHA-1 as the base for an OBJ_OFS_DELTA.Libravatar Shawn O. Pearce1-0/+1
Because we are currently cheating and never supplying the delta base for an OBJ_OFS_DELTA we get a random SHA-1 in the delta base field. Instead lets clear the hash out so its at least all 0's. This is somewhat more obvious that something fishy is going on, like we don't actually have the SHA-1 of the base handy. :) Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-05unpack_sha1_file(): detect corrupt loose object files.Libravatar Junio C Hamano1-8/+19
We did not detect broken loose object files, either when underlying inflate() signalled the breakage, nor inflate() finished and we had garbage trailing at the end. We do better now. We also make unpack_sha1_file() a static function to sha1_file.c, since it is not used by anybody outside. Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-28index_fd(): convert blob only if it is a regular file.Libravatar Junio C Hamano1-1/+1
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-28index_fd(): pass optional path parameter as hint for blob conversionLibravatar Junio C Hamano1-3/+3
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-28index_fd(): use enum object_type instead of type name string.Libravatar Junio C Hamano1-6/+7
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-27convert object type handling from a string to a numberLibravatar Nicolas Pitre1-90/+82
We currently have two parallel notation for dealing with object types in the code: a string and a numerical value. One of them is obviously redundent, and the most used one requires more stack space and a bunch of strcmp() all over the place. This is an initial step for the removal of the version using a char array found in object reading code paths. The patch is unfortunately large but there is no sane way to split it in smaller parts without breaking the system. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-27formalize typename(), and add its reverse type_from_string()Libravatar Nicolas Pitre1-14/+5
Sometime typename() is used, sometimes type_names[] is accessed directly. Let's enforce typename() all the time which allows for validating the type. Also let's add a function to go from a name to a type and use it instead of manual memcpy() when appropriate. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-27sha1_file.c: don't ignore an error condition in sha1_loose_object_info()Libravatar Nicolas Pitre1-1/+1
Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-27sha1_file.c: cleanup "offset" usageLibravatar Nicolas Pitre1-59/+51
First there are too many offsets there and it is getting confusing. So 'offset' is now 'curpos' to distinguish from other offsets like 'obj_offset'. Then structures like x = foo(x, &y) are now done as y = foo(&x). It looks more natural that the result y be returned directly and x be passed as reference to be updated in place. This has the effect of reducing some line length and removing a few, needing a bit less stack space, and it even reduces the compiled code size. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-27sha1_file.c: cleanup hdr usageLibravatar Nicolas Pitre1-12/+12
Let's have hdr be a simple char pointer/array when possible, and let's reduce its storage to 32 bytes. Especially for sha1_loose_object_info() where 128 bytes is way excessive and wastes extra CPU cycles inflating. The object type is already restricted to 10 bytes in parse_sha1_header() and the size, even if it is 64 bits, will fit in 20 decimal numbers. So 32 bytes is plenty. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-22Merge branches 'lt/crlf' and 'jc/apply-config'Libravatar Junio C Hamano1-2/+21
* lt/crlf: Teach core.autocrlf to 'git apply' t0020: add test for auto-crlf Make AutoCRLF ternary variable. Lazy man's auto-CRLF * jc/apply-config: t4119: test autocomputing -p<n> for traditional diff input. git-apply: guess correct -p<n> value for non-git patches. git-apply: notice "diff --git" patch again Fix botched "leak fix" t4119: add test for traditional patch and different p_value apply: fix memory leak in prefix_one() git-apply: require -p<n> when working in a subdirectory. git-apply: do not lose cwd when run from a subdirectory. Teach 'git apply' to look at $HOME/.gitconfig even outside of a repository Teach 'git apply' to look at $GIT_DIR/config
2007-02-15pretend-sha1: grave bugfix.Libravatar Junio C Hamano1-1/+3
We stashed away objects that we pretend to have, but did not save the actual data. Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-14sha1_file.c: Round the mmap offset to half the window size.Libravatar Alexandre Julliard1-5/+3
This ensures that a given area is mapped at most twice, and greatly reduces the virtual address space usage. Signed-off-by: Alexandre Julliard <julliard@winehq.org> Acked-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-14Lazy man's auto-CRLFLibravatar Linus Torvalds1-2/+21
It currently does NOT know about file attributes, so it does its conversion purely based on content. Maybe that is more in the "git philosophy" anyway, since content is king, but I think we should try to do the file attributes to turn it off on demand. Anyway, BY DEFAULT it is off regardless, because it requires a [core] AutoCRLF = true in your config file to be enabled. We could make that the default for Windows, of course, the same way we do some other things (filemode etc). But you can actually enable it on UNIX, and it will cause: - "git update-index" will write blobs without CRLF - "git diff" will diff working tree files without CRLF - "git checkout" will write files to the working tree _with_ CRLF and things work fine. Funnily, it actually shows an odd file in git itself: git clone -n git test-crlf cd test-crlf git config core.autocrlf true git checkout git diff shows a diff for "Documentation/docbook-xsl.css". Why? Because we have actually checked in that file *with* CRLF! So when "core.autocrlf" is true, we'll always generate a *different* hash for it in the index, because the index hash will be for the content _without_ CRLF. Is this complete? I dunno. It seems to work for me. It doesn't use the filename at all right now, and that's probably a deficiency (we could certainly make the "is_binary()" heuristics also take standard filename heuristics into account). I don't pass in the filename at all for the "index_fd()" case (git-update-index), so that would need to be passed around, but this actually works fine. NOTE NOTE NOTE! The "is_binary()" heuristics are totally made-up by yours truly. I will not guarantee that they work at all reasonable. Caveat emptor. But it _is_ simple, and it _is_ safe, since it's all off by default. The patch is pretty simple - the biggest part is the new "convert.c" file, but even that is really just basic stuff that anybody can write in "Teaching C 101" as a final project for their first class in programming. Not to say that it's bug-free, of course - but at least we're not talking about rocket surgery here. Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-13Mark places that need blob munging later for CRLF conversion.Libravatar Linus Torvalds1-0/+1
Here's a patch that I think we can merge right now. There may be other places that need this, but this at least points out the three places that read/write working tree files for git update-index, checkout and diff respectively. That should cover a lot of it [jc: git-apply uses an entirely different codepath both for reading and writing]. Some day we can actually implement it. In the meantime, this points out a place for people to start. We *can* even start with a really simple "we do CRLF conversion automatically, regardless of filename" kind of approach, that just look at the data (all three cases have the _full_ file data already in memory) and says "ok, this is text, so let's convert to/from DOS format directly". THAT somebody can write in ten minutes, and it would already make git much nicer on a DOS/Windows platform, I suspect. And it would be totally zero-cost if you just make it a config option (but please make it dynamic with the _default_ just being 0/1 depending on whether it's UNIX/Windows, just so that UNIX people can _test_ it easily). Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-05Add pretend_sha1_file() interface.Libravatar Junio C Hamano1-0/+57
The new interface allows an application to temporarily hash a small number of objects and pretend that they are available in the object store without actually writing them. Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-03Assorted typo fixesLibravatar Pavel Roskin1-1/+1
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-02Don't leak file descriptors from unavailable pack files.Libravatar Shawn O. Pearce1-1/+16
If open_packed_git failed it may have been because the packfile actually exists and is readable, but some sort of verification did not pass. In this case open_packed_git left pack_fd filled in, as the file descriptor is valid. We don't want to leak the file descriptor, nor do we want to allow someone in the future to use this packed_git. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-01Don't find objects in packs which aren't available anymore.Libravatar Shawn O. Pearce1-0/+12
Matthias Lederhofer identified a race condition where a Git reader process was able to locate an object in a packed_git index, but was then preempted while a `git repack -a -d` ran and completed. By the time the reader was able to seek in the packfile to get the object data, the packfile no longer existed on disk. In this particular case the reader process did not attempt to open the packfile before it was deleted, so it did not already have the pack_fd field popuplated. With the packfile itself gone, there was no way for the reader to open it and fetch the data. I'm fixing the race condition by teaching find_pack_entry to ignore a packed_git whose packfile is not currently open and which cannot be opened. If none of the currently known packs can supply the object, we will return 0 and the caller will decide the object is not available. If this is the first attempt at finding an object, the caller will reprepare_packed_git and try again. If it was the second attempt, the caller will typically return NULL back, and an error message about a missing object will be reported. This patch does not address the situation of a reader which is being starved out by a tight sequence of `git repack -a -d` runs. In this particular case the reader will try twice, probably fail both times, and declare the object in question cannot be found. As it is highly unlikely that a real world `git repack -a -d` can complete faster than a reader can open a packfile, so I don't think this is a huge concern. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-01Refactor open_packed_git to return an error code.Libravatar Shawn O. Pearce1-15/+16
Because I want to reuse open_packed_git in a context where I don't want the process to die if the packfile in question is bogus, I'm changing its behavior to return error("...") rather than die("...") when it detects something is wrong with the packfile it was given. Right now we still must die out of use_pack should open_packed_git fail, as none of use_pack's callers are prepared to handle a failure from that function. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-01Correct comment in prepare_packed_git_one.Libravatar Shawn O. Pearce1-1/+4
After staring at the comment and the associated for loop, I realized the comment was completely bogus. The section of code its talking about is trying to avoid duplicate mapping of the same packfile. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-01Cleanup prepare_packed_git_one to reuse install_packed_git.Libravatar Shawn O. Pearce1-2/+1
There is little point in having the linked list insertion code appearing in install_packed_git, and then again just 30 lines further down in the same file. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-01-24Refactor the pack header reading function out of receive-pack.cLibravatar Junio C Hamano1-0/+21
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-01-22sha1_file.c: Avoid multiple calls to find_pack_entry().Libravatar Peter Eriksen1-10/+9
We used to call find_pack_entry() twice from read_sha1_file() in order to avoid printing an error message, when the object did not exist. This is fixed by moving the call to error() to the only place it really could be called. Signed-off-by: Peter Eriksen <s022018@student.dtu.dk> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-01-18Use fixed-size integers for .idx file I/OLibravatar Junio C Hamano1-3/+3
This attempts to finish what Simon started in the previous commit. Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-01-17Document pack .idx file format upgrade strategy.Libravatar Shawn O. Pearce1-3/+14
Way back when Junio developed the 64 bit index topic he came up with a means of changing the .idx file format so that older Git clients would recognize that they don't understand the file and refuse to read it, while newer clients could tell the difference between the old-style and new-style .idx files. Unfortunately this wasn't recorded anywhere. This change documents how we might go about changing the .idx file format by using a special signature in the first four bytes. Credit (and possible blame) goes completely to Junio for thinking up this technique. The change also modifies the error message of the current Git code so that users get a recommendation to upgrade their Git software should this version or later encounter a new-style .idx which it cannot process. We already do this for the .pack files, but since we usually process the .idx files first its important that these files are recognized and encourage an upgrade. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-01-14Remove read_or_die in favor of better error messages.Libravatar Shawn O. Pearce1-2/+4
Originally I introduced read_or_die for the purpose of reading the pack header and trailer, and I was too lazy to print proper error messages. Linus Torvalds <torvalds@osdl.org>: > For a read error, at the very least you have to say WHICH FILE > couldn't be read, because it's usually a matter of some file just > being too short, not some system-wide problem. and of course Linus is right. Make it so. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-01-11Clean up write_in_full() usersLibravatar Linus Torvalds1-8/+1
With the new-and-improved write_in_full() semantics, where a partial write simply always returns a real error (and always sets 'errno' when that happens, including for the disk full case), a lot of the callers of write_in_full() were just unnecessarily complex. In particular, there's no reason to ever check for a zero length or return: if the length was zero, we'll return zero, otherwise, if a disk full resulted in the actual write() system call returning zero the write_in_full() logic would have correctly turned that into a negative return value, with 'errno' set to ENOSPC. I really wish every "write_in_full()" user would just check against "<0" now, but this fixes the nasty and stupid ones. Signed-off-by: Linus Torvalds <torvalds@osdl.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-01-11Avoid errors and warnings when attempting to do I/O on zero bytesLibravatar Eric Wong1-0/+2
Unfortunately, while {read,write}_in_full do take into account zero-sized reads/writes; their die and whine variants do not. I have a repository where there are zero-sized files in the history that was triggering these things. Signed-off-by: Eric Wong <normalperson@yhbt.net> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-01-09Fix warnings in sha1_file.c - use C99 printf format if availableLibravatar Pavel Roskin1-4/+11
Signed-off-by: Pavel Roskin <proski@gnu.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-01-08short i/o: fix calls to write to use xwrite or write_in_fullLibravatar Andy Whitcroft1-13/+6
We have a number of badly checked write() calls. Often we are expecting write() to write exactly the size we requested or fail, this fails to handle interrupts or short writes. Switch to using the new write_in_full(). Otherwise we at a minimum need to check for EINTR and EAGAIN, where this is appropriate use xwrite(). Note, the changes to config handling are much larger and handled in the next patch in the sequence. Signed-off-by: Andy Whitcroft <apw@shadowen.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-01-08short i/o: fix calls to read to use xread or read_in_fullLibravatar Andy Whitcroft1-1/+1
We have a number of badly checked read() calls. Often we are expecting read() to read exactly the size we requested or fail, this fails to handle interrupts or short reads. Add a read_in_full() providing those semantics. Otherwise we at a minimum need to check for EINTR and EAGAIN, where this is appropriate use xread(). Signed-off-by: Andy Whitcroft <apw@shadowen.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29mmap: set FD_CLOEXEC for file descriptors we keep open for mmap()Libravatar Junio C Hamano1-0/+11
I do not have any proof that this matters to any existing problems I am seeing, but I do not think of any reason not to do this. Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29Replace mmap with xmmap, better handling MAP_FAILED.Libravatar Shawn O. Pearce1-13/+5
In some cases we did not even bother to check the return value of mmap() and just assume it worked. This is bad, because if we are out of virtual address space the kernel returned MAP_FAILED and we would attempt to dereference that address, segfaulting without any real error output to the user. We are replacing all calls to mmap() with xmmap() and moving all MAP_FAILED checking into that single location. If a mmap call fails we try to release enough least-recently-used pack windows to possibly succeed, then retry the mmap() attempt. If we cannot mmap even after releasing pack memory then we die() as none of our callers have any reasonable recovery strategy for a failed mmap. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29Release pack windows before reporting out of memory.Libravatar Shawn O. Pearce1-0/+7
If we are about to fail because this process has run out of memory we should first try to automatically control our appetite for address space by releasing enough least-recently-used pack windows to gain back enough memory such that we might actually be able to meet the current allocation request. This should help users who have fairly large repositories but are working on systems with relatively small virtual address space. Many times we see reports on the mailing list of these users running out of memory during various Git operations. Dynamically decreasing the amount of pack memory used when the demand for heap memory is increasing is an intelligent solution to this problem. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29Create pack_report() as a debugging aid.Libravatar Shawn O. Pearce1-0/+31
Much like the alloc_report() function can be useful to report on object allocation statistics while debugging the new pack_report() function can be useful to report on the behavior of the mmap window code used for packfile access. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29Support unmapping windows on 'temporary' packfiles.Libravatar Shawn O. Pearce1-16/+28
If a command opens a packfile for only temporary access and does not install the struct packed_git* into the global packed_git list then we are unable to unmap any inactive windows within that packed_git, causing the overall process to exceed core.packedGitLimit. We cannot force the callers to install their temporary packfile into the packed_git chain as doing so would allow that (possibly corrupt but currently being verified) temporary packfile to become part of the local ODB, which may allow it to be considered for object resolution when it may not actually be a valid packfile. So to support unmapping the windows of these temporary packfiles we also scan the windows of the struct packed_git which was supplied to use_pack(). Since commands only work with one temporary packfile at a time scanning the one supplied to use_pack() and all packs installed into packed_git should cover everything available in memory. We also have to be careful to not close the file descriptor of the packed_git which was handed to use_pack() when all of that packfile's windows have been unmapped, as we are already past the open call that would open the packfile and need the file descriptor to be ready for mmap() after unuse_one_window returns. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29Improve error message when packfile mmap fails.Libravatar Shawn O. Pearce1-1/+3
If we are unable to mmap the a region of the packfile with the mmap() system call there may be a good reason why, such as a closed file descriptor or out of address space. Reporting the system level error message can help to debug such problems. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29Fully activate the sliding window pack access.Libravatar Shawn O. Pearce1-13/+53
This finally turns on the sliding window behavior for packfile data access by mapping limited size windows and chaining them under the packed_git->windows list. We consider a given byte offset to be within the window only if there would be at least 20 bytes (one hash worth of data) accessible after the requested offset. This range selection relates to the contract that use_pack() makes with its callers, allowing them to access one hash or one object header without needing to call use_pack() for every byte of data obtained. In the worst case scenario we will map the same page of data twice into memory: once at the end of one window and once again at the start of the next window. This duplicate page mapping will happen only when an object header or a delta base reference is spanned over the end of a window and is always limited to just one page of duplication, as no sane operating system will ever have a page size smaller than a hash. I am assuming that the possible wasted page of virtual address space is going to perform faster than the alternatives, which would be to copy the object header or ref delta into a temporary buffer prior to parsing, or to check the window range on every byte during header parsing. We may decide to revisit this decision in the future since this is just a gut instinct decision and has not actually been proven out by experimental testing. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29Unmap individual windows rather than entire files.Libravatar Shawn O. Pearce1-15/+30
To support multiple windows per packfile we need to unmap only one window at a time from that packfile, leaving any other windows in place and available for reference. We treat all windows from all packfiles equally; the least recently used, not-in-use window across all packfiles will always be closed first. If we have unmapped all windows in a packfile then we can also close the packfile's file descriptor as its possible we won't need to map any window from that file in the near future. This decision about when to close the pack file descriptor may need to be revisited in the future after additional testing on several different platforms can be performed. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29Document why header parsing won't exceed a window.Libravatar Shawn O. Pearce1-4/+12
When we parse the object header or the delta base reference we don't bother to loop over use_pack() calls. The reason we don't need to bother with calling use_pack for each byte accessed is that use_pack will always promise us at least 20 bytes (really the hash size) after the offset. This promise from use_pack simplifies a lot of code in the header parsing logic, as well as helps out the zlib library by ensuring there's always some data for it to consume during an inflate call. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29Loop over pack_windows when inflating/accessing data.Libravatar Shawn O. Pearce1-7/+15
When multiple mmaps start getting used for all pack file access it is not possible to get all data associated with a specific object in one contiguous memory region. This limitation prevents simply passing a single address and length to SHA1_Update or to inflate. Instead we need to loop until we have processed all data of interest. As we loop over the data we are always interested in reusing the same window 'cursor', as the prior window will no longer be of any use to us. This allows the use_pack() call to automatically decrement the use count of the prior window before setting up access for us to the next window. Within each loop we need to make use of the available length output parameter of use_pack() to tell us how many bytes are available in the current memory region, as we cannot tell otherwise. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29Replace use_packed_git with window cursors.Libravatar Shawn O. Pearce1-46/+64
Part of the implementation concept of the sliding mmap window for pack access is to permit multiple windows per pack to be mapped independently. Since the inuse_cnt is associated with the mmap and not with the file, this value is in struct pack_window and needs to be incremented/decremented for each pack_window accessed by any code. To faciliate that implementation we need to replace all uses of use_packed_git() and unuse_packed_git() with a different API that follows struct pack_window objects rather than struct packed_git. The way this works is when we need to start accessing a pack for the first time we should setup a new window 'cursor' by declaring a local and setting it to NULL: struct pack_windows *w_curs = NULL; To obtain the memory region which contains a specific section of the pack file we invoke use_pack(), supplying the address of our current window cursor: unsigned int len; unsigned char *addr = use_pack(p, &w_curs, offset, &len); the returned address `addr` will be the first byte at `offset` within the pack file. The optional variable len will also be updated with the number of bytes remaining following the address. Multiple calls to use_pack() with the same window cursor will update the window cursor, moving it from one window to another when necessary. In this way each window cursor variable maintains only one struct pack_window inuse at a time. Finally before exiting the scope which originally declared the window cursor we must invoke unuse_pack() to unuse the current window (which may be different from the one that was first obtained from use_pack): unuse_pack(&w_curs); This implementation is still not complete with regards to multiple windows, as only one window per pack file is supported right now. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29Refactor how we open pack files to prepare for multiple windows.Libravatar Shawn O. Pearce1-40/+46
To efficiently support mmaping of multiple regions of the same pack file we want to keep the pack's file descriptor open while we are actively working with that pack. So we are now keeping that file descriptor in packed_git.pack_fd and closing it only after we unmap the last window. This is going to increase the number of file descriptors that are in use at once, however that will be bounded by the total number of pack files present and therefore should not be very high. It is a small tradeoff which we may need to revisit after some testing can be done on various repositories and systems. For code clarity we also want to seperate out the implementation of how we open a pack file from the implementation which locates a suitable window (or makes a new one) from the given pack file. Since this is a rather large delta I'm taking advantage of doing it now, in a fairly isolated change. When we open a pack file we need to examine the header and trailer without having a mmap in place, as we may only need to mmap the middle section of this particular pack. Consequently the verification code has been refactored to make use of the new read_or_die function. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>