summaryrefslogtreecommitdiff
path: root/builtin-pack-objects.c
AgeCommit message (Collapse)AuthorFilesLines
2008-03-13pack-objects: proper pack time stamping with --max-pack-sizeLibravatar Nicolas Pitre1-0/+26
Runtime pack access is done in the pack file mtime order since recent packs are more likely to contain frequently used objects than old packs. However the --max-pack-size option can produce multiple packs with mtime in the reversed order as newer objects are always written first. Let's modify mtime of later pack files (when any) so they appear older than preceding ones when a repack creates multiple packs. Signed-off-by: Nicolas Pitre <nico@cam.org>
2008-03-04git-pack-objects: Automatically pack annotated tags if object was packedLibravatar Shawn O. Pearce1-2/+22
The new option "--include-tag" allows the caller to request that any annotated tag be included into the packfile if the object the tag references was also included as part of the packfile. This option can be useful on the server side of a native git transport, where the server knows what commits it is including into a packfile to update the client. If new annotated tags have been introduced then we can also include them in the packfile, saving the client from needing to request them through a second connection. This change only introduces the backend option and provides a test. Protocol extensions to make this useful in fetch-pack/upload-pack are still necessary to activate the logic during transport. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-03-02Merge branch 'np/verify-pack'Libravatar Junio C Hamano1-150/+10
* np/verify-pack: add storage size output to 'git verify-pack -v' fix unimplemented packed_object_info_detail() features make verify_one_pack() a bit less wrong wrt packed_git structure factorize revindex code out of builtin-pack-objects.c Conflicts: Makefile
2008-03-02Merge branch 'mk/maint-parse-careful'Libravatar Junio C Hamano1-1/+0
* mk/maint-parse-careful: receive-pack: use strict mode for unpacking objects index-pack: introduce checking mode unpack-objects: prevent writing of inconsistent objects unpack-object: cache for non written objects add common fsck error printing function builtin-fsck: move common object checking code to fsck.c builtin-fsck: reports missing parent commits Remove unused object-ref code builtin-fsck: move away from object-refs to fsck_walk add generic, type aware object chain walker Conflicts: Makefile builtin-fsck.c
2008-03-01factorize revindex code out of builtin-pack-objects.cLibravatar Nicolas Pitre1-150/+10
No functional change. This is needed to fix verify-pack in a later patch. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-27Merge branch 'jm/free'Libravatar Junio C Hamano1-2/+1
* jm/free: Avoid unnecessary "if-before-free" tests. Conflicts: builtin-branch.c
2008-02-26Revert "pack-objects: Print a message describing the number of threads for ↵Libravatar Junio C Hamano1-3/+0
packing" This reverts commit 6c723f5e6bc579e06a904874f1ceeb8ff2b5a17c. The additional message may be interesting for git developers, but not useful for the end users, and clutters the output.
2008-02-25Remove unused object-ref codeLibravatar Martin Koegler1-1/+0
Signed-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-23pack-objects: Print a message describing the number of threads for packingLibravatar Brandon Casey1-0/+3
Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-23pack-objects: Add runtime detection of online CPU'sLibravatar Andreas Ericsson1-4/+10
Packing objects can be done in parallell nowadays, but it's only done if the config option pack.threads is set to a value above 1. Because of that, the code-path used is often not the most optimal one. This patch adds a routine to detect the number of online CPU's at runtime (online_cpus()). When pack.threads (or --threads=) is given a value of 0, the number of threads is set to the number of online CPU's. This feature is also documented. As per Nicolas Pitre's recommendations, the default is still to run pack-objects single-threaded unless explicitly activated, either by configuration or by command line parameter. The routine online_cpus() is a rework of "numcpus.c", written by one Philip Willoughby <pgw99@doc.ic.ac.uk>. numcpus.c is in the public domain and can presently be downloaded from http://csgsoft.doc.ic.ac.uk/numcpus/ Signed-off-by: Andreas Ericsson <ae@op5.se> Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-22Avoid unnecessary "if-before-free" tests.Libravatar Jim Meyering1-2/+1
This change removes all obvious useless if-before-free tests. E.g., it replaces code like this: if (some_expression) free (some_expression); with the now-equivalent: free (some_expression); It is equivalent not just because POSIX has required free(NULL) to work for a long time, but simply because it has worked for so long that no reasonable porting target fails the test. Here's some evidence from nearly 1.5 years ago: http://www.winehq.org/pipermail/wine-patches/2006-October/031544.html FYI, the change below was prepared by running the following: git ls-files -z | xargs -0 \ perl -0x3b -pi -e \ 's/\bif\s*\(\s*(\S+?)(?:\s*!=\s*NULL)?\s*\)\s+(free\s*\(\s*\1\s*\))/$2/s' Note however, that it doesn't handle brace-enclosed blocks like "if (x) { free (x); }". But that's ok, since there were none like that in git sources. Beware: if you do use the above snippet, note that it can produce syntactically invalid C code. That happens when the affected "if"-statement has a matching "else". E.g., it would transform this if (x) free (x); else foo (); into this: free (x); else foo (); There were none of those here, either. If you're interested in automating detection of the useless tests, you might like the useless-if-before-free script in gnulib: [it *does* detect brace-enclosed free statements, and has a --name=S option to make it detect free-like functions with different names] http://git.sv.gnu.org/gitweb/?p=gnulib.git;a=blob;f=build-aux/useless-if-before-free Addendum: Remove one more (in imap-send.c), spotted by Jean-Luc Herren <jlh@gmx.ch>. Signed-off-by: Jim Meyering <meyering@redhat.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-17check return code of prepare_revision_walkLibravatar Martin Koegler1-1/+2
A failure in prepare_revision_walk can be caused by a not parseable object. Signed-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-13Merge branch 'maint'Libravatar Junio C Hamano1-11/+2
* maint: config: add test cases for empty value and no value config variables. cvsimport: have default merge regex also match beginning of commit message git clone -s documentation: force a new paragraph for the NOTE status: suggest "git rm --cached" to unstage for initial commit Protect get_author_ident_from_commit() from filenames in work tree upload-pack: Initialize the exec-path. bisect: use verbatim commit subject in the bisect log git-cvsimport.txt: fix '-M' description. Revert "pack-objects: only throw away data during memory pressure"
2008-02-12Revert "pack-objects: only throw away data during memory pressure"Libravatar Junio C Hamano1-11/+2
This reverts commit 9c2174350cc0ae0f6bad126e15fe1f9f044117ab. Nico analyzed and found out that this does not really help, and I agree with it. By the time this gets into action and data is actively thrown away, performance simply goes down the drain due to the data constantly being reloaded over and over and over and over and over and over again, to the point of virtually making no relative progress at all. The previous behavior of enforcing the memory limit by dynamically shrinking the window size at least had the effect of allowing some kind of progress, even if the end result wouldn't be optimal. And that's the whole point behind this memory limiting feature: allowing some progress to be made when resources are too limited to let the repack go unbounded.
2008-02-11Merge branch 'maint'Libravatar Junio C Hamano1-2/+11
* maint: (35 commits) config.c: guard config parser from value=NULL builtin-log.c: guard config parser from value=NULL imap-send.c: guard config parser from value=NULL wt-status.c: guard config parser from value=NULL setup.c: guard config parser from value=NULL remote.c: guard config parser from value=NULL merge-recursive.c: guard config parser from value=NULL http.c: guard config parser from value=NULL help.c: guard config parser from value=NULL git.c: guard config parser from value=NULL diff.c: guard config parser from value=NULL convert.c: guard config parser from value=NULL connect.c: guard config parser from value=NULL builtin-tag.c: guard config parser from value=NULL builtin-show-branch.c: guard config parser from value=NULL builtin-reflog.c: guard config parser from value=NULL builtin-log.c: guard config parser from value=NULL builtin-config.c: guard config parser from value=NULL builtin-commit.c: guard config parser from value=NULL builtin-branch.c: guard config parser from value=NULL ...
2008-02-11pack-objects: only throw away data during memory pressureLibravatar Martin Koegler1-2/+11
If pack-objects hit the memory limit, it deletes objects from the delta window. This patch make it only delete the data, which is recomputed, if needed again. Signed-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at> Acked-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-09Introduce the config variable pack.packSizeLimitLibravatar Johannes Schindelin1-1/+9
"git pack-objects" has the option --max-pack-size to limit the file size of the packs to a certain amount of bytes. On platforms where the pack file size is limited by filesystem constraints, it is easy to forget this option, and this option does not exist for "git gc" to begin with. So introduce a config variable to set the default maximum, but make this overrideable by the command line. Suggested by Tor Arvid Lund. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-01-21pack-objects: Fix segfault when object count is less than thread countLibravatar Nicolas Pitre1-1/+2
When partitioning the work amongst threads, dividing the number of objects by the number of threads may return 0 when there are less objects than threads; this will cause the subsequent code to segfault when accessing list[sub_size-1]. Allow some threads to have zero objects to work on instead of barfing, while letting others to have more. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-01-10pack-objects: remove redundant and wrong call to deflateEnd()Libravatar Junio C Hamano1-1/+1
We somehow called deflateEnd() on a stream that we have called deflateEnd() on already. In fact, the second deflateEnd() has always been returning Z_STREAM_ERROR. We just never checked the error return from that particular deflateEnd(). The first one returns 0 for success. We might want to tighten the check even more to check that. Noticed by Marco. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-01-04Don't access line[-1] for a zero-length "line" from fgets.Libravatar Jim Meyering1-1/+1
A NUL byte at beginning of file, or just after a newline would provoke an invalid buf[-1] access in a few places. * builtin-grep.c (cmd_grep): Don't access buf[-1]. * builtin-pack-objects.c (get_object_list): Likewise. * builtin-rev-list.c (read_revisions_from_stdin): Likewise. * bundle.c (read_bundle_header): Likewise. * server-info.c (read_pack_info_file): Likewise. * transport.c (insert_packed_refs): Likewise. Signed-off-by: Jim Meyering <meyering@redhat.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-17Plug a resource leak in threaded pack-objects code.Libravatar Johannes Sixt1-2/+2
A mutex and a condition variable is allocated for each thread and torn down when the thread terminates. However, for certain workloads it can happen that some threads are actually not started at all. In this case we would leak the mutex and condition variable. Now we allocate them only for those threads that are actually started. Signed-off-by: Johannes Sixt <johannes.sixt@telecom.at> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-16threaded pack-objects: Use condition variables for thread communication.Libravatar Johannes Sixt1-50/+79
In the threaded pack-objects code the main thread and the worker threads must mutually signal that they have assigned a new pack of work or have completed their work, respectively. Previously, the code used mutexes that were locked in one thread and unlocked from a different thread, which is bogus (and happens to work on Linux). Here we rectify the implementation by using condition variables: There is one condition variable on which the main thread waits until a thread requests new work; and each worker thread has its own condition variable on which it waits until it is assigned new work or signaled to terminate. As a cleanup, the worker threads are spawned only after the initial work packages have been assigned. Signed-off-by: Johannes Sixt <johannes.sixt@telecom.at> Acked-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-10pack-objects: more threaded load balancing fix with often changed pathsLibravatar Nicolas Pitre1-0/+10
The code that splits the object list amongst work threads tries to do so on "path" boundaries not to prevent good delta matches. However, in some cases, a few paths may largely dominate the hash distribution and it is not possible to have good load balancing without ignoring those boundaries. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-08pack-objects: fix threaded load balancingLibravatar Nicolas Pitre1-32/+85
The current method consists of a master thread serving chunks of objects to work threads when they're done with their previous chunk. The issue is to determine the best chunk size: making it too large creates poor load balancing, while making it too small has a negative effect on pack size because of the increased number of chunk boundaries and poor delta window utilization. This patch implements a completely different approach by initially splitting the work in large chunks uniformly amongst all threads, and whenever a thread is done then it steals half of the remaining work from another thread with the largest amount of unprocessed objects. This has the advantage of greatly reducing the number of chunk boundaries with an almost perfect load balancing. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-08pack-objects: reverse the delta search sort listLibravatar Nicolas Pitre1-20/+21
It is currently sorted and then walked backward. Not only this doesn't feel natural for my poor brain, but it would make the next patch less obvious as well. So reverse the sort order, and reverse the list walking direction, which effectively produce the exact same end result as before. Also bring the relevant comment nearer the actual code and adjust it accordingly, with minor additional clarifications. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-08pack-objects: fix delta cache size accountingLibravatar Nicolas Pitre1-5/+5
The wrong value was substracted from delta_cache_size when replacing a cached delta, as trg_entry->delta_size was used after the old size had been replaced by the new size. Noticed by Linus. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-11-14Fix rev-list when showing objects involving submodulesLibravatar Linus Torvalds1-1/+1
The function mark_tree_uninteresting() assumed that the tree entries are blob when they are not trees. This is not so. Since we do not traverse into submodules (yet), the gitlinks should be ignored. In general, we should try to start moving away from using the "S_ISLNK()" like things for internal git state. It was a mistake to just assume the numbers all were same across all systems in the first place. This implementation converts to the "object_type", and then uses a case statement. Noticed by Ilari on IRC. Test script taken from an earlier version by Dscho. Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-11-04Merge branch 'np/pack'Libravatar Junio C Hamano1-7/+12
* np/pack: pack-objects: get rid of an ugly cast make the pack index version configurable Conflicts: builtin-pack-objects.c
2007-11-02pack-objects: get rid of an ugly castLibravatar Nicolas Pitre1-6/+6
... when calling write_idx_file(). Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-11-02make the pack index version configurableLibravatar Nicolas Pitre1-0/+6
It is a good idea to use pack index version 2 all the time since it has proper protection against propagation of certain pack corruptions when repacking which is not possible with index version 1, as demonstrated in test t5302. Hence this config option. The default is still pack index version 1. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-10-30add throughput display to git-pushLibravatar Nicolas Pitre1-1/+1
This one triggers only when git-pack-objects is called with --all-progress and --stdout which is the combination used by git-push. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-10-30relax usage of the progress APILibravatar Nicolas Pitre1-12/+6
Since it is now OK to pass a null pointer to display_progress() and stop_progress() resulting in a no-op, then we can simplify the code and remove a bunch of lines by not making those calls conditional all the time. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-10-30make struct progress an opaque typeLibravatar Nicolas Pitre1-8/+8
This allows for better management of progress "object" existence, as well as making the progress display implementation more independent from its callers. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-10-18Change 'Deltifying objects' to 'Compressing objects'Libravatar Shawn O. Pearce1-1/+1
Recently I was referred to the Grammar Police as the git-pack-objects progress message 'Deltifying %u objects' is considered to be not proper English to at least some small but vocal segment of the English speaking population. Techncially we are applying delta compression to these objects at this stage, so the new term is slightly more acceptable to the Grammar Police but is also just as correct. Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2007-10-17fix const issues with some functionsLibravatar Nicolas Pitre1-1/+1
Two functions, namely write_idx_file() and open_pack_file(), currently return a const pointer. However that pointer is either a copy of the first argument, or set to a malloc'd buffer when that first argument is null. In the later case it is wrong to qualify that pointer as const since ownership of the buffer is transferred to the caller to dispose of, and obviously the free() function is not meant to be passed const pointers. Making the return pointer not const causes a warning when the first argument is returned since that argument is also marked const. The correct thing to do is therefore to remove the const qualifiers, avoiding the need for ugly casts only to silence some warnings. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2007-10-17pack-objects.c: fix some global variable abuse and memory leaksLibravatar Nicolas Pitre1-14/+15
To keep things well layered, sha1close() now returns the file descriptor when it doesn't close the file. An ugly cast was added to the return of write_idx_file() to avoid a warning. A proper fix will come separately. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2007-10-17pack-objects: no delta possible with only one object in the listLibravatar Nicolas Pitre1-1/+1
... so don't even try in that case, and save another useless line of progress display. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2007-10-17more compact progress displayLibravatar Nicolas Pitre1-11/+5
Each progress can be on a single line instead of two. [sp: Changed "Checking files out" to "Checking out files" at Johannes Sixt's suggestion as it better explains the action that is taking place] Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2007-10-03Merge branch 'jc/autogc'Libravatar Junio C Hamano1-2/+93
* jc/autogc: git-gc --auto: run "repack -A -d -l" as necessary. git-gc --auto: restructure the way "repack" command line is built. git-gc --auto: protect ourselves from accumulated cruft git-gc --auto: add documentation. git-gc --auto: move threshold check to need_to_gc() function. repack -A -d: use --keep-unreachable when repacking pack-objects --keep-unreachable Export matches_pack_name() and fix its return value Invoke "git gc --auto" from commit, merge, am and rebase. Implement git gc --auto
2007-09-17pack-objects --keep-unreachableLibravatar Junio C Hamano1-2/+93
This new option is meant to be used in conjunction with the options "git repack -a -d" usually invokes the underlying pack-objects with. When this option is given, objects unreachable from the refs in packs named with --unpacked= option are added to the resulting pack, in addition to the reachable objects that are not in packs marked with *.keep files. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-14builtin-pack-objects.c: avoid bogus gcc warningsLibravatar Junio C Hamano1-6/+6
These empty statement marcos can solicit bogus "statement with no effect" warnings; squelch them. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-12threaded delta search: proper locking for cache accountingLibravatar Nicolas Pitre1-4/+20
Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-10threaded delta search: add pack.threads config variableLibravatar Nicolas Pitre1-0/+11
Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-10fix threaded delta search lockingLibravatar Nicolas Pitre1-5/+6
Found by Jeff King. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-09threaded delta search: specify number of threads at run timeLibravatar Nicolas Pitre1-5/+21
This adds a --threads=<n> parameter to 'git pack-objects' with documentation. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-09threaded delta search: better chunck split pointLibravatar Nicolas Pitre1-0/+5
Try to keep object with the same name hash together. Suggested by Martin Koegler. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-09threaded delta search: refine work allocationLibravatar Nicolas Pitre1-14/+45
With this, each thread get repeatedly assigned the next available chunk of objects to process until the whole list is done. The idea is to have reasonably small chunks so that all CPUs remain busy with a minimum number of threads for as long as there is data to process. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-06basic threaded delta searchLibravatar Nicolas Pitre1-1/+82
this is still rough, hence it is disabled by default. You need to compile with "make THREADED_DELTA_SEARCH=1 ..." at the moment. Threading is done on different portions of the object list to be deltified. This is currently done by spliting the list into n parts and then a thread is spawned for each of them. A better method would consist of spliting the list into more smaller parts and have the n threads pick the next part available. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-06rearrange delta search progress reportingLibravatar Nicolas Pitre1-9/+14
This is to help threadification of the delta search code, with a bonus consistency check. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-05localize window memory usage accountingLibravatar Nicolas Pitre1-14/+14
This is to help threadification of delta searching. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>