summaryrefslogtreecommitdiff
path: root/fetch.c
AgeCommit message (Collapse)AuthorFilesLines
2006-06-17Shrink "struct object" a bitLibravatar Linus Torvalds1-8/+6
This shrinks "struct object" by a small amount, by getting rid of the "struct type *" pointer and replacing it with a 3-bit bitfield instead. In addition, we merge the bitfields and the "flags" field, which incidentally should also remove a useless 4-byte padding from the object when in 64-bit mode. Now, our "struct object" is still too damn large, but it's now less obviously bloated, and of the remaining fields, only the "util" (which is not used by most things) is clearly something that should be eventually discarded. This shrinks the "git-rev-list --all" memory use by about 2.5% on the kernel archive (and, perhaps more importantly, on the larger mozilla archive). That may not sound like much, but I suspect it's more on a 64-bit platform. There are other remaining inefficiencies (the parent lists, for example, probably have horrible malloc overhead), but this was pretty obvious. Most of the patch is just changing the comparison of the "type" pointer from one of the constant string pointers to the appropriate new TYPE_xxx small integer constant. Signed-off-by: Linus Torvalds <torvalds@osdl.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-06-06ref-log: style fixes.Libravatar Junio C Hamano1-2/+4
A few style fixes to get the code in line with the rest. - asterisk to make a type a pointer to something goes in front of the variable, not at the end of the base type. E.g. a pointer to an integer is "int *ip", not "int* ip". - open parenthesis for function parameter list, unlike syntactic constructs, comes immediately after the function name. E.g. "if (foo) bar();" not "if(foo) bar ();". - "else" does not come on the same line as the closing brace of corresponding "if". The style is mostly a matter of personal taste, and people may disagree, but consistency is important. Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-06-03Merge branch 'lt/tree-2'Libravatar Junio C Hamano1-9/+22
* lt/tree-2: fetch.c: do not call process_tree() from process_tree(). tree_entry(): new tree-walking helper function adjust to the rebased series by Linus. Remove "tree->entries" tree-entry list from tree parser Switch "read_tree_recursive()" over to tree-walk functionality Make "tree_entry" have a SHA1 instead of a union of object pointers Add raw tree buffer info to "struct tree" Remove last vestiges of generic tree_entry_list Convert fetch.c: process_tree() to raw tree walker Convert "mark_tree_uninteresting()" to raw tree walker Remove unused "zeropad" entry from tree_list_entry fsck-objects: avoid unnecessary tree_entry_list usage Remove "tree->entries" tree-entry list from tree parser builtin-read-tree.c: avoid tree_entry_list in prime_cache_tree_rec() Switch "read_tree_recursive()" over to tree-walk functionality Make "tree_entry" have a SHA1 instead of a union of object pointers Make "struct tree" contain the pointer to the tree buffer
2006-06-02fetch.c: do not call process_tree() from process_tree().Libravatar Junio C Hamano1-3/+10
This function reads a freshly fetched tree object, and schedules the objects pointed by it for further fetching, so doing lookup_tree() and process_tree() recursively from there does not make much sense. We need to use process() on it to make sure we fetch it first, and leave the recursive processing to later stages. Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-05-31fetch.c: do not pass uninitialized lock to unlock_ref().Libravatar Junio C Hamano1-4/+7
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-05-30tree_entry(): new tree-walking helper functionLibravatar Linus Torvalds1-11/+5
This adds a "tree_entry()" function that combines the common operation of doing a "tree_entry_extract()" + "update_tree_entry()". It also has a simplified calling convention, designed for simple loops that traverse over a whole tree: the arguments are pointers to the tree descriptor and a name_entry structure to fill in, and it returns a boolean "true" if there was an entry left to be gotten in the tree. This allows tree traversal with struct tree_desc desc; struct name_entry entry; desc.buf = tree->buffer; desc.size = tree->size; while (tree_entry(&desc, &entry) { ... use "entry.{path, sha1, mode, pathlen}" ... } which is not only shorter than writing it out in full, it's hopefully less error prone too. [ It's actually a tad faster too - we don't need to recalculate the entry pathlength in both extract and update, but need to do it only once. Also, some callers can avoid doing a "strlen()" on the result, since it's returned as part of the name_entry structure. However, by now we're talking just 1% speedup on "git-rev-list --objects --all", and we're definitely at the point where tree walking is no longer the issue any more. ] NOTE! Not everybody wants to use this new helper function, since some of the tree walkers very much on purpose do the descriptor update separately from the entry extraction. So the "extract + update" sequence still remains as the core sequence, this is just a simplified interface. We should probably add a silly two-line inline helper function for initializing the descriptor from the "struct tree" too, just to cut down on the noise from that common "desc" initializer. Signed-off-by: Linus Torvalds <torvalds@osdl.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-05-29Convert fetch.c: process_tree() to raw tree walkerLibravatar Linus Torvalds1-9/+15
This leaves only the horrid code in builtin-read-tree.c using the old interface. Some day I will gather the strength to tackle that one too. Signed-off-by: Linus Torvalds <torvalds@osdl.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-05-29Remove "tree->entries" tree-entry list from tree parserLibravatar Linus Torvalds1-5/+11
Instead, just use the tree buffer directly, and use the tree-walk infrastructure to walk the buffers instead of the tree-entry list. The tree-entry list is inefficient, and generates tons of small allocations for no good reason. The tree-walk infrastructure is generally no harder to use than following a linked list, and allows us to do most tree parsing in-place. Some programs still use the old tree-entry lists, and are a bit painful to convert without major surgery. For them we have a helper function that creates a temporary tree-entry list on demand. Signed-off-by: Linus Torvalds <torvalds@osdl.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-05-24Merge branch 'master' into sp/reflogLibravatar Junio C Hamano1-5/+2
* master: (90 commits) fetch.c: remove an unused variable and dead code. Clean up sha1 file writing Builtin git-cat-file builtin format-patch: squelch content-type for 7-bit ASCII CMIT_FMT_EMAIL: Q-encode Subject: and display-name part of From: fields. add more informative error messages to git-mktag remove the artificial restriction tagsize < 8kb git-rebase: use canonical A..B syntax to format-patch git-format-patch: now built-in. fmt-patch: Support --attach fmt-patch: understand old <his> notation Teach fmt-patch about --keep-subject Teach fmt-patch about --numbered fmt-patch: implement -o <dir> fmt-patch: output file names to stdout Teach fmt-patch to write individual files. built-in tar-tree and remote tar-tree Builtin git-diff-files, git-diff-index, git-diff-stages, and git-diff-tree. Builtin git-show-branch. Builtin git-apply. ...
2006-05-24fetch.c: remove an unused variable and dead code.Libravatar Junio C Hamano1-17/+3
Funnily enough, this variable was never assigned ever since it was introduced, and has been protecting some code that has never been executed. Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-05-19Log ref updates made by fetch.Libravatar Shawn Pearce1-2/+15
If a ref is changed by http-fetch, local-fetch or ssh-fetch record the change and the remote URL/name in the log for the ref. This requires loading the config file to check logAllRefUpdates. Also fixed a bug in the ref lock generation; the log file name was not being produced right due to a bad prefix length. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-05-17Improve abstraction of ref lock/write.Libravatar Shawn Pearce1-15/+17
Created 'struct ref_lock' to contain the data necessary to perform a ref update. This change improves writing a ref as the file names are generated only once (rather than twice) and supports following symrefs (up to the maximum depth). Further the ref_lock structure provides room to extend the update API with ref logging. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-10-10[PATCH] Don't fetch objects that exist in the local repositoryLibravatar Nick Hengeveld1-1/+1
Be sure not to fetch objects that already exist in the local repository. The main process loop no longer performs this check, http-fetch now checks prior to starting a new request queue entry and when fetch_object() is called, and local-fetch now checks when fetch_object() is called. As discussed in this thread: http://marc.theaimsgroup.com/?t=112854890500001 Signed-off-by: Nick Hengeveld <nickh@reactrix.com>
2005-09-27[PATCH] Implement --recover for git-*-fetchLibravatar Daniel Barkalow1-1/+4
With the --recover option, we verify that we have absolutely everything reachable from the target, not assuming that things reachable from refs will be complete. Signed-off-by: Daniel Barkalow <barkalow@iabervon.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-23[PATCH] fetch.c: Plug memory leak in process_tree()Libravatar Sergey Vlasov1-0/+1
When freeing a tree entry, must free its name too. Signed-off-by: Sergey Vlasov <vsu@altlinux.ru> Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-23[PATCH] fetch.c: Do not build object ref listsLibravatar Sergey Vlasov1-0/+1
The fetch code does not need object ref lists; by disabling them we can save some time and memory. Signed-off-by: Sergey Vlasov <vsu@altlinux.ru> Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-22[PATCH] fetch.c: Remove call to parse_object() from process()Libravatar Sergey Vlasov1-1/+0
The call to parse_object() in process() is not actually needed - if the object type is unknown, parse_object() will be called by loop(); if the type is known, the object will be parsed by the appropriate process_*() function. After this change blobs which exist locally are no longer parsed, which gives about 2x CPU usage improvement; the downside is that there will be no warnings for existing corrupted blobs, but detecting such corruption is the job of git-fsck-objects, not the fetch programs. Newly fetched objects are still checked for corruption in http-fetch.c and ssh-fetch.c (local-fetch.c does not seem to do it, but the removed parse_object() call would not be reached for new objects anyway). Signed-off-by: Sergey Vlasov <vsu@altlinux.ru> Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-22[PATCH] fetch.c: Clean up object flag definitionsLibravatar Sergey Vlasov1-3/+3
Remove holes left after deleting flags, and use shifts to emphasize that flags are single bits. Signed-off-by: Sergey Vlasov <vsu@altlinux.ru> Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-22[PATCH] fetch.c: Remove redundant test of TO_SCAN in process()Libravatar Sergey Vlasov1-2/+0
If the SEEN flag was not set, the TO_SCAN flag cannot be set, therefore testing it is pointless. Signed-off-by: Sergey Vlasov <vsu@altlinux.ru> Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-22[PATCH] fetch.c: Remove some duplicated code in process()Libravatar Sergey Vlasov1-8/+5
It does not matter if we call prefetch() or set the TO_SCAN flag before or after adding the object to process_queue. However, doing it before object_list_insert() allows us to kill 3 lines of duplicated code. Signed-off-by: Sergey Vlasov <vsu@altlinux.ru> Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-22[PATCH] fetch.c: Remove redundant TO_FETCH flagLibravatar Sergey Vlasov1-3/+1
The TO_FETCH flag also became redundant after adding the SEEN flag - it was set and checked in process() to prevent adding the same object to process_queue multiple times, but now SEEN guards against this. Signed-off-by: Sergey Vlasov <vsu@altlinux.ru> Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-22[PATCH] fetch.c: Remove redundant SCANNED flagLibravatar Sergey Vlasov1-6/+1
After adding the SEEN flag, the SCANNED flag became obviously redundant - each object can get into process_queue through process() only once, and therefore multiple calls to process_object() for the same object are not possible. Signed-off-by: Sergey Vlasov <vsu@altlinux.ru> Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-22[PATCH] fetch.c: Make process() look at each object only onceLibravatar Sergey Vlasov1-0/+5
The process() function is very often called multiple times for the same object (because lots of trees refer to the same blobs), but did not have a fast check for this, therefore a lot of useless calls to has_sha1_file() and parse_object() were made before discovering that nothing needs to be done. This patch adds the SEEN flag which is used in process() to make it look at each object only once. When testing git-local-fetch on the repository of GIT, this gives a 14x improvement in CPU usage (mainly because the redundant calls to parse_object() are now avoided - parse_object() always unpacks and parses the object data, even if it was already parsed before). Signed-off-by: Sergey Vlasov <vsu@altlinux.ru> Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-22[PATCH] fetch.c: Remove useless lookup_object_type() call in process()Libravatar Sergey Vlasov1-13/+10
In all places where process() is called except the one in pull() (which is executed only once) the pointer to the object is already available, so pass it as the argument to process() instead of sha1 and avoid an unneeded call to lookup_object_type(). Signed-off-by: Sergey Vlasov <vsu@altlinux.ru> Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-18fetch() assumes we do not have the object.Libravatar Junio C Hamano1-1/+1
Bugfix for the previous one. Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-18Improve the safety check used in fetch.cLibravatar Junio C Hamano1-32/+44
The recent safety check to trust only the commits we have made things impossibly slow and turn out to waste a lot of memory. This commit fixes it with the following improvements: - mark already scanned objects and avoid rescanning the same object again; - free the tree entries when we have scanned the tree entries; this is the same as b0d8923ec01fd91b75ab079034f89ced91500157 which reduced memory usage by rev-list; - plug memory leak from the object_list dequeuing code; - use the process_queue not just for fetching but for scanning, to make things tail recursive to avoid deep recursion; the deep recursion was especially prominent when we cloned a big pack. - avoid has_sha1_file() call when we already know we do not have that object. Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-16[PATCH] fetch.c: cleanupsLibravatar Junio C Hamano1-11/+8
Clean-ups suggested by Sergey Vlasov and acked by Daniel Barkalow. Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-15Avoid wasting memory while keeping track of what we have during fetch.Libravatar Junio C Hamano1-0/+1
Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-15[PATCH] Fix fetch completeness assumptionsLibravatar Daniel Barkalow1-2/+26
Don't assume that any commit we have is complete; assume that any ref we have is complete. Signed-off-by: Daniel Barkalow <barkalow@iabervon.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-09-07Big tool rename.Libravatar Junio C Hamano1-0/+210
As promised, this is the "big tool rename" patch. The primary differences since 0.99.6 are: (1) git-*-script are no more. The commands installed do not have any such suffix so users do not have to remember if something is implemented as a shell script or not. (2) Many command names with 'cache' in them are renamed with 'index' if that is what they mean. There are backward compatibility symblic links so that you and Porcelains can keep using the old names, but the backward compatibility support is expected to be removed in the near future. Signed-off-by: Junio C Hamano <junkio@cox.net>