| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Commit c6458e6 (index-pack: kill union delta_base to save
memory, 2015-04-18) refactored the comparison functions used
in sorting and binary searching our delta list. The
resulting code does something like:
int cmp_offsets(off_t a, off_t b)
{
return a - b;
}
This works most of the time, but produces nonsensical
results when the difference between the two offsets is
larger than what can be stored in an "int". This can lead to
unresolved deltas if the packsize is larger than 2G (even on
64-bit systems, an int is still typically 32 bits):
$ git clone git://github.com/mozilla/gecko-dev
Cloning into 'gecko-dev'...
remote: Counting objects: 4800161, done.
remote: Compressing objects: 100% (178/178), done.
remote: Total 4800161 (delta 88), reused 0 (delta 0), pack-reused 4799978
Receiving objects: 100% (4800161/4800161), 2.21 GiB | 3.26 MiB/s, done.
Resolving deltas: 99% (3808820/3811944), completed with 0 local objects.
fatal: pack has 3124 unresolved deltas
fatal: index-pack failed
We can fix it by doing direct comparisons between the
offsets and returning constants; the callers only care about
the sign of the comparison, not the magnitude.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
Once we know the number of objects in the input pack, we allocate an
array of nr_objects of struct delta_entry. On x86-64, this struct is
32 bytes long. The union delta_base, which is part of struct
delta_entry, provides enough space to store either ofs-delta (8 bytes)
or ref-delta (20 bytes).
Because ofs-delta encoding is more efficient space-wise and more
performant at runtime than ref-delta encoding, Git packers try to use
ofs-delta whenever possible, and it is expected that objects encoded
as ref-delta are minority.
In the best clone case where no ref-delta object is present, we waste
(20-8) * nr_objects bytes because of this union. That's about 38MB out
of 100MB for deltas[] with 3.4M objects, or 38%. deltas[] would be
around 62MB without the waste.
This patch attempts to eliminate that. deltas[] array is split into
two: one for ofs-delta and one for ref-delta. Many functions are also
duplicated because of this split. With this patch, ofs_deltas[] array
takes 51MB. ref_deltas[] should remain unallocated in clone case (0
bytes). This array grows as we see ref-delta. We save about half in
this case, or 25% of total bookkeeping.
The saving is more than the calculation above because some padding in
the old delta_entry struct is removed. ofs_delta_entry is 16 bytes,
including the 4 bytes padding. That's 13MB for padding, but packing
the struct could break platforms that do not support unaligned
access. If someone on 32-bit is really low on memory and only deals
with packs smaller than 2G, using 32-bit off_t would eliminate the
padding and save 27MB on top.
A note about ofs_deltas allocation. We could use ref_deltas memory
allocation strategy for ofs_deltas. But that probably just adds more
overhead on top. ofs-deltas are generally the majority (1/2 to 2/3) in
any pack. Incremental realloc may lead to too many memcpy. And if we
preallocate, say 1/2 or 2/3 of nr_objects initially, the growth rate
of ALLOC_GROW() could make this array larger than nr_objects, wasting
more memory.
Brought-up-by: Matthew Sporleder <msporleder@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
For each object in the input pack, we need one struct object_entry. On
x86-64, this struct is 64 bytes long. Although:
- The 8 bytes for delta_depth and base_object_no are only useful when
show_stat is set. And it's never set unless someone is debugging.
- The three fields hdr_size, type and real_type take 4 bytes each
even though they never use more than 4 bits.
By moving delta_depth and base_object_no out of struct object_entry
and make the other 3 fields one byte long instead of 4, we shrink 25%
of this struct.
On a 3.4M object repo (*) that's about 53MB. The saving is less
impressive compared to index-pack memory use for basic bookkeeping (**),
about 16%.
(*) linux-2.6.git already has 4M objects as of v3.19-rc7 so this is
not an unrealistic number of objects that we have to deal with.
(**) 3.4M * (sizeof(object_entry) + sizeof(delta_entry)) = 311MB
Brought-up-by: Matthew Sporleder <msporleder@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
* ak/cat-file-clean-up:
cat-file: use "type" and "size" from outer scope
|
|
* ak/show-branch-usage-string:
show-branch: fix indentation of usage string
|
|
Noticed-by: Jean-Noël Avila <jn.avila@free.fr>
Signed-off-by: Ralf Thielow <ralf.thielow@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
* rc/for-each-ref-tracking:
for-each-ref: always check stat_tracking_info()'s return value
|
|
* ak/fewer-includes:
cat-file: remove unused includes
git.c: remove unnecessary #includes
|
|
* ak/show-branch-usage-string:
show-branch: line-wrap show-branch usage
|
|
* km/log-usage-string-i18n:
log.c: fix translation markings
|
|
"git remote add $name $URL" is now allowed when "url.$URL.insteadOf"
is already defined.
* js/remote-add-with-insteadof:
Add a regression test for 'git remote add <existing> <same-url>'
git remote: allow adding remotes agreeing with url.<...>.insteadOf
|
|
In cat_one_file(), "type" and "size" variables are defined in the
function scope, and then two variables of the same name are defined
in a block in one of the if/else statement, hiding the definitions
in the outer scope.
Because the values of the outer variables before the control enters
this scope, however, do not have to be preserved, we can remove
useless definitions of variables from the inner scope safely without
breaking anything.
Signed-off-by: Alexander Kuleshov <kuleshovmail@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
The code handling %(upstream:track) and %(upstream:trackshort)
assumed that it always had a valid branch that had been sanitized
earlier in populate_value(), and thus did not check the return value
of the call to stat_tracking_info().
While there is indeed some sanitization code that basically
corresponds to stat_tracking_info() returning 0 (no base branch
set), the function can also return -1 when the base branch did exist
but has since then been deleted.
In this case, num_ours and num_theirs had undefined values and a
call to `git for-each-ref --format="%(upstream:track)"` could print
spurious values such as
[behind -111794512]
[ahead 38881640, behind 5103867]
even for repositories with one single commit.
Verify stat_tracking_info()'s return value and do not print anything
if it returns -1. This behavior also matches the documentation ("has
no effect if the ref does not have tracking information associated
with it").
Helped-by: Eric Sunshine <sunshine@sunshineco.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
* mg/add-ignore-errors:
add: ignore only ignored files
|
|
Earlier we made "rev-list --object-edge" more aggressively list the
objects at the edge commits, in order to reduce number of objects
fetched into a shallow repository, but the change affected cases
other than "fetching into a shallow repository" and made it
unusably slow (e.g. fetching into a normal repository should not
have to suffer the overhead from extra processing). Limit it to a
more specific case by introducing --objects-edge-aggressive, a new
option to rev-list.
* bc/fetch-thin-less-aggressive-in-normal-repository:
pack-objects: use --objects-edge-aggressive for shallow repos
rev-list: add an option to mark fewer edges as uninteresting
Documentation: add missing article in rev-list-options.txt
|
|
* rs/simplify-parsing-commit-tree-S:
commit-tree: simplify parsing of option -S using skip_prefix()
|
|
* rs/plug-strbuf-leak-in-merge:
merge: release strbuf after use in suggest_conflicts()
|
|
"git checkout-index --temp=$target $path" did not work correctly
for paths outside the current subdirectory in the project.
* es/checkout-index-temp:
checkout-index: fix --temp relative path mangling
t2004: demonstrate broken relative path printing
t2004: standardize file naming in symlink test
t2004: drop unnecessary write-tree/read-tree
t2004: modernize style
|
|
- "exec_cmd.h" became unnecessary at b931aa5a (Call builtin ls-tree
in git-cat-file -p, 2006-05-26), when it changed an earlier code
that delegated tree display to "ls-tree" via the run_command()
API (hence needing "exec_cmd.h") to call cmd_ls_tree() directly.
We should have removed the include in the same commit, but we
forgot to do so.
- "diff.h" was added at e5fba602 (textconv: support for cat_file,
2010-06-15), together with "userdiff.h", but "userdiff.h" can be
included without including "diff.h"; the header was unnecessary
from the beginning.
- "tag.h" and "tree.h" were necessary since 8e440259 (Use blob_,
commit_, tag_, and tree_type throughout., 2006-04-02) to check
the type of object by comparing typename with tree_type and
tag_type (pointers to extern strings).
21666f1a (convert object type handling from a string to a number,
2007-02-26) made these <type>_type strings unnecessary, and it
could have switched to include "object.h", which is necessary to
use typename(), but it forgot to do so. Because "tag.h" and
"tree.h" include "object.h", it did not need to explicitly
include "object.h" in order to start using typename() itself.
We do not even have to include "object.h" after removing these
two #includes, because "builtin.h" includes "commit.h" which in
turn includes "object.h" these days. This happened at 7b9c0a69
(git-commit-tree: make it usable from other builtins,
2008-07-01).
Signed-off-by: Alexander Kuleshov <kuleshovmail@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
Signed-off-by: Alexander Kuleshov <kuleshovmail@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
The get_merge_bases*() API was easy to misuse by careless
copy&paste coders, leaving object flags tainted in the commits that
needed to be traversed.
* jc/merge-bases:
get_merge_bases(): always clean-up object flags
bisect: clean flags after checking merge bases
|
|
Allow "git clone --reference" to be used more safely.
* jc/clone-borrow:
clone: --dissociate option to mark that reference is only temporary
|
|
The parse_options API expects an array of alternative usage lines
to which it automatically ads the language-appropriate "or" when
displaying. Each of these options is marked for translation with N_
and then later translated when gettext is called on each element
of the array.
Since the N_ macro just expands to its argument, if two N_-marked
strings appear next to each other without being separated by anything
else such as a comma, the preprocessor will join them into one string.
In that case two separate strings get marked for translation, but at
runtime they have been joined into a single string passed to gettext
which then fails to get translated because the combined string was
never marked for translation.
Fix this by properly separating the two N_ marked strings with
a comma and removing the embedded "\n" and " or:" that are
properly supplied by the parse_options API.
Signed-off-by: Kyle J. McKay <mackyle@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
checkout-index --temp only properly prints relative paths which are
descendants of the current directory. Paths in ancestor or sibling
directories (or their children) are often printed in mangled form. For
example:
mkdir a bbb &&
>file &&
>bbb/file &&
git update-index --add file bbb/file &&
cd a &&
git checkout-index --temp ../file ../bbb/file
prints:
.merge_file_ooblek le
.merge_file_igloo0 b/file
rather than the correct:
.merge_file_ooblek ../file
.merge_file_igloo0 ../bbb/file
Internally, given the above example, checkout-index prefixes each input
argument with the name of the current directory ("a/", in this case),
and then assumes that it can simply skip forward by strlen("a/") bytes
to recover the original name. This works for files in the current
directory or its descendants, but fails for files in ancestors or
siblings (or their children) due to path normalization.
For instance, given "../file", "a/" is prepended, giving "a/../file".
Path normalization folds out "a/../", resulting in "file". Attempting
to recover the original name by skipping strlen("a/") bytes gives the
incorrect "le" rather than the desired "../file".
Fix this by taking advantage of write_name_quoted_relative() to recover
the original name properly, rather than assuming that it can be
recovered by skipping strlen(prefix) bytes.
As a bonus, this also fixes a bug in which checkout-index --temp
accessed and printed memory beyond the end-of-string. For instance,
within a subdirectory named "subdirectory", and given argument
"../file", prefixing would give "subdirectory/../file", which would
become "file" after normalization. checkout-index would then attempt to
recover the original name by skipping strlen("subdirectory/") bytes of
"file", which placed it well beyond end-of-string. Despite this error,
it often appeared to give the correct result, but only due to an
accident of implementation which left an apparently correct copy of the
path in memory following the normalized value. In particular, handed
"subdirectory/../file", in-place processing by normalize_path_copy_len()
resulted in "file\0rectory/../file". When checkout-index skipped
strlen("subdirectory/") bytes, it ended up back at "../file" and thus
appeared to give the correct answer, despite being past end-of-string.
Reported-by: Russ Cox <rsc@golang.org>
Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
When fetching into or pushing from a shallow repository, we want to
aggressively mark edges as uninteresting, since this decreases the pack
size. However, aggressively marking edges can negatively affect
performance on large non-shallow repositories with lots of refs.
Teach pack-objects a --shallow option to indicate that we're pushing
from or fetching into a shallow repository. Use
--objects-edge-aggressive only for shallow repositories and otherwise
use --objects-edge, which performs better in the general case. Update
the callers to pass the --shallow option when they are dealing with a
shallow repository.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
In commit fbd4a70 (list-objects: mark more commits as edges in
mark_edges_uninteresting - 2013-08-16), we marked an increasing number
of edges uninteresting. This change, and the subsequent change to make
this conditional on --objects-edge, are used by --thin to make much
smaller packs for shallow clones.
Unfortunately, they cause a significant performance regression when
pushing non-shallow clones with lots of refs (23.322 seconds vs.
4.785 seconds with 22400 refs). Add an option to git rev-list,
--objects-edge-aggressive, that preserves this more aggressive behavior,
while leaving --objects-edge to provide more performant behavior.
Preserve the current behavior for the moment by using the aggressive
option.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
Signed-off-by: Rene Scharfe <l.s.r@web.de>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
Signed-off-by: Rene Scharfe <l.s.r@web.de>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
"git update-ref --stdin"'s verify command did not work well when
<oldvalue>, which is documented as optional, was missing.
* mh/update-ref-verify:
update-ref: fix "verify" command with missing <oldvalue>
t1400: add some more tests of "update-ref --stdin"'s verify command
|
|
When adding a remote, we make sure that the remote does not exist
already. However, this test was not quite correct: when the
url.<...>.insteadOf config variable was set to the remote name to be
added, the code would assume that the remote exists already.
Let's allow adding remotes when there is a url.<...>.insteadOf setting
when both the name and the URL agree with the remote to be added.
It might seem like a mistake to compare against remote->url[0] without
verifying that remote->url_nr >=1, but at this point a missing URL has
been filled by the name already, therefore url_nr cannot be zero.
Noticed by Anastas Dancha.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
* rs/use-strbuf-complete-line:
use strbuf_complete_line() for adding a newline if needed
|
|
Recent update to "git commit" broke amending an existing commit
with bogus author/committer lines without a valid e-mail address.
* jk/commit-date-approxidate:
commit: always populate GIT_AUTHOR_* variables
commit: loosen ident checks when generating template
|
|
New tag object format validation added in 2.2 showed garbage
after a tagname it reported in its error message.
* js/fsck-tag-validation:
index-pack: terminate object buffers with NUL
fsck: properly bound "invalid tag name" error message
|
|
"git branch -d" (delete) and "git branch -m" (move) learned to
honor "-f" (force) flag; unlike many other subcommands, the way to
force these have been with separate "-D/-M" options, which was
inconsistent.
* mg/branch-d-m-f:
branch: allow -f with -m and -d
t3200-branch: test -M
|
|
"git ls-tree" does not support path selection based on negative
pathspecs, but did not error out when negative pathspecs are given.
* nd/ls-tree-pathspec:
t3102: style modernization
t3102: document that ls-tree does not yet support negated pathspec
ls-tree: disable negative pathspec because it's not supported
ls-tree: remove path filtering logic in show_tree
tree.c: update read_tree_recursive callback to pass strbuf as base
|
|
* rt/for-each-ref-spell-tcl-as-Tcl:
for-each-ref: correct spelling of Tcl in option description
|
|
"git push" into a repository with a working tree normally refuses
to modify the branch that is checked out. The command learned to
optionally do an equivalent of "git reset --hard" only when there
is no change to the working tree and the index instead, which would
be useful to "deploy" by pushing into a repository.
* js/push-to-deploy:
t5516: more tests for receive.denyCurrentBranch=updateInstead
receive-pack: add another option for receive.denyCurrentBranch
|
|
The function sometimes returned a non-freeable memory and some
other times returned a piece of memory that must be freed.
* jc/exec-cmd-system-path-leak-fix:
system_path(): always return free'able memory to the caller
|
|
"git am" learned "--message-id" option to copy the message ID of
the incoming e-mail to the log message of resulting commit.
* pb/am-message-id-footer:
git-am: add --message-id/--no-message-id
git-mailinfo: add --message-id
|
|
"git remote update --prune" to drop many refs has been optimized.
* mh/simplify-repack-without-refs:
sort_string_list(): rename to string_list_sort()
prune_remote(): iterate using for_each_string_list_item()
prune_remote(): rename local variable
repack_without_refs(): make the refnames argument a string_list
prune_remote(): sort delete_refs_list references en masse
prune_remote(): initialize both delete_refs lists in a single loop
prune_remote(): exit early if there are no stale references
|
|
Some filesystems assign filemodes in a strange way, fooling then
automatic "filemode trustability" check done during a new
repository creation.
* tb/config-core-filemode-check-on-broken-fs:
init-db: improve the filemode trustability check
|
|
"git add --ignore-errors ..." did not ignore an error to
give a file that did not exist.
* mg/add-ignore-errors:
add: ignore only ignored files
|
|
"git interpret-trailers" learned to properly handle the
"Conflicts:" block at the end.
* cc/interpret-trailers-more:
trailer: add test with an old style conflict block
trailer: reuse ignore_non_trailer() to ignore conflict lines
commit: make ignore_non_trailer() non static
merge & sequencer: turn "Conflicts:" hint into a comment
builtin/commit.c: extract ignore_non_trailer() helper function
merge & sequencer: unify codepaths that write "Conflicts:" hint
builtin/merge.c: drop a parameter that is never used
|
|
Git 2.0 was supposed to make the "simple" mode for the default of
"git push", but it didn't.
* jk/push-simple:
push: truly use "simple" as default, not "upstream"
|
|
"git init" (hence "git clone") initialized the per-repository
configuration file .git/config with x-bit by mistake.
* mh/config-flip-xbit-back-after-checking:
create_default_files(): don't set u+x bit on $GIT_DIR/config
|
|
* rs/receive-pack-use-labs:
use labs() for variables of type long instead of abs()
|
|
"git config --get-color" did not parse its command line arguments
carefully.
* jk/colors-fix:
t4026: test "normal" color
config: fix parsing of "git config --get-color some.key -1"
docs: describe ANSI 256-color mode
|
|
"git checkout $treeish $path", when $path in the index and the
working tree already matched what is in $treeish at the $path,
still overwrote the $path unnecessarily.
* jk/checkout-from-tree:
checkout $tree: do not throw away unchanged index entries
|
|
Signed-off-by: Alexander Kuleshov <kuleshovmail@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
Jeff King
Nguyễn Thái Ngọc Duy
Junio C Hamano
Ralf Thielow
Alexander Kuleshov
Raphael Kubo da Costa
Kyle J. McKay
Eric Sunshine
brian m. carlson
René Scharfe
Johannes Schindelin