diff options
Diffstat (limited to 'Documentation')
55 files changed, 1946 insertions, 215 deletions
diff --git a/Documentation/CodingGuidelines b/Documentation/CodingGuidelines index 227f46ae40..45465bc0c9 100644 --- a/Documentation/CodingGuidelines +++ b/Documentation/CodingGuidelines @@ -489,16 +489,11 @@ For Python scripts: - We follow PEP-8 (http://www.python.org/dev/peps/pep-0008/). - - As a minimum, we aim to be compatible with Python 2.6 and 2.7. + - As a minimum, we aim to be compatible with Python 2.7. - Where required libraries do not restrict us to Python 2, we try to also be compatible with Python 3.1 and later. - - When you must differentiate between Unicode literals and byte string - literals, it is OK to use the 'b' prefix. Even though the Python - documentation for version 2.6 does not mention this prefix, it has - been supported since version 2.6.0. - Error Messages - Do not end error messages with a full stop. diff --git a/Documentation/Makefile b/Documentation/Makefile index 15d9d04f31..ecd0b340b1 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -93,6 +93,7 @@ TECH_DOCS += technical/protocol-capabilities TECH_DOCS += technical/protocol-common TECH_DOCS += technical/protocol-v2 TECH_DOCS += technical/racy-git +TECH_DOCS += technical/reftable TECH_DOCS += technical/send-pack-pipeline TECH_DOCS += technical/shallow TECH_DOCS += technical/signature-format diff --git a/Documentation/MyFirstContribution.txt b/Documentation/MyFirstContribution.txt index 427274df4d..d85c9b5143 100644 --- a/Documentation/MyFirstContribution.txt +++ b/Documentation/MyFirstContribution.txt @@ -1179,8 +1179,8 @@ look at the section below this one for some context.) [[after-approval]] === After Review Approval -The Git project has four integration branches: `pu`, `next`, `master`, and -`maint`. Your change will be placed into `pu` fairly early on by the maintainer +The Git project has four integration branches: `seen`, `next`, `master`, and +`maint`. Your change will be placed into `seen` fairly early on by the maintainer while it is still in the review process; from there, when it is ready for wider testing, it will be merged into `next`. Plenty of early testers use `next` and may report issues. Eventually, changes in `next` will make it to `master`, diff --git a/Documentation/RelNotes/2.28.0.txt b/Documentation/RelNotes/2.28.0.txt new file mode 100644 index 0000000000..6baf781380 --- /dev/null +++ b/Documentation/RelNotes/2.28.0.txt @@ -0,0 +1,236 @@ +Git 2.28 Release Notes +====================== + +Updates since v2.27 +------------------- + +Backward compatibility notes + + * "fetch.writeCommitGraph" is deemed to be still a bit too risky and + is no longer part of the "feature.experimental" set. + + +UI, Workflows & Features + + * The commands in the "diff" family learned to honor "diff.relative" + configuration variable. + + * The check in "git fsck" to ensure that the tree objects are sorted + still had corner cases it missed unsorted entries. + + * The interface to redact sensitive information in the trace output + has been simplified. + + * The command line completion (in contrib/) learned to complete + options that the "git switch" command takes. + + * "git diff" used to take arguments in random and nonsense range + notation, e.g. "git diff A..B C", "git diff A..B C...D", etc., + which has been cleaned up. + + * "git diff-files" has been taught to say paths that are marked as + intent-to-add are new files, not modified from an empty blob. + + * "git status" learned to report the status of sparse checkout. + + * "git difftool" has trouble dealing with paths added to the index + with the intent-to-add bit. + + * "git fast-export --anonymize" learned to take customized mapping to + allow its users to tweak its output more usable for debugging. + + * The command line completion support (in contrib/) used to be + prepared to work with "set -u" but recent changes got a bit more + sloppy. This has been corrected. + + * "git gui" now allows opening work trees from the start-up dialog. + + +Performance, Internal Implementation, Development Support etc. + + * Code optimization for a common case. + (merge 8777616e4d an/merge-single-strategy-optim later to maint). + + * We've adopted a convention that any on-stack structure can be + initialized to have zero values in all fields with "= { 0 }", + even when the first field happens to be a pointer, but sparse + complained that a null pointer should be spelled NULL for a long + time. Start using -Wno-universal-initializer option to squelch + it (the latest sparse has it on by default). + + * "git log -L..." now takes advantage of the "which paths are touched + by this commit?" info stored in the commit-graph system. + + * As FreeBSD is not the only platform whose regexp library reports + a REG_ILLSEQ error when fed invalid UTF-8, add logic to detect that + automatically and skip the affected tests. + + * "git bugreport" learns to report what shell is in use. + + * Support for GIT_CURL_VERBOSE has been rewritten in terms of + GIT_TRACE_CURL. + + * Preliminary clean-ups around refs API, plus file format + specification documentation for the reftable backend. + + * Workaround breakage in MSVC build, where "curl-config --cflags" + gives settings appropriate for GCC build. + + * Code clean-up of "git clean" resulted in a fix of recent + performance regression. + + * Code clean-up in the codepath that serves "git fetch" continues. + + * "git merge-base --is-ancestor" is taught to take advantage of the + commit graph. + + * Rewrite of parts of the scripted "git submodule" Porcelain command + continues; this time it is "git submodule set-branch" subcommand's + turn. + + * The "fetch/clone" protocol has been updated to allow the server to + instruct the clients to grab pre-packaged packfile(s) in addition + to the packed object data coming over the wire. + + * A misdesigned strbuf_write_fd() function has been retired. + + * SHA-256 migration work continues, including CVS/SVN interface. + + * A few fields in "struct commit" that do not have to always be + present have been moved to commit slabs. + + * API cleanup for get_worktrees() + + * By renumbering object flag bits, "struct object" managed to lose + bloated inter-field padding. + + * The name of the primary branch in existing repositories, and the + default name used for the first branch in newly created + repositories, is made configurable, so that we can eventually wean + ourselves off of the hardcoded 'master'. + + * The effort to avoid using test_must_fail on non-git command continues. + + * In 2.28-rc0, we corrected a bug that some repository extensions are + honored by mistake even in a version 0 repositories (these + configuration variables in extensions.* namespace were supposed to + have special meaning in repositories whose version numbers are 1 or + higher), but this was a bit too big a change. The behaviour in + recent versions of Git where certain extensions.* were honored by + mistake even in version 0 repositories has been restored. + + +Fixes since v2.27 +----------------- + + * The "--prepare-p4-only" option of "git p4" is supposed to stop + after replaying one changeset, but kept going (by mistake?) + + * The error message from "git checkout -b foo -t bar baz" was + confusing. + + * Some repositories in the wild have commits that record nonsense + committer timezone (e.g. rails.git); "git fast-import" learned an + option to pass these nonsense timestamps intact to allow recreating + existing repositories as-is. + (merge d42a2fb72f en/fast-import-looser-date later to maint). + + * The command line completion script (in contrib/) tried to complete + "git stash -p" as if it were "git stash push -p", but it was too + aggressive and also affected "git stash show -p", which has been + corrected. + (merge fffd0cf520 vs/complete-stash-show-p-fix later to maint). + + * On-the-wire protocol v2 easily falls into a deadlock between the + remote-curl helper and the fetch-pack process when the server side + prematurely throws an error and disconnects. The communication has + been updated to make it more robust. + + * "git checkout -p" did not handle a newly added path at all. + (merge 2c8bd8471a js/checkout-p-new-file later to maint). + + * The code to parse "git bisect start" command line was lax in + validating the arguments. + (merge 4d9005ff5d cb/bisect-helper-parser-fix later to maint). + + * Reduce memory usage during "diff --quiet" in a worktree with too + many stat-unmatched paths. + (merge d2d7fbe129 jk/diff-memuse-optim-with-stat-unmatch later to maint). + + * The reflog entries for "git clone" and "git fetch" did not + anonymize the URL they operated on. + (merge 46da295a77 js/reflog-anonymize-for-clone-and-fetch later to maint). + + * The behaviour of "sparse-checkout" in the state "git clone + --no-checkout" left was changed accidentally in 2.27, which has + been corrected. + + * Use of negative pathspec, while collecting paths including + untracked ones in the working tree, was broken. + + * The same worktree directory must be registered only once, but + "git worktree move" allowed this invariant to be violated, which + has been corrected. + (merge 810382ed37 es/worktree-duplicate-paths later to maint). + + * The effect of sparse checkout settings on submodules is documented. + (merge e7d7c73249 en/sparse-with-submodule-doc later to maint). + + * Code clean-up around "git branch" with a minor bugfix. + (merge dc44639904 dl/branch-cleanup later to maint). + + * A branch name used in a test has been clarified to match what is + going on. + (merge 08dc26061f pb/t4014-unslave later to maint). + + * An in-code comment in "git diff" has been updated. + (merge c592fd4c83 dl/diff-usage-comment-update later to maint). + + * The documentation and some tests have been adjusted for the recent + renaming of "pu" branch to "seen". + (merge 6dca5dbf93 js/pu-to-seen later to maint). + + * The code to push changes over "dumb" HTTP had a bad interaction + with the commit reachability code due to incorrect allocation of + object flag bits, which has been corrected. + (merge 64472d15e9 bc/http-push-flagsfix later to maint). + + * "git send-email --in-reply-to=<msg>" did not use the In-Reply-To: + header with the value given from the command line, and let it be + overridden by the value on In-Reply-To: header in the messages + being sent out (if exists). + (merge f9f60d7066 ra/send-email-in-reply-to-from-command-line-wins later to maint). + + * "git log -Lx,y:path --before=date" lost track of where the range + should be because it didn't take the changes made by the youngest + commits that are omitted from the output into account. + + * When "fetch.writeCommitGraph" configuration is set in a shallow + repository and a fetch moves the shallow boundary, we wrote out + broken commit-graph files that do not match the reality, which has + been corrected. + + * "git checkout" failed to catch an error from fstat() after updating + a path in the working tree. + (merge 35e6e212fd mt/entry-fstat-fallback-fix later to maint). + + * When an aliased command, whose output is piped to a pager by git, + gets killed by a signal, the pager got into a funny state, which + has been corrected (again). + (merge c0d73a59c9 ta/wait-on-aliased-commands-upon-signal later to maint). + + * The code to produce progress output from "git commit-graph --write" + had a few breakages, which have been fixed. + + * Other code cleanup, docfix, build fix, etc. + (merge 2c31a7aa44 jx/pkt-line-doc-count-fix later to maint). + (merge d63ae31962 cb/t5608-cleanup later to maint). + (merge 788db145c7 dl/t-readme-spell-git-correctly later to maint). + (merge 45a87a83bb dl/python-2.7-is-the-floor-version later to maint). + (merge b75a219904 es/advertise-contribution-doc later to maint). + (merge 0c9a4f638a rs/pull-leakfix later to maint). + (merge d546fe2874 rs/commit-reach-leakfix later to maint). + (merge 087bf5409c mk/pb-pretty-email-without-domain-part-fix later to maint). + (merge 5f4ee57ad9 es/worktree-code-cleanup later to maint). + (merge 0172f7834a cc/cat-file-usage-update later to maint). + (merge 81de0c01cf ma/rebase-doc-typofix later to maint). diff --git a/Documentation/RelNotes/2.29.0.txt b/Documentation/RelNotes/2.29.0.txt new file mode 100644 index 0000000000..72e6682dd6 --- /dev/null +++ b/Documentation/RelNotes/2.29.0.txt @@ -0,0 +1,67 @@ +Git 2.29 Release Notes +====================== + +Updates since v2.28 +------------------- + +UI, Workflows & Features + + * "git help log" has been enhanced by sharing more material from the + documentation for the underlying "git rev-list" command. + + * "git for-each-ref --format=<>" learned %(contents:size). + + +Performance, Internal Implementation, Development Support etc. + + * The changed-path Bloom filter is improved using ideas from an + independent implementation. + + * Updates to the changed-paths bloom filter. + + * The test framework has been updated so that most tests will run + with predictable (artificial) timestamps. + + * Preliminary clean-up of the refs API in preparation for adding a + new refs backend "reftable". + + * Dev support to limit the use of test_must_fail to only git commands. + + +Fixes since v2.28 +----------------- + + * "git clone --separate-git-dir=$elsewhere" used to stomp on the + contents of the existing directory $elsewhere, which has been + taught to fail when $elsewhere is not an empty directory. + (merge dfaa209a79 bw/fail-cloning-into-non-empty later to maint). + + + * With the base fix to 2.27 regresion, any new extensions in a v0 + repository would still be silently honored, which is not quite + right. Instead, complain and die loudly. + (merge ec91ffca04 jk/reject-newer-extensions-in-v0 later to maint). + + * Fetching from a lazily cloned repository resulted at the server + side in attempts to lazy fetch objects that the client side has, + many of which will not be available from the third-party anyway. + (merge 77aa0941ce jt/avoid-lazy-fetching-upon-have-check later to maint). + + * Fix to an ancient bug caused by an over-eager attempt for + optimization. + (merge a98f7fb366 rs/add-index-entry-optim-fix later to maint). + + * Pushing a ref whose name contains non-ASCII character with the + "--force-with-lease" option did not work over smart HTTP protocol, + which has been corrected. + (merge cd85b447bf bc/push-cas-cquoted-refname later to maint). + + * "git mv src dst", when src is an unmerged path, errored out + correctly but with an incorrect error message to claim that src is + not tracked, which has been clarified. + (merge 9b906af657 ct/mv-unmerged-path-error later to maint). + + * Fix to a regression introduced during 2.27 cycle. + (merge cada7308ad en/fill-directory-exponential later to maint). + + * Other code cleanup, docfix, build fix, etc. diff --git a/Documentation/SubmittingPatches b/Documentation/SubmittingPatches index 4515cab519..291b61e262 100644 --- a/Documentation/SubmittingPatches +++ b/Documentation/SubmittingPatches @@ -3,8 +3,9 @@ Submitting Patches == Guidelines -Here are some guidelines for people who want to contribute their code -to this software. +Here are some guidelines for people who want to contribute their code to this +software. There is also a link:MyFirstContribution.html[step-by-step tutorial] +available which covers many of these same guidelines. [[base-branch]] === Decide what to base your work on. @@ -18,7 +19,7 @@ change is relevant to. base your work on the tip of the topic. * A new feature should be based on `master` in general. If the new - feature depends on a topic that is in `pu`, but not in `master`, + feature depends on a topic that is in `seen`, but not in `master`, base your work on the tip of that topic. * Corrections and enhancements to a topic not yet in `master` should @@ -27,7 +28,7 @@ change is relevant to. into the series. * In the exceptional case that a new feature depends on several topics - not in `master`, start working on `next` or `pu` privately and send + not in `master`, start working on `next` or `seen` privately and send out patches for discussion. Before the final merge, you may have to wait until some of the dependent topics graduate to `master`, and rebase your work. @@ -37,7 +38,7 @@ change is relevant to. these parts should be based on their trees. To find the tip of a topic branch, run `git log --first-parent -master..pu` and look for the merge commit. The second parent of this +master..seen` and look for the merge commit. The second parent of this commit is the tip of the topic branch. [[separate-commits]] @@ -423,7 +424,7 @@ help you find out who they are. and cooked further and eventually graduates to `master`. In any time between the (2)-(3) cycle, the maintainer may pick it up -from the list and queue it to `pu`, in order to make it easier for +from the list and queue it to `seen`, in order to make it easier for people play with it without having to pick up and apply the patch to their trees themselves. @@ -434,7 +435,7 @@ their trees themselves. master. `git pull --rebase` will automatically skip already-applied patches, and will let you know. This works only if you rebase on top of the branch in which your patch has been merged (i.e. it will not - tell you if your patch is merged in pu if you rebase on top of + tell you if your patch is merged in `seen` if you rebase on top of master). * Read the Git mailing list, the maintainer regularly posts messages diff --git a/Documentation/config/diff.txt b/Documentation/config/diff.txt index ff09f1cf73..c3ae136eba 100644 --- a/Documentation/config/diff.txt +++ b/Documentation/config/diff.txt @@ -105,6 +105,10 @@ diff.mnemonicPrefix:: diff.noprefix:: If set, 'git diff' does not show any source or destination prefix. +diff.relative:: + If set to 'true', 'git diff' does not show changes outside of the directory + and show pathnames relative to the current directory. + diff.orderFile:: File indicating how to order files within a diff. See the '-O' option to linkgit:git-diff[1] for details. diff --git a/Documentation/config/feature.txt b/Documentation/config/feature.txt index 4e3a5c0ceb..c0cbf2bb1c 100644 --- a/Documentation/config/feature.txt +++ b/Documentation/config/feature.txt @@ -15,13 +15,9 @@ feature.experimental:: * `fetch.negotiationAlgorithm=skipping` may improve fetch negotiation times by skipping more commits at a time, reducing the number of round trips. + -* `fetch.writeCommitGraph=true` writes a commit-graph after every `git fetch` -command that downloads a pack-file from a remote. Using the `--split` option, -most executions will create a very small commit-graph file on top of the -existing commit-graph file(s). Occasionally, these files will merge and the -write may take longer. Having an updated commit-graph file helps performance -of many Git commands, including `git merge-base`, `git push -f`, and -`git log --graph`. +* `protocol.version=2` speeds up fetches from repositories with many refs by +allowing the client to specify which refs to list before the server lists +them. feature.manyFiles:: Enable config options that optimize for repos with many files in the diff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt index b1a9b1461d..b20394038d 100644 --- a/Documentation/config/fetch.txt +++ b/Documentation/config/fetch.txt @@ -90,5 +90,4 @@ fetch.writeCommitGraph:: the existing commit-graph file(s). Occasionally, these files will merge and the write may take longer. Having an updated commit-graph file helps performance of many Git commands, including `git merge-base`, - `git push -f`, and `git log --graph`. Defaults to false, unless - `feature.experimental` is true. + `git push -f`, and `git log --graph`. Defaults to false. diff --git a/Documentation/config/init.txt b/Documentation/config/init.txt index 46fa8c6a08..dc77f8c844 100644 --- a/Documentation/config/init.txt +++ b/Documentation/config/init.txt @@ -1,3 +1,7 @@ init.templateDir:: Specify the directory from which templates will be copied. (See the "TEMPLATE DIRECTORY" section of linkgit:git-init[1].) + +init.defaultBranch:: + Allows overriding the default branch name e.g. when initializing + a new repository or when cloning an empty repository. diff --git a/Documentation/config/protocol.txt b/Documentation/config/protocol.txt index 0b40141613..c46e9b3d00 100644 --- a/Documentation/config/protocol.txt +++ b/Documentation/config/protocol.txt @@ -48,7 +48,8 @@ protocol.version:: If set, clients will attempt to communicate with a server using the specified protocol version. If the server does not support it, communication falls back to version 0. - If unset, the default is `0`. + If unset, the default is `0`, unless `feature.experimental` + is enabled, in which case the default is `2`. Supported versions: + -- diff --git a/Documentation/diff-options.txt b/Documentation/diff-options.txt index bb31f0c42b..7987d72b02 100644 --- a/Documentation/diff-options.txt +++ b/Documentation/diff-options.txt @@ -643,15 +643,18 @@ ifndef::git-format-patch[] -R:: Swap two inputs; that is, show differences from index or on-disk file to tree contents. +endif::git-format-patch[] --relative[=<path>]:: +--no-relative:: When run from a subdirectory of the project, it can be told to exclude changes outside the directory and show pathnames relative to it with this option. When you are not in a subdirectory (e.g. in a bare repository), you can name which subdirectory to make the output relative to by giving a <path> as an argument. -endif::git-format-patch[] + `--no-relative` can be used to countermand both `diff.relative` config + option and previous `--relative`. -a:: --text:: diff --git a/Documentation/git-branch.txt b/Documentation/git-branch.txt index 135206ff4a..03c0824d52 100644 --- a/Documentation/git-branch.txt +++ b/Documentation/git-branch.txt @@ -12,7 +12,7 @@ SYNOPSIS [-v [--abbrev=<length> | --no-abbrev]] [--column[=<options>] | --no-column] [--sort=<key>] [(--merged | --no-merged) [<commit>]] - [--contains [<commit]] [--no-contains [<commit>]] + [--contains [<commit>]] [--no-contains [<commit>]] [--points-at <object>] [--format=<format>] [(-r | --remotes) | (-a | --all)] [--list] [<pattern>...] diff --git a/Documentation/git-bugreport.txt b/Documentation/git-bugreport.txt index 9edad66a63..66e88c2e31 100644 --- a/Documentation/git-bugreport.txt +++ b/Documentation/git-bugreport.txt @@ -29,6 +29,7 @@ The following information is captured automatically: - uname sysname, release, version, and machine strings - Compiler-specific info string - A list of enabled hooks + - $SHELL This tool is invoked via the typical Git setup process, which means that in some cases, it might not be able to launch - for example, if a relevant config file diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt index 8eca671b82..8e192d87db 100644 --- a/Documentation/git-cat-file.txt +++ b/Documentation/git-cat-file.txt @@ -10,7 +10,7 @@ SYNOPSIS -------- [verse] 'git cat-file' (-t [--allow-unknown-type]| -s [--allow-unknown-type]| -e | -p | <type> | --textconv | --filters ) [--path=<path>] <object> -'git cat-file' (--batch | --batch-check) [ --textconv | --filters ] [--follow-symlinks] +'git cat-file' (--batch[=<format>] | --batch-check[=<format>]) [ --textconv | --filters ] [--follow-symlinks] DESCRIPTION ----------- diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt index 08d6045c4a..c898310099 100644 --- a/Documentation/git-clone.txt +++ b/Documentation/git-clone.txt @@ -259,7 +259,7 @@ maintain a branch with no references other than a single cloned branch. This is useful e.g. to maintain minimal clones of the default branch of some repository for search indexing. ---recurse-submodules[=<pathspec]:: +--recurse-submodules[=<pathspec>]:: After the clone is created, initialize and clone submodules within based on the provided pathspec. If no pathspec is provided, all submodules are initialized and cloned. diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt index a3d996787b..17405c73a9 100644 --- a/Documentation/git-commit-graph.txt +++ b/Documentation/git-commit-graph.txt @@ -47,8 +47,10 @@ with `--stdin-commits` or `--reachable`.) + With the `--stdin-commits` option, generate the new commit graph by walking commits starting at the commits specified in stdin as a list -of OIDs in hex, one OID per line. (Cannot be combined with -`--stdin-packs` or `--reachable`.) +of OIDs in hex, one OID per line. OIDs that resolve to non-commits +(either directly, or by peeling tags) are silently ignored. OIDs that +are malformed, or do not exist generate an error. (Cannot be combined +with `--stdin-packs` or `--reachable`.) + With the `--reachable` option, generate the new commit graph by walking commits starting at all refs. (Cannot be combined with `--stdin-commits` @@ -60,7 +62,10 @@ existing commit-graph file. With the `--changed-paths` option, compute and write information about the paths changed between a commit and its first parent. This operation can take a while on large repositories. It provides significant performance gains -for getting history of a directory or a file with `git log -- <path>`. +for getting history of a directory or a file with `git log -- <path>`. If +this option is given, future commit-graph writes will automatically assume +that this option was intended. Use `--no-changed-paths` to stop storing this +data. + With the `--split[=<strategy>]` option, write the commit-graph as a chain of multiple commit-graph files stored in diff --git a/Documentation/git-diff.txt b/Documentation/git-diff.txt index 37781cf175..727f24d16e 100644 --- a/Documentation/git-diff.txt +++ b/Documentation/git-diff.txt @@ -11,15 +11,17 @@ SYNOPSIS [verse] 'git diff' [<options>] [<commit>] [--] [<path>...] 'git diff' [<options>] --cached [<commit>] [--] [<path>...] -'git diff' [<options>] <commit> <commit> [--] [<path>...] +'git diff' [<options>] <commit> [<commit>...] <commit> [--] [<path>...] +'git diff' [<options>] <commit>...<commit> [--] [<path>...] 'git diff' [<options>] <blob> <blob> 'git diff' [<options>] --no-index [--] <path> <path> DESCRIPTION ----------- Show changes between the working tree and the index or a tree, changes -between the index and a tree, changes between two trees, changes between -two blob objects, or changes between two files on disk. +between the index and a tree, changes between two trees, changes resulting +from a merge, changes between two blob objects, or changes between two +files on disk. 'git diff' [<options>] [--] [<path>...]:: @@ -61,9 +63,19 @@ two blob objects, or changes between two files on disk. This is to view the changes between two arbitrary <commit>. +'git diff' [<options>] <commit> <commit>... <commit> [--] [<path>...]:: + + This form is to view the results of a merge commit. The first + listed <commit> must be the merge itself; the remaining two or + more commits should be its parents. A convenient way to produce + the desired set of revisions is to use the {caret}@ suffix. + For instance, if `master` names a merge commit, `git diff master + master^@` gives the same combined diff as `git show master`. + 'git diff' [<options>] <commit>..<commit> [--] [<path>...]:: - This is synonymous to the previous form. If <commit> on + This is synonymous to the earlier form (without the "..") for + viewing the changes between two arbitrary <commit>. If <commit> on one side is omitted, it will have the same effect as using HEAD instead. @@ -196,7 +208,8 @@ linkgit:git-difftool[1], linkgit:git-log[1], linkgit:gitdiffcore[7], linkgit:git-format-patch[1], -linkgit:git-apply[1] +linkgit:git-apply[1], +linkgit:git-show[1] GIT --- diff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt index e8950de3ba..1978dbdc6a 100644 --- a/Documentation/git-fast-export.txt +++ b/Documentation/git-fast-export.txt @@ -119,6 +119,11 @@ by keeping the marks the same across runs. the shape of the history and stored tree. See the section on `ANONYMIZING` below. +--anonymize-map=<from>[:<to>]:: + Convert token `<from>` to `<to>` in the anonymized output. If + `<to>` is omitted, map `<from>` to itself (i.e., do not + anonymize it). See the section on `ANONYMIZING` below. + --reference-excluded-parents:: By default, running a command such as `git fast-export master~5..master` will not include the commit master{tilde}5 @@ -238,6 +243,30 @@ collapse "User 0", "User 1", etc into "User X"). This produces a much smaller output, and it is usually easy to quickly confirm that there is no private data in the stream. +Reproducing some bugs may require referencing particular commits or +paths, which becomes challenging after refnames and paths have been +anonymized. You can ask for a particular token to be left as-is or +mapped to a new value. For example, if you have a bug which reproduces +with `git rev-list sensitive -- secret.c`, you can run: + +--------------------------------------------------- +$ git fast-export --anonymize --all \ + --anonymize-map=sensitive:foo \ + --anonymize-map=secret.c:bar.c \ + >stream +--------------------------------------------------- + +After importing the stream, you can then run `git rev-list foo -- bar.c` +in the anonymized repository. + +Note that paths and refnames are split into tokens at slash boundaries. +The command above would anonymize `subdir/secret.c` as something like +`path123/bar.c`; you could then search for `bar.c` in the anonymized +repository to determine the final pathname. + +To make referencing the final pathname simpler, you can map each path +component; so if you also anonymize `subdir` to `publicdir`, then the +final pathname would be `publicdir/bar.c`. LIMITATIONS ----------- diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt index 77c6b3d001..7d9aad2a7e 100644 --- a/Documentation/git-fast-import.txt +++ b/Documentation/git-fast-import.txt @@ -293,7 +293,14 @@ by users who are located in the same location and time zone. In this case a reasonable offset from UTC could be assumed. + Unlike the `rfc2822` format, this format is very strict. Any -variation in formatting will cause fast-import to reject the value. +variation in formatting will cause fast-import to reject the value, +and some sanity checks on the numeric values may also be performed. + +`raw-permissive`:: + This is the same as `raw` except that no sanity checks on + the numeric epoch and local offset are performed. This can + be useful when trying to filter or import an existing history + with e.g. bogus timezone values. `rfc2822`:: This is the standard email format as described by RFC 2822. diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt index 5b1909fdf4..45b6d8e633 100644 --- a/Documentation/git-fetch.txt +++ b/Documentation/git-fetch.txt @@ -255,14 +255,14 @@ refspec. * Using refspecs explicitly: + ------------------------------------------------ -$ git fetch origin +pu:pu maint:tmp +$ git fetch origin +seen:seen maint:tmp ------------------------------------------------ + -This updates (or creates, as necessary) branches `pu` and `tmp` in +This updates (or creates, as necessary) branches `seen` and `tmp` in the local repository by fetching from the branches (respectively) -`pu` and `maint` from the remote repository. +`seen` and `maint` from the remote repository. + -The `pu` branch will be updated even if it does not fast-forward, +The `seen` branch will be updated even if it does not fast-forward, because it is prefixed with a plus sign; `tmp` will not be. * Peek at a remote's branch, without configuring the remote in your local diff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt index 6dcd39f6f6..2ea71c5f6c 100644 --- a/Documentation/git-for-each-ref.txt +++ b/Documentation/git-for-each-ref.txt @@ -232,12 +232,27 @@ Fields that have name-email-date tuple as its value (`author`, `committer`, and `tagger`) can be suffixed with `name`, `email`, and `date` to extract the named component. -The complete message in a commit and tag object is `contents`. -Its first line is `contents:subject`, where subject is the concatenation -of all lines of the commit message up to the first blank line. The next -line is `contents:body`, where body is all of the lines after the first -blank line. The optional GPG signature is `contents:signature`. The -first `N` lines of the message is obtained using `contents:lines=N`. +The message in a commit or a tag object is `contents`, from which +`contents:<part>` can be used to extract various parts out of: + +contents:size:: + The size in bytes of the commit or tag message. + +contents:subject:: + The first paragraph of the message, which typically is a + single line, is taken as the "subject" of the commit or the + tag message. + +contents:body:: + The remainder of the commit or the tag message that follows + the "subject". + +contents:signature:: + The optional GPG signature of the tag. + +contents:lines=N:: + The first `N` lines of the message. + Additionally, the trailers as interpreted by linkgit:git-interpret-trailers[1] are obtained as `trailers` (or by using the historical alias `contents:trailers`). Non-trailer lines from the trailer block can be omitted diff --git a/Documentation/git-help.txt b/Documentation/git-help.txt index f71db0daa2..69c0c5c34e 100644 --- a/Documentation/git-help.txt +++ b/Documentation/git-help.txt @@ -8,7 +8,7 @@ git-help - Display help information about Git SYNOPSIS -------- [verse] -'git help' [-a|--all [--[no-]verbose]] [-g|--guide] +'git help' [-a|--all [--[no-]verbose]] [-g|--guides] [-i|--info|-m|--man|-w|--web] [COMMAND|GUIDE] DESCRIPTION @@ -21,7 +21,7 @@ on the standard output. If the option `--all` or `-a` is given, all available commands are printed on the standard output. -If the option `--guide` or `-g` is given, a list of the useful +If the option `--guides` or `-g` is given, a list of the useful Git guides is also printed on the standard output. If a command, or a guide, is given, a manual page for that command or diff --git a/Documentation/git-http-fetch.txt b/Documentation/git-http-fetch.txt index 666b042679..4deb4893f5 100644 --- a/Documentation/git-http-fetch.txt +++ b/Documentation/git-http-fetch.txt @@ -9,7 +9,7 @@ git-http-fetch - Download from a remote Git repository via HTTP SYNOPSIS -------- [verse] -'git http-fetch' [-c] [-t] [-a] [-d] [-v] [-w filename] [--recover] [--stdin] <commit> <url> +'git http-fetch' [-c] [-t] [-a] [-d] [-v] [-w filename] [--recover] [--stdin | --packfile=<hash> | <commit>] <url> DESCRIPTION ----------- @@ -40,6 +40,13 @@ commit-id:: <commit-id>['\t'<filename-as-in--w>] +--packfile=<hash>:: + Instead of a commit id on the command line (which is not expected in + this case), 'git http-fetch' fetches the packfile directly at the given + URL and uses index-pack to generate corresponding .idx and .keep files. + The hash is used to determine the name of the temporary file and is + arbitrary. The output of index-pack is printed to stdout. + --recover:: Verify that everything reachable from target is fetched. Used after an earlier fetch is interrupted. diff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt index d5b7560bfe..9316d9a80b 100644 --- a/Documentation/git-index-pack.txt +++ b/Documentation/git-index-pack.txt @@ -93,6 +93,14 @@ OPTIONS --max-input-size=<size>:: Die, if the pack is larger than <size>. +--object-format=<hash-algorithm>:: + Specify the given object format (hash algorithm) for the pack. The valid + values are 'sha1' and (if enabled) 'sha256'. The default is the algorithm for + the current repository (set by `extensions.objectFormat`), or 'sha1' if no + value is set or outside a repository. ++ +This option cannot be used with --stdin. + NOTES ----- diff --git a/Documentation/git-init.txt b/Documentation/git-init.txt index adc6adfd38..ddfe265da5 100644 --- a/Documentation/git-init.txt +++ b/Documentation/git-init.txt @@ -10,7 +10,8 @@ SYNOPSIS -------- [verse] 'git init' [-q | --quiet] [--bare] [--template=<template_directory>] - [--separate-git-dir <git dir>] [--object-format=<format] + [--separate-git-dir <git dir>] [--object-format=<format>] + [-b <branch-name> | --initial-branch=<branch-name>] [--shared[=<permissions>]] [directory] @@ -67,6 +68,12 @@ repository. + If this is reinitialization, the repository will be moved to the specified path. +-b <branch-name:: +--initial-branch=<branch-name>:: + +Use the specified name for the initial branch in the newly created repository. +If not specified, fall back to the default name: `master`. + --shared[=(false|true|umask|group|all|world|everybody|0xxx)]:: Specify that the Git repository is to be shared amongst several users. This diff --git a/Documentation/git-log.txt b/Documentation/git-log.txt index 20e6d21a74..3fd26d5212 100644 --- a/Documentation/git-log.txt +++ b/Documentation/git-log.txt @@ -15,9 +15,12 @@ DESCRIPTION ----------- Shows the commit logs. -The command takes options applicable to the `git rev-list` +:git-log: 1 +include::rev-list-description.txt[] + +The command takes options applicable to the linkgit:git-rev-list[1] command to control what is shown and how, and options applicable to -the `git diff-*` commands to control how the changes +the linkgit:git-diff[1] command to control how the changes each commit introduces are shown. diff --git a/Documentation/git-ls-remote.txt b/Documentation/git-ls-remote.txt index 0a5c8b7d49..492e573856 100644 --- a/Documentation/git-ls-remote.txt +++ b/Documentation/git-ls-remote.txt @@ -101,9 +101,9 @@ f25a265a342aed6041ab0cc484224d9ca54b6f41 refs/tags/v0.99.1 7ceca275d047c90c0c7d5afb13ab97efdf51bd6e refs/tags/v0.99.3 c5db5456ae3b0873fc659c19fafdde22313cc441 refs/tags/v0.99.2 0918385dbd9656cab0d1d81ba7453d49bbc16250 refs/tags/junio-gpg-pub -$ git ls-remote http://www.kernel.org/pub/scm/git/git.git master pu rc +$ git ls-remote http://www.kernel.org/pub/scm/git/git.git master seen rc 5fe978a5381f1fbad26a80e682ddd2a401966740 refs/heads/master -c781a84b5204fb294c9ccc79f8b3baceeb32c061 refs/heads/pu +c781a84b5204fb294c9ccc79f8b3baceeb32c061 refs/heads/seen $ git remote add korg http://www.kernel.org/pub/scm/git/git.git $ git ls-remote --tags korg v\* d6602ec5194c87b0fc87103ca4d67251c76f233a refs/tags/v0.99 diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt index 4624cfd288..374d2486f7 100644 --- a/Documentation/git-rebase.txt +++ b/Documentation/git-rebase.txt @@ -259,7 +259,7 @@ See also INCOMPATIBLE OPTIONS below. unchanged as a result. If a temporary stash entry was created using --autostash, it will be saved to the stash list. ---apply: +--apply:: Use applying strategies to rebase (calling `git-am` internally). This option may become a no-op in the future once the merge backend handles everything the apply one does. diff --git a/Documentation/git-rev-list.txt b/Documentation/git-rev-list.txt index 025c911436..5da66232dc 100644 --- a/Documentation/git-rev-list.txt +++ b/Documentation/git-rev-list.txt @@ -14,44 +14,8 @@ SYNOPSIS DESCRIPTION ----------- -List commits that are reachable by following the `parent` links from the -given commit(s), but exclude commits that are reachable from the one(s) -given with a '{caret}' in front of them. The output is given in reverse -chronological order by default. - -You can think of this as a set operation. Commits given on the command -line form a set of commits that are reachable from any of them, and then -commits reachable from any of the ones given with '{caret}' in front are -subtracted from that set. The remaining commits are what comes out in the -command's output. Various other options and paths parameters can be used -to further limit the result. - -Thus, the following command: - ------------------------------------------------------------------------ - $ git rev-list foo bar ^baz ------------------------------------------------------------------------ - -means "list all the commits which are reachable from 'foo' or 'bar', but -not from 'baz'". - -A special notation "'<commit1>'..'<commit2>'" can be used as a -short-hand for "{caret}'<commit1>' '<commit2>'". For example, either of -the following may be used interchangeably: - ------------------------------------------------------------------------ - $ git rev-list origin..HEAD - $ git rev-list HEAD ^origin ------------------------------------------------------------------------ - -Another special notation is "'<commit1>'...'<commit2>'" which is useful -for merges. The resulting set of commits is the symmetric difference -between the two operands. The following two commands are equivalent: - ------------------------------------------------------------------------ - $ git rev-list A B --not $(git merge-base --all A B) - $ git rev-list A...B ------------------------------------------------------------------------ +:git-rev-list: 1 +include::rev-list-description.txt[] 'rev-list' is a very essential Git command, since it provides the ability to build and traverse commit ancestry graphs. For diff --git a/Documentation/git-show-index.txt b/Documentation/git-show-index.txt index 424e4ba84c..39b1d8eaa1 100644 --- a/Documentation/git-show-index.txt +++ b/Documentation/git-show-index.txt @@ -9,7 +9,7 @@ git-show-index - Show packed archive index SYNOPSIS -------- [verse] -'git show-index' +'git show-index' [--object-format=<hash-algorithm>] DESCRIPTION @@ -36,6 +36,15 @@ Note that you can get more information on a packfile by calling linkgit:git-verify-pack[1]. However, as this command considers only the index file itself, it's both faster and more flexible. +OPTIONS +------- + +--object-format=<hash-algorithm>:: + Specify the given object format (hash algorithm) for the index file. The + valid values are 'sha1' and (if enabled) 'sha256'. The default is the + algorithm for the current repository (set by `extensions.objectFormat`), or + 'sha1' if no value is set or outside a repository.. + GIT --- Part of the linkgit:git[1] suite diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index 7c8943af7a..a0eeaeb02e 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -200,10 +200,32 @@ directory. SUBMODULES ---------- -If your repository contains one or more submodules, then those submodules will -appear based on which you initialized with the `git submodule` command. If -your sparse-checkout patterns exclude an initialized submodule, then that -submodule will still appear in your working directory. +If your repository contains one or more submodules, then submodules +are populated based on interactions with the `git submodule` command. +Specifically, `git submodule init -- <path>` will ensure the submodule +at `<path>` is present, while `git submodule deinit [-f] -- <path>` +will remove the files for the submodule at `<path>` (including any +untracked files, uncommitted changes, and unpushed history). Similar +to how sparse-checkout removes files from the working tree but still +leaves entries in the index, deinitialized submodules are removed from +the working directory but still have an entry in the index. + +Since submodules may have unpushed changes or untracked files, +removing them could result in data loss. Thus, changing sparse +inclusion/exclusion rules will not cause an already checked out +submodule to be removed from the working copy. Said another way, just +as `checkout` will not cause submodules to be automatically removed or +initialized even when switching between branches that remove or add +submodules, using `sparse-checkout` to reduce or expand the scope of +"interesting" files will not cause submodules to be automatically +deinitialized or initialized either. + +Further, the above facts mean that there are multiple reasons that +"tracked" files might not be present in the working copy: sparsity +pattern application from sparse-checkout, and submodule initialization +state. Thus, commands like `git grep` that work on tracked files in +the working copy may return results that are limited by either or both +of these restrictions. SEE ALSO diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt index c9ed2bf3d5..7e5f995f77 100644 --- a/Documentation/git-submodule.txt +++ b/Documentation/git-submodule.txt @@ -183,7 +183,7 @@ set-branch (-d|--default) [--] <path>:: Sets the default remote tracking branch for the submodule. The `--branch` option allows the remote branch to be specified. The `--default` option removes the submodule.<name>.branch configuration - key, which causes the tracking branch to default to 'master'. + key, which causes the tracking branch to default to the remote 'HEAD'. set-url [--] <path> <newurl>:: Sets the URL of the specified submodule to <newurl>. Then, it will @@ -284,7 +284,7 @@ OPTIONS `.gitmodules` for `update --remote`. A special value of `.` is used to indicate that the name of the branch in the submodule should be the same name as the current branch in the current repository. If the - option is not specified, it defaults to 'master'. + option is not specified, it defaults to the remote 'HEAD'. -f:: --force:: @@ -322,10 +322,10 @@ OPTIONS the superproject's recorded SHA-1 to update the submodule, use the status of the submodule's remote-tracking branch. The remote used is branch's remote (`branch.<name>.remote`), defaulting to `origin`. - The remote branch used defaults to `master`, but the branch name may - be overridden by setting the `submodule.<name>.branch` option in - either `.gitmodules` or `.git/config` (with `.git/config` taking - precedence). + The remote branch used defaults to the remote `HEAD`, but the branch + name may be overridden by setting the `submodule.<name>.branch` + option in either `.gitmodules` or `.git/config` (with `.git/config` + taking precedence). + This works for any of the supported update procedures (`--checkout`, `--rebase`, etc.). The only change is the source of the target SHA-1. diff --git a/Documentation/git-worktree.txt b/Documentation/git-worktree.txt index 85d92c9761..4796c3c05e 100644 --- a/Documentation/git-worktree.txt +++ b/Documentation/git-worktree.txt @@ -126,7 +126,9 @@ OPTIONS locked working tree path, specify `--force` twice. + `move` refuses to move a locked working tree unless `--force` is specified -twice. +twice. If the destination is already assigned to some other working tree but is +missing (for instance, if `<new-path>` was deleted manually), then `--force` +allows the move to proceed; use --force twice if the destination is locked. + `remove` refuses to remove an unclean working tree unless `--force` is used. To remove a locked working tree, specify `--force` twice. diff --git a/Documentation/git.txt b/Documentation/git.txt index 12890841c4..3e50065198 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -721,8 +721,6 @@ of clones and fetches. Enables a curl full trace dump of all incoming and outgoing data, including descriptive information, of the git transport protocol. This is similar to doing curl `--trace-ascii` on the command line. - This option overrides setting the `GIT_CURL_VERBOSE` environment - variable. See `GIT_TRACE` for available trace output options. `GIT_TRACE_CURL_NO_DATA`:: @@ -777,11 +775,10 @@ for full details. See `GIT_TRACE2` for available trace output options and link:technical/api-trace2.html[Trace2 documentation] for full details. -`GIT_REDACT_COOKIES`:: - This can be set to a comma-separated list of strings. When a curl trace - is enabled (see `GIT_TRACE_CURL` above), whenever a "Cookies:" header - sent by the client is dumped, values of cookies whose key is in that - list (case-sensitive) are redacted. +`GIT_TRACE_REDACT`:: + By default, when tracing is activated, Git redacts the values of + cookies, the "Authorization:" header, and the "Proxy-Authorization:" + header. Set this variable to `0` to prevent this redaction. `GIT_LITERAL_PATHSPECS`:: Setting this variable to `1` will cause Git to treat all diff --git a/Documentation/giteveryday.txt b/Documentation/giteveryday.txt index 1bd919f92b..faba2ef088 100644 --- a/Documentation/giteveryday.txt +++ b/Documentation/giteveryday.txt @@ -278,13 +278,13 @@ $ git am -3 -i -s ./+to-apply <4> $ compile/test $ git switch -c hold/linus && git am -3 -i -s ./+hold-linus <5> $ git switch topic/one && git rebase master <6> -$ git switch -C pu next <7> +$ git switch -C seen next <7> $ git merge topic/one topic/two && git merge hold/linus <8> $ git switch maint $ git cherry-pick master~4 <9> $ compile/test $ git tag -s -m "GIT 0.99.9x" v0.99.9x <10> -$ git fetch ko && for branch in master maint next pu <11> +$ git fetch ko && for branch in master maint next seen <11> do git show-branch ko/$branch $branch <12> done @@ -294,14 +294,14 @@ $ git push --follow-tags ko <13> <1> see what you were in the middle of doing, if anything. <2> see which branches haven't been merged into `master` yet. Likewise for any other integration branches e.g. `maint`, `next` -and `pu` (potential updates). +and `seen`. <3> read mails, save ones that are applicable, and save others that are not quite ready (other mail readers are available). <4> apply them, interactively, with your sign-offs. <5> create topic branch as needed and apply, again with sign-offs. <6> rebase internal topic branch that has not been merged to the master or exposed as a part of a stable branch. -<7> restart `pu` every time from the next. +<7> restart `seen` every time from the next. <8> and bundle topic branches still cooking. <9> backport a critical fix. <10> create a signed tag. @@ -323,7 +323,7 @@ repository at kernel.org, and looks like this: fetch = refs/heads/*:refs/remotes/ko/* push = refs/heads/master push = refs/heads/next - push = +refs/heads/pu + push = +refs/heads/seen push = refs/heads/maint ------------ diff --git a/Documentation/githooks.txt b/Documentation/githooks.txt index 81f2a87e88..31b601e4bc 100644 --- a/Documentation/githooks.txt +++ b/Documentation/githooks.txt @@ -404,6 +404,35 @@ Both standard output and standard error output are forwarded to `git send-pack` on the other end, so you can simply `echo` messages for the user. +reference-transaction +~~~~~~~~~~~~~~~~~~~~~ + +This hook is invoked by any Git command that performs reference +updates. It executes whenever a reference transaction is prepared, +committed or aborted and may thus get called multiple times. + +The hook takes exactly one argument, which is the current state the +given reference transaction is in: + + - "prepared": All reference updates have been queued to the + transaction and references were locked on disk. + + - "committed": The reference transaction was committed and all + references now have their respective new value. + + - "aborted": The reference transaction was aborted, no changes + were performed and the locks have been released. + +For each reference update that was added to the transaction, the hook +receives on standard input a line of the format: + + <old-value> SP <new-value> SP <ref-name> LF + +The exit status of the hook is ignored for any state except for the +"prepared" state. In the "prepared" state, a non-zero exit status will +cause the transaction to be aborted. The hook will not be called with +"aborted" state in that case. + push-to-checkout ~~~~~~~~~~~~~~~~ diff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt index 67275fd187..539b4e1997 100644 --- a/Documentation/gitmodules.txt +++ b/Documentation/gitmodules.txt @@ -49,9 +49,9 @@ submodule.<name>.update:: submodule.<name>.branch:: A remote branch name for tracking updates in the upstream submodule. - If the option is not specified, it defaults to 'master'. A special - value of `.` is used to indicate that the name of the branch in the - submodule should be the same name as the current branch in the + If the option is not specified, it defaults to the remote 'HEAD'. + A special value of `.` is used to indicate that the name of the branch + in the submodule should be the same name as the current branch in the current repository. See the `--remote` documentation in linkgit:git-submodule[1] for details. diff --git a/Documentation/gitremote-helpers.txt b/Documentation/gitremote-helpers.txt index f48a031dc3..6f1e269ae4 100644 --- a/Documentation/gitremote-helpers.txt +++ b/Documentation/gitremote-helpers.txt @@ -238,6 +238,9 @@ the remote repository. `--signed-tags=verbatim` to linkgit:git-fast-export[1]. In the absence of this capability, Git will use `--signed-tags=warn-strip`. +'object-format':: + This indicates that the helper is able to interact with the remote + side using an explicit hash algorithm extension. COMMANDS @@ -257,12 +260,14 @@ Support for this command is mandatory. 'list':: Lists the refs, one per line, in the format "<value> <name> [<attr> ...]". The value may be a hex sha1 hash, "@<dest>" for - a symref, or "?" to indicate that the helper could not get the - value of the ref. A space-separated list of attributes follows - the name; unrecognized attributes are ignored. The list ends - with a blank line. + a symref, ":<keyword> <value>" for a key-value pair, or + "?" to indicate that the helper could not get the value of the + ref. A space-separated list of attributes follows the name; + unrecognized attributes are ignored. The list ends with a + blank line. + See REF LIST ATTRIBUTES for a list of currently defined attributes. +See REF LIST KEYWORDS for a list of currently defined keywords. + Supported if the helper has the "fetch" or "import" capability. @@ -405,7 +410,9 @@ Supported if the helper has the "connect" capability. trying to fall back). After line feed terminating the positive (empty) response, the output of the service starts. Messages (both request and response) must consist of zero or more - PKT-LINEs, terminating in a flush packet. The client must not + PKT-LINEs, terminating in a flush packet. Response messages will + then have a response end packet after the flush packet to + indicate the end of a response. The client must not expect the server to store any state in between request-response pairs. After the connection ends, the remote helper exits. + @@ -430,6 +437,18 @@ attributes are defined. This ref is unchanged since the last import or fetch, although the helper cannot necessarily determine what value that produced. +REF LIST KEYWORDS +----------------- + +The 'list' command may produce a list of key-value pairs. +The following keys are defined. + +'object-format':: + The refs are using the given hash algorithm. This keyword is only + used if the server and client both support the object-format + extension. + + OPTIONS ------- @@ -514,6 +533,14 @@ set by Git if the remote helper has the 'option' capability. transaction. If successful, all refs will be updated, or none will. If the remote side does not support this capability, the push will fail. +'option object-format' {'true'|algorithm}:: + If 'true', indicate that the caller wants hash algorithm information + to be passed back from the remote. This mode is used when fetching + refs. ++ +If set to an algorithm, indicate that the caller wants to interact with +the remote side using that algorithm. + SEE ALSO -------- linkgit:git-remote[1] diff --git a/Documentation/gitworkflows.txt b/Documentation/gitworkflows.txt index abc0dc6bc7..47cf97f9be 100644 --- a/Documentation/gitworkflows.txt +++ b/Documentation/gitworkflows.txt @@ -85,15 +85,15 @@ As a given feature goes from experimental to stable, it also There is a fourth official branch that is used slightly differently: -* 'pu' (proposed updates) is an integration branch for things that are - not quite ready for inclusion yet (see "Integration Branches" - below). +* 'seen' (patches seen by the maintainer) is an integration branch for + things that are not quite ready for inclusion yet (see "Integration + Branches" below). Each of the four branches is usually a direct descendant of the one above it. Conceptually, the feature enters at an unstable branch (usually 'next' -or 'pu'), and "graduates" to 'master' for the next release once it is +or 'seen'), and "graduates" to 'master' for the next release once it is considered stable enough. @@ -207,7 +207,7 @@ If you make it (very) clear that this branch is going to be deleted right after the testing, you can even publish this branch, for example to give the testers a chance to work with it, or other developers a chance to see if their in-progress work will be compatible. `git.git` -has such an official throw-away integration branch called 'pu'. +has such an official throw-away integration branch called 'seen'. Branch management for a release @@ -291,8 +291,8 @@ This will not happen if the content of the branches was verified as described in the previous section. -Branch management for next and pu after a feature release -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Branch management for next and seen after a feature release +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ After a feature release, the integration branch 'next' may optionally be rewound and rebuilt from the tip of 'master' using the surviving @@ -319,8 +319,8 @@ so. If you do this, then you should make a public announcement indicating that 'next' was rewound and rebuilt. -The same rewind and rebuild process may be followed for 'pu'. A public -announcement is not necessary since 'pu' is a throw-away branch, as +The same rewind and rebuild process may be followed for 'seen'. A public +announcement is not necessary since 'seen' is a throw-away branch, as described above. diff --git a/Documentation/howto/maintain-git.txt b/Documentation/howto/maintain-git.txt index 73be8b49f8..a67130debb 100644 --- a/Documentation/howto/maintain-git.txt +++ b/Documentation/howto/maintain-git.txt @@ -66,7 +66,7 @@ this mailing list after each feature release is made. demonstrated to be regression free. New changes are tested in 'next' before merged to 'master'. - - 'pu' branch is used to publish other proposed changes that do + - 'seen' branch is used to publish other proposed changes that do not yet pass the criteria set for 'next'. - The tips of 'master' and 'maint' branches will not be rewound to @@ -76,7 +76,7 @@ this mailing list after each feature release is made. of the cycle. - Usually 'master' contains all of 'maint' and 'next' contains all - of 'master'. 'pu' contains all the topics merged to 'next', but + of 'master'. 'seen' contains all the topics merged to 'next', but is rebuilt directly on 'master'. - The tip of 'master' is meant to be more stable than any @@ -229,12 +229,12 @@ by doing the following: series?) - Prepare 'jch' branch, which is used to represent somewhere - between 'master' and 'pu' and often is slightly ahead of 'next'. + between 'master' and 'seen' and often is slightly ahead of 'next'. - $ Meta/Reintegrate master..pu >Meta/redo-jch.sh + $ Meta/Reintegrate master..seen >Meta/redo-jch.sh The result is a script that lists topics to be merged in order to - rebuild 'pu' as the input to Meta/Reintegrate script. Remove + rebuild 'seen' as the input to Meta/Reintegrate script. Remove later topics that should not be in 'jch' yet. Add a line that consists of '### match next' before the name of the first topic in the output that should be in 'jch' but not in 'next' yet. @@ -291,29 +291,29 @@ by doing the following: merged to 'master'. This may lose '### match next' marker; add it again to the appropriate place when it happens. - - Rebuild 'pu'. + - Rebuild 'seen'. - $ Meta/Reintegrate master..pu >Meta/redo-pu.sh + $ Meta/Reintegrate master..seen >Meta/redo-seen.sh - Edit the result by adding new topics that are not still in 'pu' + Edit the result by adding new topics that are not still in 'seen' in the script. Then - $ git checkout -B pu jch - $ sh Meta/redo-pu.sh + $ git checkout -B seen jch + $ sh Meta/redo-seen.sh - When all is well, clean up the redo-pu.sh script with + When all is well, clean up the redo-seen.sh script with - $ sh Meta/redo-pu.sh -u + $ sh Meta/redo-seen.sh -u Double check by running - $ git branch --no-merged pu + $ git branch --no-merged seen to see there is no unexpected leftover topics. At this point, build-test the result for semantic conflicts, and if there are, prepare an appropriate merge-fix first (see - appendix), and rebuild the 'pu' branch from scratch, starting at + appendix), and rebuild the 'seen' branch from scratch, starting at the tip of 'jch'. - Update "What's cooking" message to review the updates to @@ -323,14 +323,14 @@ by doing the following: $ Meta/cook - This script inspects the history between master..pu, finds tips + This script inspects the history between master..seen, finds tips of topic branches, compares what it found with the current contents in Meta/whats-cooking.txt, and updates that file. - Topics not listed in the file but are found in master..pu are + Topics not listed in the file but are found in master..seen are added to the "New topics" section, topics listed in the file that - are no longer found in master..pu are moved to the "Graduated to + are no longer found in master..seen are moved to the "Graduated to master" section, and topics whose commits changed their states - (e.g. used to be only in 'pu', now merged to 'next') are updated + (e.g. used to be only in 'seen', now merged to 'next') are updated with change markers "<<" and ">>". Look for lines enclosed in "<<" and ">>"; they hold contents from @@ -360,7 +360,7 @@ Observations Some observations to be made. * Each topic is tested individually, and also together with other - topics cooking first in 'pu', then in 'jch' and then in 'next'. + topics cooking first in 'seen', then in 'jch' and then in 'next'. Until it matures, no part of it is merged to 'master'. * A topic already in 'next' can get fixes while still in @@ -411,7 +411,7 @@ new use of the variable under its old name. When these two topics are merged together, the reference to the variable newly added by the latter topic will still use the old name in the result. -The Meta/Reintegrate script that is used by redo-jch and redo-pu +The Meta/Reintegrate script that is used by redo-jch and redo-seen scripts implements a crude but usable way to work this issue around. When the script merges branch $X, it checks if "refs/merge-fix/$X" exists, and if so, the effect of it is squashed into the result of @@ -431,14 +431,14 @@ commit that can be squashed into a result of mechanical merge to correct semantic conflicts. After finding that the result of merging branch "ai/topic" to an -integration branch had such a semantic conflict, say pu~4, check the +integration branch had such a semantic conflict, say seen~4, check the problematic merge out on a detached HEAD, edit the working tree to fix the semantic conflict, and make a separate commit to record the fix-up: - $ git checkout pu~4 + $ git checkout seen~4 $ git show -s --pretty=%s ;# double check - Merge branch 'ai/topic' to pu + Merge branch 'ai/topic' to seen $ edit $ git commit -m 'merge-fix/ai/topic' -a @@ -450,9 +450,9 @@ result: Then double check the result by asking Meta/Reintegrate to redo the merge: - $ git checkout pu~5 ;# the parent of the problem merge + $ git checkout seen~5 ;# the parent of the problem merge $ echo ai/topic | Meta/Reintegrate - $ git diff pu~4 + $ git diff seen~4 This time, because you prepared refs/merge-fix/ai/topic, the resulting merge should have been tweaked to include the fix for the @@ -464,7 +464,7 @@ branch needs this merge-fix is because another branch merged earlier to the integration branch changed the underlying assumption ai/topic branch made (e.g. ai/topic branch added a site to refer to a variable, while the other branch renamed that variable and adjusted -existing use sites), and if you changed redo-jch (or redo-pu) script +existing use sites), and if you changed redo-jch (or redo-seen) script to merge ai/topic branch before the other branch, then the above merge-fix should not be applied while merging ai/topic, but should instead be applied while merging the other branch. You would need diff --git a/Documentation/howto/rebase-from-internal-branch.txt b/Documentation/howto/rebase-from-internal-branch.txt index 02cb5f758d..f2e10a7ec8 100644 --- a/Documentation/howto/rebase-from-internal-branch.txt +++ b/Documentation/howto/rebase-from-internal-branch.txt @@ -4,7 +4,7 @@ Cc: Petr Baudis <pasky@suse.cz>, Linus Torvalds <torvalds@osdl.org> Subject: Re: sending changesets from the middle of a git tree Date: Sun, 14 Aug 2005 18:37:39 -0700 Abstract: In this article, JC talks about how he rebases the - public "pu" branch using the core Git tools when he updates + public "seen" branch using the core Git tools when he updates the "master" branch, and how "rebase" works. Also discussed is how this applies to individual developers who sends patches upstream. @@ -20,8 +20,8 @@ Petr Baudis <pasky@suse.cz> writes: > where Junio C Hamano <junkio@cox.net> told me that... >> Linus Torvalds <torvalds@osdl.org> writes: >> ->> > Junio, maybe you want to talk about how you move patches from your "pu" ->> > branch to the real branches. +>> > Junio, maybe you want to talk about how you move patches from your +>> > "seen" branch to the real branches. >> > Actually, wouldn't this be also precisely for what StGIT is intended to? -------------------------------------- @@ -33,12 +33,12 @@ the kind of task StGIT is designed to do. I just have done a simpler one, this time using only the core Git tools. -I had a handful of commits that were ahead of master in pu, and I +I had a handful of commits that were ahead of master in 'seen', and I wanted to add some documentation bypassing my usual habit of -placing new things in pu first. At the beginning, the commit +placing new things in 'seen' first. At the beginning, the commit ancestry graph looked like this: - *"pu" head + *"seen" head master --> #1 --> #2 --> #3 So I started from master, made a bunch of edits, and committed: @@ -50,7 +50,7 @@ So I started from master, made a bunch of edits, and committed: After the commit, the ancestry graph would look like this: - *"pu" head + *"seen" head master^ --> #1 --> #2 --> #3 \ \---> master @@ -58,31 +58,31 @@ After the commit, the ancestry graph would look like this: The old master is now master^ (the first parent of the master). The new master commit holds my documentation updates. -Now I have to deal with "pu" branch. +Now I have to deal with "seen" branch. This is the kind of situation I used to have all the time when Linus was the maintainer and I was a contributor, when you look -at "master" branch being the "maintainer" branch, and "pu" +at "master" branch being the "maintainer" branch, and "seen" branch being the "contributor" branch. Your work started at the tip of the "maintainer" branch some time ago, you made a lot of progress in the meantime, and now the maintainer branch has some other commits you do not have yet. And "git rebase" was written with the explicit purpose of helping to maintain branches like -"pu". You _could_ merge master to pu and keep going, but if you +"seen". You _could_ merge master to 'seen' and keep going, but if you eventually want to cherrypick and merge some but not necessarily all changes back to the master branch, it often makes later operations for _you_ easier if you rebase (i.e. carry forward -your changes) "pu" rather than merge. So I ran "git rebase": +your changes) "seen" rather than merge. So I ran "git rebase": - $ git checkout pu - $ git rebase master pu + $ git checkout seen + $ git rebase master seen What this does is to pick all the commits since the current -branch (note that I now am on "pu" branch) forked from the +branch (note that I now am on "seen" branch) forked from the master branch, and forward port these changes. master^ --> #1 --> #2 --> #3 - \ *"pu" head + \ *"seen" head \---> master --> #1' --> #2' --> #3' The diff between master^ and #1 is applied to master and @@ -92,7 +92,7 @@ commits are made similarly out of #2 and #3 commits. Old #3 is not recorded in any of the .git/refs/heads/ file anymore, so after doing this you will have dangling commit if -you ran fsck-cache, which is normal. After testing "pu", you +you ran fsck-cache, which is normal. After testing "seen", you can run "git prune" to get rid of those original three commits. While I am talking about "git rebase", I should talk about how diff --git a/Documentation/howto/revert-branch-rebase.txt b/Documentation/howto/revert-branch-rebase.txt index 149508e13b..a3e5595a56 100644 --- a/Documentation/howto/revert-branch-rebase.txt +++ b/Documentation/howto/revert-branch-rebase.txt @@ -15,7 +15,7 @@ One of the changes I pulled into the 'master' branch turns out to break building Git with GCC 2.95. While they were well-intentioned portability fixes, keeping things working with gcc-2.95 was also important. Here is what I did to revert the change in the 'master' -branch and to adjust the 'pu' branch, using core Git tools and +branch and to adjust the 'seen' branch, using core Git tools and barebone Porcelain. First, prepare a throw-away branch in case I screw things up. @@ -104,11 +104,11 @@ $ git diff master..revert-c99 says nothing. -Then we rebase the 'pu' branch as usual. +Then we rebase the 'seen' branch as usual. ------------------------------------------------ -$ git checkout pu -$ git tag pu-anchor pu +$ git checkout seen +$ git tag seen-anchor seen $ git rebase master * Applying: Redo "revert" using three-way merge machinery. First trying simple merge strategy to cherry-pick. @@ -127,11 +127,11 @@ First trying simple merge strategy to cherry-pick. First trying simple merge strategy to cherry-pick. ------------------------------------------------ -The temporary tag 'pu-anchor' is me just being careful, in case 'git +The temporary tag 'seen-anchor' is me just being careful, in case 'git rebase' screws up. After this, I can do these for sanity check: ------------------------------------------------ -$ git diff pu-anchor..pu ;# make sure we got the master fix. +$ git diff seen-anchor..seen ;# make sure we got the master fix. $ make CC=gcc-2.95 clean test ;# make sure it fixed the breakage. $ make clean test ;# make sure it did not cause other breakage. ------------------------------------------------ @@ -140,7 +140,7 @@ Everything is in the good order. I do not need the temporary branch or tag anymore, so remove them: ------------------------------------------------ -$ rm -f .git/refs/tags/pu-anchor +$ rm -f .git/refs/tags/seen-anchor $ git branch -d revert-c99 ------------------------------------------------ @@ -168,18 +168,18 @@ Committed merge 7fb9b7262a1d1e0a47bbfdcbbcf50ce0635d3f8f And the final repository status looks like this: ------------------------------------------------ -$ git show-branch --more=1 master pu rc +$ git show-branch --more=1 master seen rc ! [master] Revert "Replace zero-length array decls with []." - ! [pu] git-repack: Add option to repack all objects. + ! [seen] git-repack: Add option to repack all objects. * [rc] Merge refs/heads/master from . --- - + [pu] git-repack: Add option to repack all objects. - + [pu~1] More documentation updates. - + [pu~2] Show commits in topo order and name all commits. - + [pu~3] mailinfo and applymbox updates - + [pu~4] Document "git cherry-pick" and "git revert" - + [pu~5] Remove git-apply-patch-script. - + [pu~6] Redo "revert" using three-way merge machinery. + + [seen] git-repack: Add option to repack all objects. + + [seen~1] More documentation updates. + + [seen~2] Show commits in topo order and name all commits. + + [seen~3] mailinfo and applymbox updates + + [seen~4] Document "git cherry-pick" and "git revert" + + [seen~5] Remove git-apply-patch-script. + + [seen~6] Redo "revert" using three-way merge machinery. - [rc] Merge refs/heads/master from . ++* [master] Revert "Replace zero-length array decls with []." - [rc~1] Merge refs/heads/master from . diff --git a/Documentation/howto/update-hook-example.txt b/Documentation/howto/update-hook-example.txt index 89821ec74f..151ee84ceb 100644 --- a/Documentation/howto/update-hook-example.txt +++ b/Documentation/howto/update-hook-example.txt @@ -179,7 +179,7 @@ allowed-groups, to describe which heads can be pushed into by whom. The format of each file would look like this: refs/heads/master junio - +refs/heads/pu junio + +refs/heads/seen junio refs/heads/cogito$ pasky refs/heads/bw/.* linus refs/heads/tmp/.* .* @@ -187,6 +187,6 @@ whom. The format of each file would look like this: With this, Linus can push or create "bw/penguin" or "bw/zebra" or "bw/panda" branches, Pasky can do only "cogito", and JC can -do master and pu branches and make versioned tags. And anybody -can do tmp/blah branches. The '+' sign at the pu record means +do master and "seen" branches and make versioned tags. And anybody +can do tmp/blah branches. The '+' sign at the "seen" record means that JC can make non-fast-forward pushes on it. diff --git a/Documentation/pretty-formats.txt b/Documentation/pretty-formats.txt index 547a552463..84bbc7439a 100644 --- a/Documentation/pretty-formats.txt +++ b/Documentation/pretty-formats.txt @@ -196,8 +196,8 @@ The placeholders are: '%ce':: committer email '%cE':: committer email (respecting .mailmap, see linkgit:git-shortlog[1] or linkgit:git-blame[1]) -'%cl':: author email local-part (the part before the '@' sign) -'%cL':: author local-part (see '%cl') respecting .mailmap, see +'%cl':: committer email local-part (the part before the '@' sign) +'%cL':: committer local-part (see '%cl') respecting .mailmap, see linkgit:git-shortlog[1] or linkgit:git-blame[1]) '%cd':: committer date (format respects --date= option) '%cD':: committer date, RFC2822 style diff --git a/Documentation/rev-list-description.txt b/Documentation/rev-list-description.txt new file mode 100644 index 0000000000..a9efa7fa27 --- /dev/null +++ b/Documentation/rev-list-description.txt @@ -0,0 +1,61 @@ +List commits that are reachable by following the `parent` links from the +given commit(s), but exclude commits that are reachable from the one(s) +given with a '{caret}' in front of them. The output is given in reverse +chronological order by default. + +You can think of this as a set operation. Commits reachable from any of +the commits given on the command line form a set, and then commits reachable +from any of the ones given with '{caret}' in front are subtracted from that +set. The remaining commits are what comes out in the command's output. +Various other options and paths parameters can be used to further limit the +result. + +Thus, the following command: + +ifdef::git-rev-list[] +----------------------------------------------------------------------- +$ git rev-list foo bar ^baz +----------------------------------------------------------------------- +endif::git-rev-list[] +ifdef::git-log[] +----------------------------------------------------------------------- +$ git log foo bar ^baz +----------------------------------------------------------------------- +endif::git-log[] + +means "list all the commits which are reachable from 'foo' or 'bar', but +not from 'baz'". + +A special notation "'<commit1>'..'<commit2>'" can be used as a +short-hand for "^'<commit1>' '<commit2>'". For example, either of +the following may be used interchangeably: + +ifdef::git-rev-list[] +----------------------------------------------------------------------- +$ git rev-list origin..HEAD +$ git rev-list HEAD ^origin +----------------------------------------------------------------------- +endif::git-rev-list[] +ifdef::git-log[] +----------------------------------------------------------------------- +$ git log origin..HEAD +$ git log HEAD ^origin +----------------------------------------------------------------------- +endif::git-log[] + +Another special notation is "'<commit1>'...'<commit2>'" which is useful +for merges. The resulting set of commits is the symmetric difference +between the two operands. The following two commands are equivalent: + +ifdef::git-rev-list[] +----------------------------------------------------------------------- +$ git rev-list A B --not $(git merge-base --all A B) +$ git rev-list A...B +----------------------------------------------------------------------- +endif::git-rev-list[] +ifdef::git-log[] +----------------------------------------------------------------------- +$ git log A B --not $(git merge-base --all A B) +$ git log A...B +----------------------------------------------------------------------- +endif::git-log[] diff --git a/Documentation/revisions.txt b/Documentation/revisions.txt index 1ad95065c1..d9169c062e 100644 --- a/Documentation/revisions.txt +++ b/Documentation/revisions.txt @@ -254,6 +254,9 @@ specifying a single revision, using the notation described in the previous section, means the set of commits `reachable` from the given commit. +Specifying several revisions means the set of commits reachable from +any of the given commits. + A commit's reachable set is the commit itself and the commits in its ancestry chain. diff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt index 1beef17182..440541045d 100644 --- a/Documentation/technical/commit-graph-format.txt +++ b/Documentation/technical/commit-graph-format.txt @@ -32,7 +32,7 @@ the body into "chunks" and provide a binary lookup table at the beginning of the body. The header includes certain values, such as number of chunks and hash type. -All 4-byte numbers are in network order. +All multi-byte numbers are in network byte order. HEADER: diff --git a/Documentation/technical/http-protocol.txt b/Documentation/technical/http-protocol.txt index 9c5b6f0fac..51a79e63de 100644 --- a/Documentation/technical/http-protocol.txt +++ b/Documentation/technical/http-protocol.txt @@ -216,7 +216,7 @@ smart server reply: S: 001e# service=git-upload-pack\n S: 0000 S: 004895dcfa3633004da0049d3d0fa03f80589cbcaf31 refs/heads/maint\0multi_ack\n - S: 0042d049f6c27a2244e12041955e262a404c7faba355 refs/heads/master\n + S: 003fd049f6c27a2244e12041955e262a404c7faba355 refs/heads/master\n S: 003c2cb58b79488a98d2721cea644875a8dd0026b115 refs/tags/v1.0\n S: 003fa3c2e2402b99163d1d59756e5f207ae21cccba4c refs/tags/v1.0^{}\n S: 0000 diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt index d5ce4eea8a..a4573d12ce 100644 --- a/Documentation/technical/pack-protocol.txt +++ b/Documentation/technical/pack-protocol.txt @@ -96,7 +96,7 @@ Basically what the Git client is doing to connect to an 'upload-pack' process on the server side over the Git protocol is this: $ echo -e -n \ - "0039git-upload-pack /schacon/gitbook.git\0host=example.com\0" | + "003agit-upload-pack /schacon/gitbook.git\0host=example.com\0" | nc -v example.com 9418 @@ -171,9 +171,9 @@ with a version number (if "version=1" is sent as an Extra Parameter), and a listing of each reference it has (all branches and tags) along with the object name that each reference currently points to. - $ echo -e -n "0044git-upload-pack /schacon/gitbook.git\0host=example.com\0\0version=1\0" | + $ echo -e -n "0045git-upload-pack /schacon/gitbook.git\0host=example.com\0\0version=1\0" | nc -v example.com 9418 - 000aversion 1 + 000eversion 1 00887217a7c7e582c46cec22a130adf4b9d7d950fba0 HEAD\0multi_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress include-tag 00441d3fcd5ced445d1abc402225c0b8a1299641f497 refs/heads/integration diff --git a/Documentation/technical/packfile-uri.txt b/Documentation/technical/packfile-uri.txt new file mode 100644 index 0000000000..318713abc3 --- /dev/null +++ b/Documentation/technical/packfile-uri.txt @@ -0,0 +1,78 @@ +Packfile URIs +============= + +This feature allows servers to serve part of their packfile response as URIs. +This allows server designs that improve scalability in bandwidth and CPU usage +(for example, by serving some data through a CDN), and (in the future) provides +some measure of resumability to clients. + +This feature is available only in protocol version 2. + +Protocol +-------- + +The server advertises the `packfile-uris` capability. + +If the client then communicates which protocols (HTTPS, etc.) it supports with +a `packfile-uris` argument, the server MAY send a `packfile-uris` section +directly before the `packfile` section (right after `wanted-refs` if it is +sent) containing URIs of any of the given protocols. The URIs point to +packfiles that use only features that the client has declared that it supports +(e.g. ofs-delta and thin-pack). See protocol-v2.txt for the documentation of +this section. + +Clients should then download and index all the given URIs (in addition to +downloading and indexing the packfile given in the `packfile` section of the +response) before performing the connectivity check. + +Server design +------------- + +The server can be trivially made compatible with the proposed protocol by +having it advertise `packfile-uris`, tolerating the client sending +`packfile-uris`, and never sending any `packfile-uris` section. But we should +include some sort of non-trivial implementation in the Minimum Viable Product, +at least so that we can test the client. + +This is the implementation: a feature, marked experimental, that allows the +server to be configured by one or more `uploadpack.blobPackfileUri=<sha1> +<uri>` entries. Whenever the list of objects to be sent is assembled, all such +blobs are excluded, replaced with URIs. The client will download those URIs, +expecting them to each point to packfiles containing single blobs. + +Client design +------------- + +The client has a config variable `fetch.uriprotocols` that determines which +protocols the end user is willing to use. By default, this is empty. + +When the client downloads the given URIs, it should store them with "keep" +files, just like it does with the packfile in the `packfile` section. These +additional "keep" files can only be removed after the refs have been updated - +just like the "keep" file for the packfile in the `packfile` section. + +The division of work (initial fetch + additional URIs) introduces convenient +points for resumption of an interrupted clone - such resumption can be done +after the Minimum Viable Product (see "Future work"). + +Future work +----------- + +The protocol design allows some evolution of the server and client without any +need for protocol changes, so only a small-scoped design is included here to +form the MVP. For example, the following can be done: + + * On the server, more sophisticated means of excluding objects (e.g. by + specifying a commit to represent that commit and all objects that it + references). + * On the client, resumption of clone. If a clone is interrupted, information + could be recorded in the repository's config and a "clone-resume" command + can resume the clone in progress. (Resumption of subsequent fetches is more + difficult because that must deal with the user wanting to use the repository + even after the fetch was interrupted.) + +There are some possible features that will require a change in protocol: + + * Additional HTTP headers (e.g. authentication) + * Byte range support + * Different file formats referenced by URIs (e.g. raw object) diff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt index 2b267c0da6..36ccd14f97 100644 --- a/Documentation/technical/protocol-capabilities.txt +++ b/Documentation/technical/protocol-capabilities.txt @@ -176,6 +176,21 @@ agent strings are purely informative for statistics and debugging purposes, and MUST NOT be used to programmatically assume the presence or absence of particular features. +object-format +------------- + +This capability, which takes a hash algorithm as an argument, indicates +that the server supports the given hash algorithms. It may be sent +multiple times; if so, the first one given is the one used in the ref +advertisement. + +When provided by the client, this indicates that it intends to use the +given hash algorithm to communicate. The algorithm provided must be one +that the server supports. + +If this capability is not provided, it is assumed that the only +supported algorithm is SHA-1. + symref ------ diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt index 7e3766cafb..e597b74da3 100644 --- a/Documentation/technical/protocol-v2.txt +++ b/Documentation/technical/protocol-v2.txt @@ -33,6 +33,8 @@ In protocol v2 these special packets will have the following semantics: * '0000' Flush Packet (flush-pkt) - indicates the end of a message * '0001' Delimiter Packet (delim-pkt) - separates sections of a message + * '0002' Message Packet (response-end-pkt) - indicates the end of a response + for stateless connections Initial Client Request ---------------------- @@ -323,13 +325,26 @@ included in the client's request: indicating its sideband (1, 2, or 3), and the server may send "0005\2" (a PKT-LINE of sideband 2 with no payload) as a keepalive packet. +If the 'packfile-uris' feature is advertised, the following argument +can be included in the client's request as well as the potential +addition of the 'packfile-uris' section in the server's response as +explained below. + + packfile-uris <comma-separated list of protocols> + Indicates to the server that the client is willing to receive + URIs of any of the given protocols in place of objects in the + sent packfile. Before performing the connectivity check, the + client should download from all given URIs. Currently, the + protocols supported are "http" and "https". + The response of `fetch` is broken into a number of sections separated by delimiter packets (0001), with each section beginning with its section -header. +header. Most sections are sent only when the packfile is sent. - output = *section - section = (acknowledgments | shallow-info | wanted-refs | packfile) - (flush-pkt | delim-pkt) + output = acknowledgements flush-pkt | + [acknowledgments delim-pkt] [shallow-info delim-pkt] + [wanted-refs delim-pkt] [packfile-uris delim-pkt] + packfile flush-pkt acknowledgments = PKT-LINE("acknowledgments" LF) (nak | *ack) @@ -347,13 +362,17 @@ header. *PKT-LINE(wanted-ref LF) wanted-ref = obj-id SP refname + packfile-uris = PKT-LINE("packfile-uris" LF) *packfile-uri + packfile-uri = PKT-LINE(40*(HEXDIGIT) SP *%x20-ff LF) + packfile = PKT-LINE("packfile" LF) *PKT-LINE(%x01-03 *%x00-ff) acknowledgments section - * If the client determines that it is finished with negotiations - by sending a "done" line, the acknowledgments sections MUST be - omitted from the server's response. + * If the client determines that it is finished with negotiations by + sending a "done" line (thus requiring the server to send a packfile), + the acknowledgments sections MUST be omitted from the server's + response. * Always begins with the section header "acknowledgments" @@ -404,9 +423,6 @@ header. which the client has not indicated was shallow as a part of its request. - * This section is only included if a packfile section is also - included in the response. - wanted-refs section * This section is only included if the client has requested a ref using a 'want-ref' line and if a packfile section is also @@ -420,6 +436,20 @@ header. * The server MUST NOT send any refs which were not requested using 'want-ref' lines. + packfile-uris section + * This section is only included if the client sent + 'packfile-uris' and the server has at least one such URI to + send. + + * Always begins with the section header "packfile-uris". + + * For each URI the server sends, it sends a hash of the pack's + contents (as output by git index-pack) followed by the URI. + + * The hashes are 40 hex characters long. When Git upgrades to a new + hash algorithm, this might need to be updated. (It should match + whatever index-pack outputs after "pack\t" or "keep\t". + packfile section * This section is only included if the client has sent 'want' lines in its request and either requested that no more @@ -453,3 +483,12 @@ included in a request. This is done by sending each option as a a request. The provided options must not contain a NUL or LF character. + + object-format +~~~~~~~~~~~~~~~ + +The server can advertise the `object-format` capability with a value `X` (in the +form `object-format=X`) to notify the client that the server is able to deal +with objects using hash algorithm X. If not specified, the server is assumed to +only handle SHA-1. If the client would like to use a hash algorithm other than +SHA-1, it should specify its object-format string. diff --git a/Documentation/technical/reftable.txt b/Documentation/technical/reftable.txt new file mode 100644 index 0000000000..2951840e9c --- /dev/null +++ b/Documentation/technical/reftable.txt @@ -0,0 +1,1083 @@ +reftable +-------- + +Overview +~~~~~~~~ + +Problem statement +^^^^^^^^^^^^^^^^^ + +Some repositories contain a lot of references (e.g. android at 866k, +rails at 31k). The existing packed-refs format takes up a lot of space +(e.g. 62M), and does not scale with additional references. Lookup of a +single reference requires linearly scanning the file. + +Atomic pushes modifying multiple references require copying the entire +packed-refs file, which can be a considerable amount of data moved +(e.g. 62M in, 62M out) for even small transactions (2 refs modified). + +Repositories with many loose references occupy a large number of disk +blocks from the local file system, as each reference is its own file +storing 41 bytes (and another file for the corresponding reflog). This +negatively affects the number of inodes available when a large number of +repositories are stored on the same filesystem. Readers can be penalized +due to the larger number of syscalls required to traverse and read the +`$GIT_DIR/refs` directory. + + +Objectives +^^^^^^^^^^ + +* Near constant time lookup for any single reference, even when the +repository is cold and not in process or kernel cache. +* Near constant time verification if an object name is referred to by at least +one reference (for allow-tip-sha1-in-want). +* Efficient enumeration of an entire namespace, such as `refs/tags/`. +* Support atomic push with `O(size_of_update)` operations. +* Combine reflog storage with ref storage for small transactions. +* Separate reflog storage for base refs and historical logs. + +Description +^^^^^^^^^^^ + +A reftable file is a portable binary file format customized for +reference storage. References are sorted, enabling linear scans, binary +search lookup, and range scans. + +Storage in the file is organized into variable sized blocks. Prefix +compression is used within a single block to reduce disk space. Block +size and alignment is tunable by the writer. + +Performance +^^^^^^^^^^^ + +Space used, packed-refs vs. reftable: + +[cols=",>,>,>,>,>",options="header",] +|=============================================================== +|repository |packed-refs |reftable |% original |avg ref |avg obj +|android |62.2 M |36.1 M |58.0% |33 bytes |5 bytes +|rails |1.8 M |1.1 M |57.7% |29 bytes |4 bytes +|git |78.7 K |48.1 K |61.0% |50 bytes |4 bytes +|git (heads) |332 b |269 b |81.0% |33 bytes |0 bytes +|=============================================================== + +Scan (read 866k refs), by reference name lookup (single ref from 866k +refs), and by SHA-1 lookup (refs with that SHA-1, from 866k refs): + +[cols=",>,>,>,>",options="header",] +|========================================================= +|format |cache |scan |by name |by SHA-1 +|packed-refs |cold |402 ms |409,660.1 usec |412,535.8 usec +|packed-refs |hot | |6,844.6 usec |20,110.1 usec +|reftable |cold |112 ms |33.9 usec |323.2 usec +|reftable |hot | |20.2 usec |320.8 usec +|========================================================= + +Space used for 149,932 log entries for 43,061 refs, reflog vs. reftable: + +[cols=",>,>",options="header",] +|================================ +|format |size |avg entry +|$GIT_DIR/logs |173 M |1209 bytes +|reftable |5 M |37 bytes +|================================ + +Details +~~~~~~~ + +Peeling +^^^^^^^ + +References stored in a reftable are peeled, a record for an annotated +(or signed) tag records both the tag object, and the object it refers +to. This is analogous to storage in the packed-refs format. + +Reference name encoding +^^^^^^^^^^^^^^^^^^^^^^^ + +Reference names are an uninterpreted sequence of bytes that must pass +linkgit:git-check-ref-format[1] as a valid reference name. + +Key unicity +^^^^^^^^^^^ + +Each entry must have a unique key; repeated keys are disallowed. + +Network byte order +^^^^^^^^^^^^^^^^^^ + +All multi-byte, fixed width fields are in network byte order. + +Varint encoding +^^^^^^^^^^^^^^^ + +Varint encoding is identical to the ofs-delta encoding method used +within pack files. + +Decoder works such as: + +.... +val = buf[ptr] & 0x7f +while (buf[ptr] & 0x80) { + ptr++ + val = ((val + 1) << 7) | (buf[ptr] & 0x7f) +} +.... + +Ordering +^^^^^^^^ + +Blocks are lexicographically ordered by their first reference. + +Directory/file conflicts +^^^^^^^^^^^^^^^^^^^^^^^^ + +The reftable format accepts both `refs/heads/foo` and +`refs/heads/foo/bar` as distinct references. + +This property is useful for retaining log records in reftable, but may +confuse versions of Git using `$GIT_DIR/refs` directory tree to maintain +references. Users of reftable may choose to continue to reject `foo` and +`foo/bar` type conflicts to prevent problems for peers. + +File format +~~~~~~~~~~~ + +Structure +^^^^^^^^^ + +A reftable file has the following high-level structure: + +.... +first_block { + header + first_ref_block +} +ref_block* +ref_index* +obj_block* +obj_index* +log_block* +log_index* +footer +.... + +A log-only file omits the `ref_block`, `ref_index`, `obj_block` and +`obj_index` sections, containing only the file header and log block: + +.... +first_block { + header +} +log_block* +log_index* +footer +.... + +in a log-only file the first log block immediately follows the file +header, without padding to block alignment. + +Block size +^^^^^^^^^^ + +The file's block size is arbitrarily determined by the writer, and does +not have to be a power of 2. The block size must be larger than the +longest reference name or log entry used in the repository, as +references cannot span blocks. + +Powers of two that are friendly to the virtual memory system or +filesystem (such as 4k or 8k) are recommended. Larger sizes (64k) can +yield better compression, with a possible increased cost incurred by +readers during access. + +The largest block size is `16777215` bytes (15.99 MiB). + +Block alignment +^^^^^^^^^^^^^^^ + +Writers may choose to align blocks at multiples of the block size by +including `padding` filled with NUL bytes at the end of a block to round +out to the chosen alignment. When alignment is used, writers must +specify the alignment with the file header's `block_size` field. + +Block alignment is not required by the file format. Unaligned files must +set `block_size = 0` in the file header, and omit `padding`. Unaligned +files with more than one ref block must include the link:#Ref-index[ref +index] to support fast lookup. Readers must be able to read both aligned +and non-aligned files. + +Very small files (e.g. a single ref block) may omit `padding` and the ref +index to reduce total file size. + +Header (version 1) +^^^^^^^^^^^^^^^^^^ + +A 24-byte header appears at the beginning of the file: + +.... +'REFT' +uint8( version_number = 1 ) +uint24( block_size ) +uint64( min_update_index ) +uint64( max_update_index ) +.... + +Aligned files must specify `block_size` to configure readers with the +expected block alignment. Unaligned files must set `block_size = 0`. + +The `min_update_index` and `max_update_index` describe bounds for the +`update_index` field of all log records in this file. When reftables are +used in a stack for link:#Update-transactions[transactions], these +fields can order the files such that the prior file's +`max_update_index + 1` is the next file's `min_update_index`. + +Header (version 2) +^^^^^^^^^^^^^^^^^^ + +A 28-byte header appears at the beginning of the file: + +.... +'REFT' +uint8( version_number = 2 ) +uint24( block_size ) +uint64( min_update_index ) +uint64( max_update_index ) +uint32( hash_id ) +.... + +The header is identical to `version_number=1`, with the 4-byte hash ID +("sha1" for SHA1 and "s256" for SHA-256) append to the header. + +For maximum backward compatibility, it is recommended to use version 1 when +writing SHA1 reftables. + +First ref block +^^^^^^^^^^^^^^^ + +The first ref block shares the same block as the file header, and is 24 +bytes smaller than all other blocks in the file. The first block +immediately begins after the file header, at position 24. + +If the first block is a log block (a log-only file), its block header +begins immediately at position 24. + +Ref block format +^^^^^^^^^^^^^^^^ + +A ref block is written as: + +.... +'r' +uint24( block_len ) +ref_record+ +uint24( restart_offset )+ +uint16( restart_count ) + +padding? +.... + +Blocks begin with `block_type = 'r'` and a 3-byte `block_len` which +encodes the number of bytes in the block up to, but not including the +optional `padding`. This is always less than or equal to the file's +block size. In the first ref block, `block_len` includes 24 bytes for +the file header. + +The 2-byte `restart_count` stores the number of entries in the +`restart_offset` list, which must not be empty. Readers can use +`restart_count` to binary search between restarts before starting a +linear scan. + +Exactly `restart_count` 3-byte `restart_offset` values precedes the +`restart_count`. Offsets are relative to the start of the block and +refer to the first byte of any `ref_record` whose name has not been +prefix compressed. Entries in the `restart_offset` list must be sorted, +ascending. Readers can start linear scans from any of these records. + +A variable number of `ref_record` fill the middle of the block, +describing reference names and values. The format is described below. + +As the first ref block shares the first file block with the file header, +all `restart_offset` in the first block are relative to the start of the +file (position 0), and include the file header. This forces the first +`restart_offset` to be `28`. + +ref record +++++++++++ + +A `ref_record` describes a single reference, storing both the name and +its value(s). Records are formatted as: + +.... +varint( prefix_length ) +varint( (suffix_length << 3) | value_type ) +suffix +varint( update_index_delta ) +value? +.... + +The `prefix_length` field specifies how many leading bytes of the prior +reference record's name should be copied to obtain this reference's +name. This must be 0 for the first reference in any block, and also must +be 0 for any `ref_record` whose offset is listed in the `restart_offset` +table at the end of the block. + +Recovering a reference name from any `ref_record` is a simple concat: + +.... +this_name = prior_name[0..prefix_length] + suffix +.... + +The `suffix_length` value provides the number of bytes available in +`suffix` to copy from `suffix` to complete the reference name. + +The `update_index` that last modified the reference can be obtained by +adding `update_index_delta` to the `min_update_index` from the file +header: `min_update_index + update_index_delta`. + +The `value` follows. Its format is determined by `value_type`, one of +the following: + +* `0x0`: deletion; no value data (see transactions, below) +* `0x1`: one object name; value of the ref +* `0x2`: two object names; value of the ref, peeled target +* `0x3`: symbolic reference: `varint( target_len ) target` + +Symbolic references use `0x3`, followed by the complete name of the +reference target. No compression is applied to the target name. + +Types `0x4..0x7` are reserved for future use. + +Ref index +^^^^^^^^^ + +The ref index stores the name of the last reference from every ref block +in the file, enabling reduced disk seeks for lookups. Any reference can +be found by searching the index, identifying the containing block, and +searching within that block. + +The index may be organized into a multi-level index, where the 1st level +index block points to additional ref index blocks (2nd level), which may +in turn point to either additional index blocks (e.g. 3rd level) or ref +blocks (leaf level). Disk reads required to access a ref go up with +higher index levels. Multi-level indexes may be required to ensure no +single index block exceeds the file format's max block size of +`16777215` bytes (15.99 MiB). To achieve constant O(1) disk seeks for +lookups the index must be a single level, which is permitted to exceed +the file's configured block size, but not the format's max block size of +15.99 MiB. + +If present, the ref index block(s) appears after the last ref block. + +If there are at least 4 ref blocks, a ref index block should be written +to improve lookup times. Cold reads using the index require 2 disk reads +(read index, read block), and binary searching < 4 blocks also requires +<= 2 reads. Omitting the index block from smaller files saves space. + +If the file is unaligned and contains more than one ref block, the ref +index must be written. + +Index block format: + +.... +'i' +uint24( block_len ) +index_record+ +uint24( restart_offset )+ +uint16( restart_count ) + +padding? +.... + +The index blocks begin with `block_type = 'i'` and a 3-byte `block_len` +which encodes the number of bytes in the block, up to but not including +the optional `padding`. + +The `restart_offset` and `restart_count` fields are identical in format, +meaning and usage as in ref blocks. + +To reduce the number of reads required for random access in very large +files the index block may be larger than other blocks. However, readers +must hold the entire index in memory to benefit from this, so it's a +time-space tradeoff in both file size and reader memory. + +Increasing the file's block size decreases the index size. Alternatively +a multi-level index may be used, keeping index blocks within the file's +block size, but increasing the number of blocks that need to be +accessed. + +index record +++++++++++++ + +An index record describes the last entry in another block. Index records +are written as: + +.... +varint( prefix_length ) +varint( (suffix_length << 3) | 0 ) +suffix +varint( block_position ) +.... + +Index records use prefix compression exactly like `ref_record`. + +Index records store `block_position` after the suffix, specifying the +absolute position in bytes (from the start of the file) of the block +that ends with this reference. Readers can seek to `block_position` to +begin reading the block header. + +Readers must examine the block header at `block_position` to determine +if the next block is another level index block, or the leaf-level ref +block. + +Reading the index ++++++++++++++++++ + +Readers loading the ref index must first read the footer (below) to +obtain `ref_index_position`. If not present, the position will be 0. The +`ref_index_position` is for the 1st level root of the ref index. + +Obj block format +^^^^^^^^^^^^^^^^ + +Object blocks are optional. Writers may choose to omit object blocks, +especially if readers will not use the object name to ref mapping. + +Object blocks use unique, abbreviated 2-32 object name keys, mapping to +ref blocks containing references pointing to that object directly, or as +the peeled value of an annotated tag. Like ref blocks, object blocks use +the file's standard block size. The abbrevation length is available in +the footer as `obj_id_len`. + +To save space in small files, object blocks may be omitted if the ref +index is not present, as brute force search will only need to read a few +ref blocks. When missing, readers should brute force a linear search of +all references to lookup by object name. + +An object block is written as: + +.... +'o' +uint24( block_len ) +obj_record+ +uint24( restart_offset )+ +uint16( restart_count ) + +padding? +.... + +Fields are identical to ref block. Binary search using the restart table +works the same as in reference blocks. + +Because object names are abbreviated by writers to the shortest unique +abbreviation within the reftable, obj key lengths have a variable length. Their +length must be at least 2 bytes. Readers must compare only for common prefix +match within an obj block or obj index. + +obj record +++++++++++ + +An `obj_record` describes a single object abbreviation, and the blocks +containing references using that unique abbreviation: + +.... +varint( prefix_length ) +varint( (suffix_length << 3) | cnt_3 ) +suffix +varint( cnt_large )? +varint( position_delta )* +.... + +Like in reference blocks, abbreviations are prefix compressed within an +obj block. On large reftables with many unique objects, higher block +sizes (64k), and higher restart interval (128), a `prefix_length` of 2 +or 3 and `suffix_length` of 3 may be common in obj records (unique +abbreviation of 5-6 raw bytes, 10-12 hex digits). + +Each record contains `position_count` number of positions for matching +ref blocks. For 1-7 positions the count is stored in `cnt_3`. When +`cnt_3 = 0` the actual count follows in a varint, `cnt_large`. + +The use of `cnt_3` bets most objects are pointed to by only a single +reference, some may be pointed to by a couple of references, and very +few (if any) are pointed to by more than 7 references. + +A special case exists when `cnt_3 = 0` and `cnt_large = 0`: there are no +`position_delta`, but at least one reference starts with this +abbreviation. A reader that needs exact reference names must scan all +references to find which specific references have the desired object. +Writers should use this format when the `position_delta` list would have +overflowed the file's block size due to a high number of references +pointing to the same object. + +The first `position_delta` is the position from the start of the file. +Additional `position_delta` entries are sorted ascending and relative to +the prior entry, e.g. a reader would perform: + +.... +pos = position_delta[0] +prior = pos +for (j = 1; j < position_count; j++) { + pos = prior + position_delta[j] + prior = pos +} +.... + +With a position in hand, a reader must linearly scan the ref block, +starting from the first `ref_record`, testing each reference's object names +(for `value_type = 0x1` or `0x2`) for full equality. Faster searching by +object name within a single ref block is not supported by the reftable format. +Smaller block sizes reduce the number of candidates this step must +consider. + +Obj index +^^^^^^^^^ + +The obj index stores the abbreviation from the last entry for every obj +block in the file, enabling reduced disk seeks for all lookups. It is +formatted exactly the same as the ref index, but refers to obj blocks. + +The obj index should be present if obj blocks are present, as obj blocks +should only be written in larger files. + +Readers loading the obj index must first read the footer (below) to +obtain `obj_index_position`. If not present, the position will be 0. + +Log block format +^^^^^^^^^^^^^^^^ + +Unlike ref and obj blocks, log blocks are always unaligned. + +Log blocks are variable in size, and do not match the `block_size` +specified in the file header or footer. Writers should choose an +appropriate buffer size to prepare a log block for deflation, such as +`2 * block_size`. + +A log block is written as: + +.... +'g' +uint24( block_len ) +zlib_deflate { + log_record+ + uint24( restart_offset )+ + uint16( restart_count ) +} +.... + +Log blocks look similar to ref blocks, except `block_type = 'g'`. + +The 4-byte block header is followed by the deflated block contents using +zlib deflate. The `block_len` in the header is the inflated size +(including 4-byte block header), and should be used by readers to +preallocate the inflation output buffer. A log block's `block_len` may +exceed the file's block size. + +Offsets within the log block (e.g. `restart_offset`) still include the +4-byte header. Readers may prefer prefixing the inflation output buffer +with the 4-byte header. + +Within the deflate container, a variable number of `log_record` describe +reference changes. The log record format is described below. See ref +block format (above) for a description of `restart_offset` and +`restart_count`. + +Because log blocks have no alignment or padding between blocks, readers +must keep track of the bytes consumed by the inflater to know where the +next log block begins. + +log record +++++++++++ + +Log record keys are structured as: + +.... +ref_name '\0' reverse_int64( update_index ) +.... + +where `update_index` is the unique transaction identifier. The +`update_index` field must be unique within the scope of a `ref_name`. +See the update transactions section below for further details. + +The `reverse_int64` function inverses the value so lexicographical +ordering the network byte order encoding sorts the more recent records +with higher `update_index` values first: + +.... +reverse_int64(int64 t) { + return 0xffffffffffffffff - t; +} +.... + +Log records have a similar starting structure to ref and index records, +utilizing the same prefix compression scheme applied to the log record +key described above. + +.... + varint( prefix_length ) + varint( (suffix_length << 3) | log_type ) + suffix + log_data { + old_id + new_id + varint( name_length ) name + varint( email_length ) email + varint( time_seconds ) + sint16( tz_offset ) + varint( message_length ) message + }? +.... + +Log record entries use `log_type` to indicate what follows: + +* `0x0`: deletion; no log data. +* `0x1`: standard git reflog data using `log_data` above. + +The `log_type = 0x0` is mostly useful for `git stash drop`, removing an +entry from the reflog of `refs/stash` in a transaction file (below), +without needing to rewrite larger files. Readers reading a stack of +reflogs must treat this as a deletion. + +For `log_type = 0x1`, the `log_data` section follows +linkgit:git-update-ref[1] logging and includes: + +* two object names (old id, new id) +* varint string of committer's name +* varint string of committer's email +* varint time in seconds since epoch (Jan 1, 1970) +* 2-byte timezone offset in minutes (signed) +* varint string of message + +`tz_offset` is the absolute number of minutes from GMT the committer was +at the time of the update. For example `GMT-0800` is encoded in reftable +as `sint16(-480)` and `GMT+0230` is `sint16(150)`. + +The committer email does not contain `<` or `>`, it's the value normally +found between the `<>` in a git commit object header. + +The `message_length` may be 0, in which case there was no message +supplied for the update. + +Contrary to traditional reflog (which is a file), renames are encoded as +a combination of ref deletion and ref creation. A deletion is a log +record with a zero new_id, and a creation is a log record with a zero old_id. + +Reading the log ++++++++++++++++ + +Readers accessing the log must first read the footer (below) to +determine the `log_position`. The first block of the log begins at +`log_position` bytes since the start of the file. The `log_position` is +not block aligned. + +Importing logs +++++++++++++++ + +When importing from `$GIT_DIR/logs` writers should globally order all +log records roughly by timestamp while preserving file order, and assign +unique, increasing `update_index` values for each log line. Newer log +records get higher `update_index` values. + +Although an import may write only a single reftable file, the reftable +file must span many unique `update_index`, as each log line requires its +own `update_index` to preserve semantics. + +Log index +^^^^^^^^^ + +The log index stores the log key +(`refname \0 reverse_int64(update_index)`) for the last log record of +every log block in the file, supporting bounded-time lookup. + +A log index block must be written if 2 or more log blocks are written to +the file. If present, the log index appears after the last log block. +There is no padding used to align the log index to block alignment. + +Log index format is identical to ref index, except the keys are 9 bytes +longer to include `'\0'` and the 8-byte `reverse_int64(update_index)`. +Records use `block_position` to refer to the start of a log block. + +Reading the index ++++++++++++++++++ + +Readers loading the log index must first read the footer (below) to +obtain `log_index_position`. If not present, the position will be 0. + +Footer +^^^^^^ + +After the last block of the file, a file footer is written. It begins +like the file header, but is extended with additional data. + +.... + HEADER + + uint64( ref_index_position ) + uint64( (obj_position << 5) | obj_id_len ) + uint64( obj_index_position ) + + uint64( log_position ) + uint64( log_index_position ) + + uint32( CRC-32 of above ) +.... + +If a section is missing (e.g. ref index) the corresponding position +field (e.g. `ref_index_position`) will be 0. + +* `obj_position`: byte position for the first obj block. +* `obj_id_len`: number of bytes used to abbreviate object names in +obj blocks. +* `log_position`: byte position for the first log block. +* `ref_index_position`: byte position for the start of the ref index. +* `obj_index_position`: byte position for the start of the obj index. +* `log_index_position`: byte position for the start of the log index. + +The size of the footer is 68 bytes for version 1, and 72 bytes for +version 2. + +Reading the footer +++++++++++++++++++ + +Readers must first read the file start to determine the version +number. Then they seek to `file_length - FOOTER_LENGTH` to access the +footer. A trusted external source (such as `stat(2)`) is necessary to +obtain `file_length`. When reading the footer, readers must verify: + +* 4-byte magic is correct +* 1-byte version number is recognized +* 4-byte CRC-32 matches the other 64 bytes (including magic, and +version) + +Once verified, the other fields of the footer can be accessed. + +Empty tables +++++++++++++ + +A reftable may be empty. In this case, the file starts with a header +and is immediately followed by a footer. + +Binary search +^^^^^^^^^^^^^ + +Binary search within a block is supported by the `restart_offset` fields +at the end of the block. Readers can binary search through the restart +table to locate between which two restart points the sought reference or +key should appear. + +Each record identified by a `restart_offset` stores the complete key in +the `suffix` field of the record, making the compare operation during +binary search straightforward. + +Once a restart point lexicographically before the sought reference has +been identified, readers can linearly scan through the following record +entries to locate the sought record, terminating if the current record +sorts after (and therefore the sought key is not present). + +Restart point selection ++++++++++++++++++++++++ + +Writers determine the restart points at file creation. The process is +arbitrary, but every 16 or 64 records is recommended. Every 16 may be +more suitable for smaller block sizes (4k or 8k), every 64 for larger +block sizes (64k). + +More frequent restart points reduces prefix compression and increases +space consumed by the restart table, both of which increase file size. + +Less frequent restart points makes prefix compression more effective, +decreasing overall file size, with increased penalties for readers +walking through more records after the binary search step. + +A maximum of `65535` restart points per block is supported. + +Considerations +~~~~~~~~~~~~~~ + +Lightweight refs dominate +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The reftable format assumes the vast majority of references are single +object names valued with common prefixes, such as Gerrit Code Review's +`refs/changes/` namespace, GitHub's `refs/pulls/` namespace, or many +lightweight tags in the `refs/tags/` namespace. + +Annotated tags storing the peeled object cost an additional object name per +reference. + +Low overhead +^^^^^^^^^^^^ + +A reftable with very few references (e.g. git.git with 5 heads) is 269 +bytes for reftable, vs. 332 bytes for packed-refs. This supports +reftable scaling down for transaction logs (below). + +Block size +^^^^^^^^^^ + +For a Gerrit Code Review type repository with many change refs, larger +block sizes (64 KiB) and less frequent restart points (every 64) yield +better compression due to more references within the block compressing +against the prior reference. + +Larger block sizes reduce the index size, as the reftable will require +fewer blocks to store the same number of references. + +Minimal disk seeks +^^^^^^^^^^^^^^^^^^ + +Assuming the index block has been loaded into memory, binary searching +for any single reference requires exactly 1 disk seek to load the +containing block. + +Scans and lookups dominate +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Scanning all references and lookup by name (or namespace such as +`refs/heads/`) are the most common activities performed on repositories. +Object names are stored directly with references to optimize this use case. + +Logs are infrequently read +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Logs are infrequently accessed, but can be large. Deflating log blocks +saves disk space, with some increased penalty at read time. + +Logs are stored in an isolated section from refs, reducing the burden on +reference readers that want to ignore logs. Further, historical logs can +be isolated into log-only files. + +Logs are read backwards +^^^^^^^^^^^^^^^^^^^^^^^ + +Logs are frequently accessed backwards (most recent N records for master +to answer `master@{4}`), so log records are grouped by reference, and +sorted descending by update index. + +Repository format +~~~~~~~~~~~~~~~~~ + +Version 1 +^^^^^^^^^ + +A repository must set its `$GIT_DIR/config` to configure reftable: + +.... +[core] + repositoryformatversion = 1 +[extensions] + refStorage = reftable +.... + +Layout +^^^^^^ + +A collection of reftable files are stored in the `$GIT_DIR/reftable/` +directory: + +.... +00000001-00000001.log +00000002-00000002.ref +00000003-00000003.ref +.... + +where reftable files are named by a unique name such as produced by the +function `${min_update_index}-${max_update_index}.ref`. + +Log-only files use the `.log` extension, while ref-only and mixed ref +and log files use `.ref`. extension. + +The stack ordering file is `$GIT_DIR/reftable/tables.list` and lists the +current files, one per line, in order, from oldest (base) to newest +(most recent): + +.... +$ cat .git/reftable/tables.list +00000001-00000001.log +00000002-00000002.ref +00000003-00000003.ref +.... + +Readers must read `$GIT_DIR/reftable/tables.list` to determine which +files are relevant right now, and search through the stack in reverse +order (last reftable is examined first). + +Reftable files not listed in `tables.list` may be new (and about to be +added to the stack by the active writer), or ancient and ready to be +pruned. + +Backward compatibility +^^^^^^^^^^^^^^^^^^^^^^ + +Older clients should continue to recognize the directory as a git +repository so they don't look for an enclosing repository in parent +directories. To this end, a reftable-enabled repository must contain the +following dummy files + +* `.git/HEAD`, a regular file containing `ref: refs/heads/.invalid`. +* `.git/refs/`, a directory +* `.git/refs/heads`, a regular file + +Readers +^^^^^^^ + +Readers can obtain a consistent snapshot of the reference space by +following: + +1. Open and read the `tables.list` file. +2. Open each of the reftable files that it mentions. +3. If any of the files is missing, goto 1. +4. Read from the now-open files as long as necessary. + +Update transactions +^^^^^^^^^^^^^^^^^^^ + +Although reftables are immutable, mutations are supported by writing a +new reftable and atomically appending it to the stack: + +1. Acquire `tables.list.lock`. +2. Read `tables.list` to determine current reftables. +3. Select `update_index` to be most recent file's +`max_update_index + 1`. +4. Prepare temp reftable `tmp_XXXXXX`, including log entries. +5. Rename `tmp_XXXXXX` to `${update_index}-${update_index}.ref`. +6. Copy `tables.list` to `tables.list.lock`, appending file from (5). +7. Rename `tables.list.lock` to `tables.list`. + +During step 4 the new file's `min_update_index` and `max_update_index` +are both set to the `update_index` selected by step 3. All log records +for the transaction use the same `update_index` in their keys. This +enables later correlation of which references were updated by the same +transaction. + +Because a single `tables.list.lock` file is used to manage locking, the +repository is single-threaded for writers. Writers may have to busy-spin +(with backoff) around creating `tables.list.lock`, for up to an +acceptable wait period, aborting if the repository is too busy to +mutate. Application servers wrapped around repositories (e.g. Gerrit +Code Review) can layer their own lock/wait queue to improve fairness to +writers. + +Reference deletions +^^^^^^^^^^^^^^^^^^^ + +Deletion of any reference can be explicitly stored by setting the `type` +to `0x0` and omitting the `value` field of the `ref_record`. This serves +as a tombstone, overriding any assertions about the existence of the +reference from earlier files in the stack. + +Compaction +^^^^^^^^^^ + +A partial stack of reftables can be compacted by merging references +using a straightforward merge join across reftables, selecting the most +recent value for output, and omitting deleted references that do not +appear in remaining, lower reftables. + +A compacted reftable should set its `min_update_index` to the smallest +of the input files' `min_update_index`, and its `max_update_index` +likewise to the largest input `max_update_index`. + +For sake of illustration, assume the stack currently consists of +reftable files (from oldest to newest): A, B, C, and D. The compactor is +going to compact B and C, leaving A and D alone. + +1. Obtain lock `tables.list.lock` and read the `tables.list` file. +2. Obtain locks `B.lock` and `C.lock`. Ownership of these locks +prevents other processes from trying to compact these files. +3. Release `tables.list.lock`. +4. Compact `B` and `C` into a temp file +`${min_update_index}-${max_update_index}_XXXXXX`. +5. Reacquire lock `tables.list.lock`. +6. Verify that `B` and `C` are still in the stack, in that order. This +should always be the case, assuming that other processes are adhering to +the locking protocol. +7. Rename `${min_update_index}-${max_update_index}_XXXXXX` to +`${min_update_index}-${max_update_index}.ref`. +8. Write the new stack to `tables.list.lock`, replacing `B` and `C` +with the file from (4). +9. Rename `tables.list.lock` to `tables.list`. +10. Delete `B` and `C`, perhaps after a short sleep to avoid forcing +readers to backtrack. + +This strategy permits compactions to proceed independently of updates. + +Each reftable (compacted or not) is uniquely identified by its name, so +open reftables can be cached by their name. + +Alternatives considered +~~~~~~~~~~~~~~~~~~~~~~~ + +bzip packed-refs +^^^^^^^^^^^^^^^^ + +`bzip2` can significantly shrink a large packed-refs file (e.g. 62 MiB +compresses to 23 MiB, 37%). However the bzip format does not support +random access to a single reference. Readers must inflate and discard +while performing a linear scan. + +Breaking packed-refs into chunks (individually compressing each chunk) +would reduce the amount of data a reader must inflate, but still leaves +the problem of indexing chunks to support readers efficiently locating +the correct chunk. + +Given the compression achieved by reftable's encoding, it does not seem +necessary to add the complexity of bzip/gzip/zlib. + +Michael Haggerty's alternate format +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Michael Haggerty proposed +link:https://lore.kernel.org/git/CAMy9T_HCnyc1g8XWOOWhe7nN0aEFyyBskV2aOMb_fe%2BwGvEJ7A%40mail.gmail.com/[an +alternate] format to reftable on the Git mailing list. This format uses +smaller chunks, without the restart table, and avoids block alignment +with padding. Reflog entries immediately follow each ref, and are thus +interleaved between refs. + +Performance testing indicates reftable is faster for lookups (51% +faster, 11.2 usec vs. 5.4 usec), although reftable produces a slightly +larger file (+ ~3.2%, 28.3M vs 29.2M): + +[cols=">,>,>,>",options="header",] +|===================================== +|format |size |seek cold |seek hot +|mh-alt |28.3 M |23.4 usec |11.2 usec +|reftable |29.2 M |19.9 usec |5.4 usec +|===================================== + +JGit Ketch RefTree +^^^^^^^^^^^^^^^^^^ + +https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg03073.html[JGit Ketch] +proposed +link:https://lore.kernel.org/git/CAJo%3DhJvnAPNAdDcAAwAvU9C4RVeQdoS3Ev9WTguHx4fD0V_nOg%40mail.gmail.com/[RefTree], +an encoding of references inside Git tree objects stored as part of the +repository's object database. + +The RefTree format adds additional load on the object database storage +layer (more loose objects, more objects in packs), and relies heavily on +the packer's delta compression to save space. Namespaces which are flat +(e.g. thousands of tags in refs/tags) initially create very large loose +objects, and so RefTree does not address the problem of copying many +references to modify a handful. + +Flat namespaces are not efficiently searchable in RefTree, as tree +objects in canonical formatting cannot be binary searched. This fails +the need to handle a large number of references in a single namespace, +such as GitHub's `refs/pulls`, or a project with many tags. + +LMDB +^^^^ + +David Turner proposed +https://lore.kernel.org/git/1455772670-21142-26-git-send-email-dturner@twopensource.com/[using +LMDB], as LMDB is lightweight (64k of runtime code) and GPL-compatible +license. + +A downside of LMDB is its reliance on a single C implementation. This +makes embedding inside JGit (a popular reimplementation of Git) +difficult, and hoisting onto virtual storage (for JGit DFS) virtually +impossible. + +A common format that can be supported by all major Git implementations +(git-core, JGit, libgit2) is strongly preferred. diff --git a/Documentation/user-manual.txt b/Documentation/user-manual.txt index 833652983f..fd480b8645 100644 --- a/Documentation/user-manual.txt +++ b/Documentation/user-manual.txt @@ -347,7 +347,7 @@ $ git branch -r origin/man origin/master origin/next - origin/pu + origin/seen origin/todo ------------------------------------------------ |