diff options
Diffstat (limited to 'Documentation/technical')
| -rw-r--r-- | Documentation/technical/api-directory-listing.txt | 2 | ||||
| -rw-r--r-- | Documentation/technical/api-gitattributes.txt | 2 | ||||
| -rw-r--r-- | Documentation/technical/commit-graph-format.txt | 10 | ||||
| -rw-r--r-- | Documentation/technical/commit-graph.txt | 22 | ||||
| -rw-r--r-- | Documentation/technical/directory-rename-detection.txt | 115 | ||||
| -rw-r--r-- | Documentation/technical/hash-function-transition.txt | 202 | ||||
| -rw-r--r-- | Documentation/technical/http-protocol.txt | 4 | ||||
| -rw-r--r-- | Documentation/technical/pack-protocol.txt | 7 | ||||
| -rw-r--r-- | Documentation/technical/protocol-v2.txt | 31 |
9 files changed, 261 insertions, 134 deletions
diff --git a/Documentation/technical/api-directory-listing.txt b/Documentation/technical/api-directory-listing.txt index 4f44ca24f6..5abb8e8b1f 100644 --- a/Documentation/technical/api-directory-listing.txt +++ b/Documentation/technical/api-directory-listing.txt @@ -54,7 +54,7 @@ The notable options are: this case, the contents are returned as individual entries. + If this is set, files and directories that explicitly match an ignore -pattern are reported. Implicity ignored directories (directories that +pattern are reported. Implicitly ignored directories (directories that do not match an ignore pattern, but whose contents are all ignored) are not reported, instead all of the contents are reported. diff --git a/Documentation/technical/api-gitattributes.txt b/Documentation/technical/api-gitattributes.txt index e7cbb7c13a..45f0df600f 100644 --- a/Documentation/technical/api-gitattributes.txt +++ b/Documentation/technical/api-gitattributes.txt @@ -146,7 +146,7 @@ To get the values of all attributes associated with a file: * Iterate over the `attr_check.items[]` array to examine the attribute names and values. The name of the attribute - described by a `attr_check.items[]` object can be retrieved via + described by an `attr_check.items[]` object can be retrieved via `git_attr_name(check->items[i].attr)`. (Please note that no items will be returned for unset attributes, so `ATTR_UNSET()` will return false for all returned `attr_check.items[]` objects.) diff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt index ad6af8105c..cc0474ba3e 100644 --- a/Documentation/technical/commit-graph-format.txt +++ b/Documentation/technical/commit-graph-format.txt @@ -18,9 +18,9 @@ metadata, including: the graph file. These positional references are stored as unsigned 32-bit integers -corresponding to the array position withing the list of commit OIDs. We -use the most-significant bit for special purposes, so we can store at most -(1 << 31) - 1 (around 2 billion) commits. +corresponding to the array position within the list of commit OIDs. Due +to some special constants we use to track parents, we can store at most +(1 << 30) + (1 << 29) + (1 << 28) - 1 (around 1.8 billion) commits. == Commit graph files have the following format: @@ -70,10 +70,10 @@ CHUNK DATA: OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes) The OIDs for all commits in the graph, sorted in ascending order. - Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes) + Commit Data (ID: {'C', 'D', 'A', 'T' }) (N * (H + 16) bytes) * The first H bytes are for the OID of the root tree. * The next 8 bytes are for the positions of the first two parents - of the ith commit. Stores value 0xffffffff if no parent in that + of the ith commit. Stores value 0x7000000 if no parent in that position. If there are more than two parents, the second value has its most-significant bit on and the other bits store an array position into the Large Edge List chunk. diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt index e1a883eb46..c664acbd76 100644 --- a/Documentation/technical/commit-graph.txt +++ b/Documentation/technical/commit-graph.txt @@ -118,9 +118,6 @@ Future Work - The commit graph feature currently does not honor commit grafts. This can be remedied by duplicating or refactoring the current graft logic. -- The 'commit-graph' subcommand does not have a "verify" mode that is - necessary for integration with fsck. - - After computing and storing generation numbers, we must make graph walks aware of generation numbers to gain the performance benefits they enable. This will mostly be accomplished by swapping a commit-date-ordered @@ -130,25 +127,6 @@ Future Work - 'log --topo-order' - 'tag --merged' -- Currently, parse_commit_gently() requires filling in the root tree - object for a commit. This passes through lookup_tree() and consequently - lookup_object(). Also, it calls lookup_commit() when loading the parents. - These method calls check the ODB for object existence, even if the - consumer does not need the content. For example, we do not need the - tree contents when computing merge bases. Now that commit parsing is - removed from the computation time, these lookup operations are the - slowest operations keeping graph walks from being fast. Consider - loading these objects without verifying their existence in the ODB and - only loading them fully when consumers need them. Consider a method - such as "ensure_tree_loaded(commit)" that fully loads a tree before - using commit->tree. - -- The current design uses the 'commit-graph' subcommand to generate the graph. - When this feature stabilizes enough to recommend to most users, we should - add automatic graph writes to common operations that create many commits. - For example, one could compute a graph on 'clone', 'fetch', or 'repack' - commands. - - A server could provide a commit graph file as part of the network protocol to avoid extra calculations by clients. This feature is only of benefit if the user is willing to trust the file, because verifying the file is correct diff --git a/Documentation/technical/directory-rename-detection.txt b/Documentation/technical/directory-rename-detection.txt new file mode 100644 index 0000000000..1c0086e287 --- /dev/null +++ b/Documentation/technical/directory-rename-detection.txt @@ -0,0 +1,115 @@ +Directory rename detection +========================== + +Rename detection logic in diffcore-rename that checks for renames of +individual files is aggregated and analyzed in merge-recursive for cases +where combinations of renames indicate that a full directory has been +renamed. + +Scope of abilities +------------------ + +It is perhaps easiest to start with an example: + + * When all of x/a, x/b and x/c have moved to z/a, z/b and z/c, it is + likely that x/d added in the meantime would also want to move to z/d by + taking the hint that the entire directory 'x' moved to 'z'. + +More interesting possibilities exist, though, such as: + + * one side of history renames x -> z, and the other renames some file to + x/e, causing the need for the merge to do a transitive rename. + + * one side of history renames x -> z, but also renames all files within + x. For example, x/a -> z/alpha, x/b -> z/bravo, etc. + + * both 'x' and 'y' being merged into a single directory 'z', with a + directory rename being detected for both x->z and y->z. + + * not all files in a directory being renamed to the same location; + i.e. perhaps most the files in 'x' are now found under 'z', but a few + are found under 'w'. + + * a directory being renamed, which also contained a subdirectory that was + renamed to some entirely different location. (And perhaps the inner + directory itself contained inner directories that were renamed to yet + other locations). + + * combinations of the above; see t/t6043-merge-rename-directories.sh for + various interesting cases. + +Limitations -- applicability of directory renames +------------------------------------------------- + +In order to prevent edge and corner cases resulting in either conflicts +that cannot be represented in the index or which might be too complex for +users to try to understand and resolve, a couple basic rules limit when +directory rename detection applies: + + 1) If a given directory still exists on both sides of a merge, we do + not consider it to have been renamed. + + 2) If a subset of to-be-renamed files have a file or directory in the + way (or would be in the way of each other), "turn off" the directory + rename for those specific sub-paths and report the conflict to the + user. + + 3) If the other side of history did a directory rename to a path that + your side of history renamed away, then ignore that particular + rename from the other side of history for any implicit directory + renames (but warn the user). + +Limitations -- detailed rules and testcases +------------------------------------------- + +t/t6043-merge-rename-directories.sh contains extensive tests and commentary +which generate and explore the rules listed above. It also lists a few +additional rules: + + a) If renames split a directory into two or more others, the directory + with the most renames, "wins". + + b) Avoid directory-rename-detection for a path, if that path is the + source of a rename on either side of a merge. + + c) Only apply implicit directory renames to directories if the other side + of history is the one doing the renaming. + +Limitations -- support in different commands +-------------------------------------------- + +Directory rename detection is supported by 'merge' and 'cherry-pick'. +Other git commands which users might be surprised to see limited or no +directory rename detection support in: + + * diff + + Folks have requested in the past that `git diff` detect directory + renames and somehow simplify its output. It is not clear whether this + would be desirable or how the output should be simplified, so this was + simply not implemented. Further, to implement this, directory rename + detection logic would need to move from merge-recursive to + diffcore-rename. + + * am + + git-am tries to avoid a full three way merge, instead calling + git-apply. That prevents us from detecting renames at all, which may + defeat the directory rename detection. There is a fallback, though; if + the initial git-apply fails and the user has specified the -3 option, + git-am will fall back to a three way merge. However, git-am lacks the + necessary information to do a "real" three way merge. Instead, it has + to use build_fake_ancestor() to get a merge base that is missing files + whose rename may have been important to detect for directory rename + detection to function. + + * rebase + + Since am-based rebases work by first generating a bunch of patches + (which no longer record what the original commits were and thus don't + have the necessary info from which we can find a real merge-base), and + then calling git-am, this implies that am-based rebases will not always + successfully detect directory renames either (see the 'am' section + above). merged-based rebases (rebase -m) and cherry-pick-based rebases + (rebase -i) are not affected by this shortcoming, and fully support + directory rename detection. diff --git a/Documentation/technical/hash-function-transition.txt b/Documentation/technical/hash-function-transition.txt index 4ab6cd1012..bc2ace2a6e 100644 --- a/Documentation/technical/hash-function-transition.txt +++ b/Documentation/technical/hash-function-transition.txt @@ -59,14 +59,11 @@ that are believed to be cryptographically secure. Goals ----- -Where NewHash is a strong 256-bit hash function to replace SHA-1 (see -"Selection of a New Hash", below): - -1. The transition to NewHash can be done one local repository at a time. +1. The transition to SHA-256 can be done one local repository at a time. a. Requiring no action by any other party. - b. A NewHash repository can communicate with SHA-1 Git servers + b. A SHA-256 repository can communicate with SHA-1 Git servers (push/fetch). - c. Users can use SHA-1 and NewHash identifiers for objects + c. Users can use SHA-1 and SHA-256 identifiers for objects interchangeably (see "Object names on the command line", below). d. New signed objects make use of a stronger hash function than SHA-1 for their security guarantees. @@ -79,7 +76,7 @@ Where NewHash is a strong 256-bit hash function to replace SHA-1 (see Non-Goals --------- -1. Add NewHash support to Git protocol. This is valuable and the +1. Add SHA-256 support to Git protocol. This is valuable and the logical next step but it is out of scope for this initial design. 2. Transparently improving the security of existing SHA-1 signed objects. @@ -87,26 +84,26 @@ Non-Goals repository. 4. Taking the opportunity to fix other bugs in Git's formats and protocols. -5. Shallow clones and fetches into a NewHash repository. (This will - change when we add NewHash support to Git protocol.) -6. Skip fetching some submodules of a project into a NewHash - repository. (This also depends on NewHash support in Git +5. Shallow clones and fetches into a SHA-256 repository. (This will + change when we add SHA-256 support to Git protocol.) +6. Skip fetching some submodules of a project into a SHA-256 + repository. (This also depends on SHA-256 support in Git protocol.) Overview -------- We introduce a new repository format extension. Repositories with this -extension enabled use NewHash instead of SHA-1 to name their objects. +extension enabled use SHA-256 instead of SHA-1 to name their objects. This affects both object names and object content --- both the names of objects and all references to other objects within an object are switched to the new hash function. -NewHash repositories cannot be read by older versions of Git. +SHA-256 repositories cannot be read by older versions of Git. -Alongside the packfile, a NewHash repository stores a bidirectional -mapping between NewHash and SHA-1 object names. The mapping is generated +Alongside the packfile, a SHA-256 repository stores a bidirectional +mapping between SHA-256 and SHA-1 object names. The mapping is generated locally and can be verified using "git fsck". Object lookups use this -mapping to allow naming objects using either their SHA-1 and NewHash names +mapping to allow naming objects using either their SHA-1 and SHA-256 names interchangeably. "git cat-file" and "git hash-object" gain options to display an object @@ -116,7 +113,7 @@ object database so that they can be named using the appropriate name (using the bidirectional hash mapping). Fetches from a SHA-1 based server convert the fetched objects into -NewHash form and record the mapping in the bidirectional mapping table +SHA-256 form and record the mapping in the bidirectional mapping table (see below for details). Pushes to a SHA-1 based server convert the objects being pushed into sha1 form so the server does not have to be aware of the hash function the client is using. @@ -125,19 +122,19 @@ Detailed Design --------------- Repository format extension ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -A NewHash repository uses repository format version `1` (see +A SHA-256 repository uses repository format version `1` (see Documentation/technical/repository-version.txt) with extensions `objectFormat` and `compatObjectFormat`: [core] repositoryFormatVersion = 1 [extensions] - objectFormat = newhash + objectFormat = sha256 compatObjectFormat = sha1 The combination of setting `core.repositoryFormatVersion=1` and populating `extensions.*` ensures that all versions of Git later than -`v0.99.9l` will die instead of trying to operate on the NewHash +`v0.99.9l` will die instead of trying to operate on the SHA-256 repository, instead producing an error message. # Between v0.99.9l and v2.7.0 @@ -155,36 +152,36 @@ repository extensions. Object names ~~~~~~~~~~~~ Objects can be named by their 40 hexadecimal digit sha1-name or 64 -hexadecimal digit newhash-name, plus names derived from those (see +hexadecimal digit sha256-name, plus names derived from those (see gitrevisions(7)). The sha1-name of an object is the SHA-1 of the concatenation of its type, length, a nul byte, and the object's sha1-content. This is the traditional <sha1> used in Git to name objects. -The newhash-name of an object is the NewHash of the concatenation of its -type, length, a nul byte, and the object's newhash-content. +The sha256-name of an object is the SHA-256 of the concatenation of its +type, length, a nul byte, and the object's sha256-content. Object format ~~~~~~~~~~~~~ The content as a byte sequence of a tag, commit, or tree object named -by sha1 and newhash differ because an object named by newhash-name refers to -other objects by their newhash-names and an object named by sha1-name +by sha1 and sha256 differ because an object named by sha256-name refers to +other objects by their sha256-names and an object named by sha1-name refers to other objects by their sha1-names. -The newhash-content of an object is the same as its sha1-content, except -that objects referenced by the object are named using their newhash-names +The sha256-content of an object is the same as its sha1-content, except +that objects referenced by the object are named using their sha256-names instead of sha1-names. Because a blob object does not refer to any -other object, its sha1-content and newhash-content are the same. +other object, its sha1-content and sha256-content are the same. -The format allows round-trip conversion between newhash-content and +The format allows round-trip conversion between sha256-content and sha1-content. Object storage ~~~~~~~~~~~~~~ Loose objects use zlib compression and packed objects use the packed format described in Documentation/technical/pack-format.txt, just like -today. The content that is compressed and stored uses newhash-content +today. The content that is compressed and stored uses sha256-content instead of sha1-content. Pack index @@ -255,10 +252,10 @@ network byte order): up to and not including the table of CRC32 values. - Zero or more NUL bytes. - The trailer consists of the following: - - A copy of the 20-byte NewHash checksum at the end of the + - A copy of the 20-byte SHA-256 checksum at the end of the corresponding packfile. - - 20-byte NewHash checksum of all of the above. + - 20-byte SHA-256 checksum of all of the above. Loose object index ~~~~~~~~~~~~~~~~~~ @@ -266,7 +263,7 @@ A new file $GIT_OBJECT_DIR/loose-object-idx contains information about all loose objects. Its format is # loose-object-idx - (newhash-name SP sha1-name LF)* + (sha256-name SP sha1-name LF)* where the object names are in hexadecimal format. The file is not sorted. @@ -292,8 +289,8 @@ To remove entries (e.g. in "git pack-refs" or "git-prune"): Translation table ~~~~~~~~~~~~~~~~~ The index files support a bidirectional mapping between sha1-names -and newhash-names. The lookup proceeds similarly to ordinary object -lookups. For example, to convert a sha1-name to a newhash-name: +and sha256-names. The lookup proceeds similarly to ordinary object +lookups. For example, to convert a sha1-name to a sha256-name: 1. Look for the object in idx files. If a match is present in the idx's sorted list of truncated sha1-names, then: @@ -301,8 +298,8 @@ lookups. For example, to convert a sha1-name to a newhash-name: name order mapping. b. Read the corresponding entry in the full sha1-name table to verify we found the right object. If it is, then - c. Read the corresponding entry in the full newhash-name table. - That is the object's newhash-name. + c. Read the corresponding entry in the full sha256-name table. + That is the object's sha256-name. 2. Check for a loose object. Read lines from loose-object-idx until we find a match. @@ -318,25 +315,25 @@ for all objects in the object store. Reading an object's sha1-content ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The sha1-content of an object can be read by converting all newhash-names -its newhash-content references to sha1-names using the translation table. +The sha1-content of an object can be read by converting all sha256-names +its sha256-content references to sha1-names using the translation table. Fetch ~~~~~ Fetching from a SHA-1 based server requires translating between SHA-1 -and NewHash based representations on the fly. +and SHA-256 based representations on the fly. SHA-1s named in the ref advertisement that are present on the client -can be translated to NewHash and looked up as local objects using the +can be translated to SHA-256 and looked up as local objects using the translation table. Negotiation proceeds as today. Any "have"s generated locally are converted to SHA-1 before being sent to the server, and SHA-1s -mentioned by the server are converted to NewHash when looking them up +mentioned by the server are converted to SHA-256 when looking them up locally. After negotiation, the server sends a packfile containing the -requested objects. We convert the packfile to NewHash format using +requested objects. We convert the packfile to SHA-256 format using the following steps: 1. index-pack: inflate each object in the packfile and compute its @@ -351,12 +348,12 @@ the following steps: (This list only contains objects reachable from the "wants". If the pack from the server contained additional extraneous objects, then they will be discarded.) -3. convert to newhash: open a new (newhash) packfile. Read the topologically +3. convert to sha256: open a new (sha256) packfile. Read the topologically sorted list just generated. For each object, inflate its - sha1-content, convert to newhash-content, and write it to the newhash - pack. Record the new sha1<->newhash mapping entry for use in the idx. + sha1-content, convert to sha256-content, and write it to the sha256 + pack. Record the new sha1<->sha256 mapping entry for use in the idx. 4. sort: reorder entries in the new pack to match the order of objects - in the pack the server generated and include blobs. Write a newhash idx + in the pack the server generated and include blobs. Write a sha256 idx file 5. clean up: remove the SHA-1 based pack file, index, and topologically sorted list obtained from the server in steps 1 @@ -388,16 +385,16 @@ send-pack. Signed Commits ~~~~~~~~~~~~~~ -We add a new field "gpgsig-newhash" to the commit object format to allow +We add a new field "gpgsig-sha256" to the commit object format to allow signing commits without relying on SHA-1. It is similar to the -existing "gpgsig" field. Its signed payload is the newhash-content of the -commit object with any "gpgsig" and "gpgsig-newhash" fields removed. +existing "gpgsig" field. Its signed payload is the sha256-content of the +commit object with any "gpgsig" and "gpgsig-sha256" fields removed. This means commits can be signed 1. using SHA-1 only, as in existing signed commit objects -2. using both SHA-1 and NewHash, by using both gpgsig-newhash and gpgsig +2. using both SHA-1 and SHA-256, by using both gpgsig-sha256 and gpgsig fields. -3. using only NewHash, by only using the gpgsig-newhash field. +3. using only SHA-256, by only using the gpgsig-sha256 field. Old versions of "git verify-commit" can verify the gpgsig signature in cases (1) and (2) without modifications and view case (3) as an @@ -405,24 +402,24 @@ ordinary unsigned commit. Signed Tags ~~~~~~~~~~~ -We add a new field "gpgsig-newhash" to the tag object format to allow +We add a new field "gpgsig-sha256" to the tag object format to allow signing tags without relying on SHA-1. Its signed payload is the -newhash-content of the tag with its gpgsig-newhash field and "-----BEGIN PGP +sha256-content of the tag with its gpgsig-sha256 field and "-----BEGIN PGP SIGNATURE-----" delimited in-body signature removed. This means tags can be signed 1. using SHA-1 only, as in existing signed tag objects -2. using both SHA-1 and NewHash, by using gpgsig-newhash and an in-body +2. using both SHA-1 and SHA-256, by using gpgsig-sha256 and an in-body signature. -3. using only NewHash, by only using the gpgsig-newhash field. +3. using only SHA-256, by only using the gpgsig-sha256 field. Mergetag embedding ~~~~~~~~~~~~~~~~~~ The mergetag field in the sha1-content of a commit contains the sha1-content of a tag that was merged by that commit. -The mergetag field in the newhash-content of the same commit contains the -newhash-content of the same tag. +The mergetag field in the sha256-content of the same commit contains the +sha256-content of the same tag. Submodules ~~~~~~~~~~ @@ -497,7 +494,7 @@ Caveats ------- Invalid objects ~~~~~~~~~~~~~~~ -The conversion from sha1-content to newhash-content retains any +The conversion from sha1-content to sha256-content retains any brokenness in the original object (e.g., tree entry modes encoded with leading 0, tree objects whose paths are not sorted correctly, and commit objects without an author or committer). This is a deliberate @@ -516,7 +513,7 @@ allow lifting this restriction. Alternates ~~~~~~~~~~ -For the same reason, a newhash repository cannot borrow objects from a +For the same reason, a sha256 repository cannot borrow objects from a sha1 repository using objects/info/alternates or $GIT_ALTERNATE_OBJECT_REPOSITORIES. @@ -524,20 +521,20 @@ git notes ~~~~~~~~~ The "git notes" tool annotates objects using their sha1-name as key. This design does not describe a way to migrate notes trees to use -newhash-names. That migration is expected to happen separately (for +sha256-names. That migration is expected to happen separately (for example using a file at the root of the notes tree to describe which hash it uses). Server-side cost ~~~~~~~~~~~~~~~~ -Until Git protocol gains NewHash support, using NewHash based storage +Until Git protocol gains SHA-256 support, using SHA-256 based storage on public-facing Git servers is strongly discouraged. Once Git -protocol gains NewHash support, NewHash based servers are likely not +protocol gains SHA-256 support, SHA-256 based servers are likely not to support SHA-1 compatibility, to avoid what may be a very expensive hash reencode during clone and to encourage peers to modernize. The design described here allows fetches by SHA-1 clients of a -personal NewHash repository because it's not much more difficult than +personal SHA-256 repository because it's not much more difficult than allowing pushes from that repository. This support needs to be guarded by a configuration option --- servers like git.kernel.org that serve a large number of clients would not be expected to bear that cost. @@ -547,7 +544,7 @@ Meaning of signatures The signed payload for signed commits and tags does not explicitly name the hash used to identify objects. If some day Git adopts a new hash function with the same length as the current SHA-1 (40 -hexadecimal digit) or NewHash (64 hexadecimal digit) objects then the +hexadecimal digit) or SHA-256 (64 hexadecimal digit) objects then the intent behind the PGP signed payload in an object signature is unclear: @@ -562,7 +559,7 @@ Does this mean Git v2.12.0 is the commit with sha1-name e7e07d5a4fcc2a203d9873968ad3e6bd4d7419d7 or the commit with new-40-digit-hash-name e7e07d5a4fcc2a203d9873968ad3e6bd4d7419d7? -Fortunately NewHash and SHA-1 have different lengths. If Git starts +Fortunately SHA-256 and SHA-1 have different lengths. If Git starts using another hash with the same length to name objects, then it will need to change the format of signed payloads using that hash to address this issue. @@ -574,24 +571,24 @@ supports four different modes of operation: 1. ("dark launch") Treat object names input by the user as SHA-1 and convert any object names written to output to SHA-1, but store - objects using NewHash. This allows users to test the code with no + objects using SHA-256. This allows users to test the code with no visible behavior change except for performance. This allows allows running even tests that assume the SHA-1 hash function, to sanity-check the behavior of the new mode. - 2. ("early transition") Allow both SHA-1 and NewHash object names in + 2. ("early transition") Allow both SHA-1 and SHA-256 object names in input. Any object names written to output use SHA-1. This allows users to continue to make use of SHA-1 to communicate with peers (e.g. by email) that have not migrated yet and prepares for mode 3. - 3. ("late transition") Allow both SHA-1 and NewHash object names in - input. Any object names written to output use NewHash. In this + 3. ("late transition") Allow both SHA-1 and SHA-256 object names in + input. Any object names written to output use SHA-256. In this mode, users are using a more secure object naming method by default. The disruption is minimal as long as most of their peers are in mode 2 or mode 3. 4. ("post-transition") Treat object names input by the user as - NewHash and write output using NewHash. This is safer than mode 3 + SHA-256 and write output using SHA-256. This is safer than mode 3 because there is less risk that input is incorrectly interpreted using the wrong hash function. @@ -601,27 +598,31 @@ The user can also explicitly specify which format to use for a particular revision specifier and for output, overriding the mode. For example: -git --output-format=sha1 log abac87a^{sha1}..f787cac^{newhash} +git --output-format=sha1 log abac87a^{sha1}..f787cac^{sha256} -Selection of a New Hash ------------------------ +Choice of Hash +-------------- In early 2005, around the time that Git was written, Xiaoyun Wang, Yiqun Lisa Yin, and Hongbo Yu announced an attack finding SHA-1 collisions in 2^69 operations. In August they published details. Luckily, no practical demonstrations of a collision in full SHA-1 were published until 10 years later, in 2017. -The hash function NewHash to replace SHA-1 should be stronger than -SHA-1 was: we would like it to be trustworthy and useful in practice -for at least 10 years. +Git v2.13.0 and later subsequently moved to a hardened SHA-1 +implementation by default that mitigates the SHAttered attack, but +SHA-1 is still believed to be weak. + +The hash to replace this hardened SHA-1 should be stronger than SHA-1 +was: we would like it to be trustworthy and useful in practice for at +least 10 years. Some other relevant properties: 1. A 256-bit hash (long enough to match common security practice; not excessively long to hurt performance and disk usage). -2. High quality implementations should be widely available (e.g. in - OpenSSL). +2. High quality implementations should be widely available (e.g., in + OpenSSL and Apple CommonCrypto). 3. The hash function's properties should match Git's needs (e.g. Git requires collision and 2nd preimage resistance and does not require @@ -630,14 +631,13 @@ Some other relevant properties: 4. As a tiebreaker, the hash should be fast to compute (fortunately many contenders are faster than SHA-1). -Some hashes under consideration are SHA-256, SHA-512/256, SHA-256x16, -K12, and BLAKE2bp-256. +We choose SHA-256. Transition plan --------------- Some initial steps can be implemented independently of one another: - adding a hash function API (vtable) -- teaching fsck to tolerate the gpgsig-newhash field +- teaching fsck to tolerate the gpgsig-sha256 field - excluding gpgsig-* from the fields copied by "git commit --amend" - annotating tests that depend on SHA-1 values with a SHA1 test prerequisite @@ -664,7 +664,7 @@ Next comes introduction of compatObjectFormat: - adding appropriate index entries when adding a new object to the object store - --output-format option -- ^{sha1} and ^{newhash} revision notation +- ^{sha1} and ^{sha256} revision notation - configuration to specify default input and output format (see "Object names on the command line" above) @@ -672,7 +672,7 @@ The next step is supporting fetches and pushes to SHA-1 repositories: - allow pushes to a repository using the compat format - generate a topologically sorted list of the SHA-1 names of fetched objects -- convert the fetched packfile to newhash format and generate an idx +- convert the fetched packfile to sha256 format and generate an idx file - re-sort to match the order of objects in the fetched packfile @@ -680,30 +680,30 @@ The infrastructure supporting fetch also allows converting an existing repository. In converted repositories and new clones, end users can gain support for the new hash function without any visible change in behavior (see "dark launch" in the "Object names on the command line" -section). In particular this allows users to verify NewHash signatures +section). In particular this allows users to verify SHA-256 signatures on objects in the repository, and it should ensure the transition code is stable in production in preparation for using it more widely. Over time projects would encourage their users to adopt the "early transition" and then "late transition" modes to take advantage of the -new, more futureproof NewHash object names. +new, more futureproof SHA-256 object names. When objectFormat and compatObjectFormat are both set, commands -generating signatures would generate both SHA-1 and NewHash signatures +generating signatures would generate both SHA-1 and SHA-256 signatures by default to support both new and old users. -In projects using NewHash heavily, users could be encouraged to adopt +In projects using SHA-256 heavily, users could be encouraged to adopt the "post-transition" mode to avoid accidentally making implicit use of SHA-1 object names. Once a critical mass of users have upgraded to a version of Git that -can verify NewHash signatures and have converted their existing +can verify SHA-256 signatures and have converted their existing repositories to support verifying them, we can add support for a -setting to generate only NewHash signatures. This is expected to be at +setting to generate only SHA-256 signatures. This is expected to be at least a year later. That is also a good moment to advertise the ability to convert -repositories to use NewHash only, stripping out all SHA-1 related +repositories to use SHA-256 only, stripping out all SHA-1 related metadata. This improves performance by eliminating translation overhead and security by avoiding the possibility of accidentally relying on the safety of SHA-1. @@ -742,16 +742,16 @@ using the old hash function. Signed objects with multiple hashes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Instead of introducing the gpgsig-newhash field in commit and tag objects -for newhash-content based signatures, an earlier version of this design -added "hash newhash <newhash-name>" fields to strengthen the existing +Instead of introducing the gpgsig-sha256 field in commit and tag objects +for sha256-content based signatures, an earlier version of this design +added "hash sha256 <sha256-name>" fields to strengthen the existing sha1-content based signatures. In other words, a single signature was used to attest to the object content using both hash functions. This had some advantages: * Using one signature instead of two speeds up the signing process. * Having one signed payload with both hashes allows the signer to - attest to the sha1-name and newhash-name referring to the same object. + attest to the sha1-name and sha256-name referring to the same object. * All users consume the same signature. Broken signatures are likely to be detected quickly using current versions of git. @@ -760,11 +760,11 @@ However, it also came with disadvantages: objects it references, even after the transition is complete and translation table is no longer needed for anything else. To support this, the design added fields such as "hash sha1 tree <sha1-name>" - and "hash sha1 parent <sha1-name>" to the newhash-content of a signed + and "hash sha1 parent <sha1-name>" to the sha256-content of a signed commit, complicating the conversion process. * Allowing signed objects without a sha1 (for after the transition is complete) complicated the design further, requiring a "nohash sha1" - field to suppress including "hash sha1" fields in the newhash-content + field to suppress including "hash sha1" fields in the sha256-content and signed payload. Lazily populated translation table @@ -772,7 +772,7 @@ Lazily populated translation table Some of the work of building the translation table could be deferred to push time, but that would significantly complicate and slow down pushes. Calculating the sha1-name at object creation time at the same time it is -being streamed to disk and having its newhash-name calculated should be +being streamed to disk and having its sha256-name calculated should be an acceptable cost. Document History @@ -814,6 +814,12 @@ Incorporated suggestions from jonathantanmy and sbeller: * avoid loose object overhead by packing more aggressively in "git gc --auto" +Later history: + + See the history of this file in git.git for the history of subsequent + edits. This document history is no longer being maintained as it + would now be superfluous to the commit log + [1] http://public-inbox.org/git/CA+55aFzJtejiCjV0e43+9oR3QuJK2PiFiLQemytoLpyJWe6P9w@mail.gmail.com/ [2] http://public-inbox.org/git/CA+55aFz+gkAsDZ24zmePQuEs1XPS9BP_s8O7Q4wQ7LV7X5-oDA@mail.gmail.com/ [3] http://public-inbox.org/git/20170306084353.nrns455dvkdsfgo5@sigill.intra.peff.net/ diff --git a/Documentation/technical/http-protocol.txt b/Documentation/technical/http-protocol.txt index 64f49d0bbb..9c5b6f0fac 100644 --- a/Documentation/technical/http-protocol.txt +++ b/Documentation/technical/http-protocol.txt @@ -338,11 +338,11 @@ server advertises capability `allow-tip-sha1-in-want` or request_end request_end = "0000" / "done" - want_list = PKT-LINE(want NUL cap_list LF) + want_list = PKT-LINE(want SP cap_list LF) *(want_pkt) want_pkt = PKT-LINE(want LF) want = "want" SP id - cap_list = *(SP capability) SP + cap_list = capability *(SP capability) have_list = *PKT-LINE("have" SP id LF) diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt index 7fee6b780a..6ac774d5f6 100644 --- a/Documentation/technical/pack-protocol.txt +++ b/Documentation/technical/pack-protocol.txt @@ -50,7 +50,8 @@ Each Extra Parameter takes the form of `<key>=<value>` or `<key>`. Servers that receive any such Extra Parameters MUST ignore all unrecognized keys. Currently, the only Extra Parameter recognized is -"version=1". +"version" with a value of '1' or '2'. See protocol-v2.txt for more +information on protocol version 2. Git Transport ------------- @@ -284,7 +285,9 @@ information is sent back to the client in the next step. The client can optionally request that pack-objects omit various objects from the packfile using one of several filtering techniques. These are intended for use with partial clone and partial fetch -operations. See `rev-list` for possible "filter-spec" values. +operations. An object that does not meet a filter-spec value is +omitted unless explicitly requested in a 'want' line. See `rev-list` +for possible filter-spec values. Once all the 'want's and 'shallow's (and optional 'deepen') are transferred, clients MUST send a flush-pkt, to tell the server side diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt index 49bda76d23..09e4e0273f 100644 --- a/Documentation/technical/protocol-v2.txt +++ b/Documentation/technical/protocol-v2.txt @@ -64,9 +64,8 @@ When using the http:// or https:// transport a client makes a "smart" info/refs request as described in `http-protocol.txt` and requests that v2 be used by supplying "version=2" in the `Git-Protocol` header. - C: Git-Protocol: version=2 - C: C: GET $GIT_URL/info/refs?service=git-upload-pack HTTP/1.0 + C: Git-Protocol: version=2 A v2 server would reply: @@ -299,12 +298,21 @@ included in the client's request: for use with partial clone and partial fetch operations. See `rev-list` for possible "filter-spec" values. +If the 'ref-in-want' feature is advertised, the following argument can +be included in the client's request as well as the potential addition of +the 'wanted-refs' section in the server's response as explained below. + + want-ref <ref> + Indicates to the server that the client wants to retrieve a + particular ref, where <ref> is the full name of a ref on the + server. + The response of `fetch` is broken into a number of sections separated by delimiter packets (0001), with each section beginning with its section header. output = *section - section = (acknowledgments | shallow-info | packfile) + section = (acknowledgments | shallow-info | wanted-refs | packfile) (flush-pkt | delim-pkt) acknowledgments = PKT-LINE("acknowledgments" LF) @@ -319,6 +327,10 @@ header. shallow = "shallow" SP obj-id unshallow = "unshallow" SP obj-id + wanted-refs = PKT-LINE("wanted-refs" LF) + *PKT-LINE(wanted-ref LF) + wanted-ref = obj-id SP refname + packfile = PKT-LINE("packfile" LF) *PKT-LINE(%x01-03 *%x00-ff) @@ -379,6 +391,19 @@ header. * This section is only included if a packfile section is also included in the response. + wanted-refs section + * This section is only included if the client has requested a + ref using a 'want-ref' line and if a packfile section is also + included in the response. + + * Always begins with the section header "wanted-refs". + + * The server will send a ref listing ("<oid> <refname>") for + each reference requested using 'want-ref' lines. + + * The server MUST NOT send any refs which were not requested + using 'want-ref' lines. + packfile section * This section is only included if the client has sent 'want' lines in its request and either requested that no more |