summaryrefslogtreecommitdiff
path: root/Documentation/technical
AgeCommit message (Collapse)AuthorFilesLines
2020-12-17Merge branch 'jh/index-v2-doc-on-fsmn'Libravatar Junio C Hamano1-2/+8
Doc update. * jh/index-v2-doc-on-fsmn: index-format.txt: document v2 format of file system monitor extension
2020-12-17Merge branch 'jb/midx-doc-update'Libravatar Junio C Hamano1-4/+0
Doc update. * jb/midx-doc-update: docs: multi-pack-index: remove note about future 'verify' work
2020-12-14index-format.txt: document v2 format of file system monitor extensionLibravatar Jeff Hostetler1-2/+8
Update the documentation of the file system monitor extension to describe version 2. The format was extended to support opaque tokens in: 56c6910028 fsmonitor: change last update timestamp on the index_state to opaque token Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-12-14docs: multi-pack-index: remove note about future 'verify' workLibravatar Johannes Berg1-4/+0
This was implemented in the 'git multi-pack-index' command and merged in 468b3221 (Merge branch 'ds/multi-pack-verify', 2018-10-10). And there's no 'git midx' command. Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-12-08Merge branch 'js/trace2-session-id'Libravatar Junio C Hamano2-2/+28
The transport layer was taught to optionally exchange the session ID assigned by the trace2 subsystem during fetch/push transactions. * js/trace2-session-id: receive-pack: log received client session ID send-pack: advertise session ID in capabilities upload-pack, serve: log received client session ID fetch-pack: advertise session ID in capabilities transport: log received server session ID serve: advertise session ID in v2 capabilities receive-pack: advertise session ID in v0 capabilities upload-pack: advertise session ID in v0 capabilities trace2: add a public function for getting the SID docs: new transfer.advertiseSID option docs: new capability to advertise session IDs
2020-12-08Merge branch 'jt/trace-error-on-warning'Libravatar Junio C Hamano1-1/+1
Like die() and error(), a call to warning() will also trigger a trace2 event. * jt/trace-error-on-warning: usage: add trace2 entry upon warning()
2020-11-24usage: add trace2 entry upon warning()Libravatar Jonathan Tan1-1/+1
Emit a trace2 error event whenever warning() is called, just like when die(), error(), or usage() is called. This helps debugging issues that would trigger warnings but not errors. In particular, this might have helped debugging an issue I encountered with commit graphs at $DAYJOB [1]. There is a tradeoff between including potentially relevant messages and cluttering up the trace output produced. I think that warning() messages should be included in traces, because by its nature, Git is used over multiple invocations of the Git tool, and a failure (currently traced) in a Git invocation might be caused by an unexpected interaction in a previous Git invocation that only has a warning (currently untraced) as a symptom - as is the case in [1]. [1] https://lore.kernel.org/git/20200629220744.1054093-1-jonathantanmy@google.com/ Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-11-11docs: new capability to advertise session IDsLibravatar Josh Steadmon2-2/+28
In future patches, we will add the ability for Git servers and clients to advertise unique session IDs via protocol capabilities. This allows for easier debugging when both client and server logs are available. Signed-off-by: Josh Steadmon <steadmon@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-10-16t6423: more involved rules for renaming directories into each otherLibravatar Elijah Newren1-0/+3
Testcases 12b and 12c were both slightly weird; they were marked as having a weird resolution, but with the note that even straightforward simple rules can give weird results when the input is bizarre. However, during optimization work for merge-ort, I discovered a significant speedup that is possible if we add one more fairly straightforward rule: we don't bother doing directory rename detection if there are no new files added to the directory on the other side of the history to be affected by the directory rename. This seems like an obvious and straightforward rule, but there was one funny corner case where directory rename detection could affect only existing files: the funny corner case where two directories are renamed into each other on opposite sides of history. In other words, it only results in a different output for testcases 12b and 12c. Since we already thought testcases 12b and 12c were weird anyway, and because the optimization often has a significant effect on common cases (but is entirely prevented if we can't change how 12b and 12c function), let's add the additional rule and tweak how 12b and 12c work. Split both testcases into two (one where we add no new files, and one where the side that doesn't rename a given directory will add files to it), and mark them with the new expectation. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-10-16t6423: update directory rename detection tests with new ruleLibravatar Elijah Newren1-4/+1
While investigating the issues highlighted by the testcase in the previous patch, I also found a shortcoming in the directory rename detection rules. Split testcase 6b into two to explain this issue and update directory-rename-detection.txt to remove one of the previous rules that I know believe to be detrimental. Also, update the wording around testcase 8e; while we are not modifying the results of that testcase, we were previously unsure of the appropriate resolution of that test and the new rule makes the previously chosen resolution for that testcase a bit more solid. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-10-16directory-rename-detection.txt: update references to regression testsLibravatar Elijah Newren1-3/+4
The regression tests for directory rename detection were renamed from t6043 to t6423 in commit 919df31955 ("Collect merge-related tests to t64xx", 2020-08-10); update this file to match. Also, add a small clarification to nearby text while we're at it. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-29Merge branch 'tb/bloom-improvements'Libravatar Junio C Hamano1-1/+1
"git commit-graph write" learned to limit the number of bloom filters that are computed from scratch with the --max-new-filters option. * tb/bloom-improvements: commit-graph: introduce 'commitGraph.maxNewFilters' builtin/commit-graph.c: introduce '--max-new-filters=<n>' commit-graph: rename 'split_commit_graph_opts' bloom: encode out-of-bounds filters as non-empty bloom/diff: properly short-circuit on max_changes bloom: use provided 'struct bloom_filter_settings' bloom: split 'get_bloom_filter()' in two commit-graph.c: store maximum changed paths commit-graph: respect 'commitGraph.readChangedPaths' t/helper/test-read-graph.c: prepare repo settings commit-graph: pass a 'struct repository *' in more places t4216: use an '&&'-chain commit-graph: introduce 'get_bloom_filter_settings()'
2020-09-25Merge branch 'jx/proc-receive-hook'Libravatar Junio C Hamano2-6/+52
"git receive-pack" that accepts requests by "git push" learned to outsource most of the ref updates to the new "proc-receive" hook. * jx/proc-receive-hook: doc: add documentation for the proc-receive hook transport: parse report options for tracking refs t5411: test updates of remote-tracking branches receive-pack: new config receive.procReceiveRefs doc: add document for capability report-status-v2 New capability "report-status-v2" for git-push receive-pack: feed report options to post-receive receive-pack: add new proc-receive hook t5411: add basic test cases for proc-receive hook transport: not report a non-head push as a branch
2020-09-22Merge branch 'cd/commit-graph-doc'Libravatar Junio C Hamano1-1/+1
Doc update. * cd/commit-graph-doc: commit-graph-format.txt: fix no-parent value
2020-09-17bloom: encode out-of-bounds filters as non-emptyLibravatar Taylor Blau1-1/+1
When a changed-path Bloom filter has either zero, or more than a certain number (commonly 512) of entries, the commit-graph machinery encodes it as "missing". More specifically, it sets the indices adjacent in the BIDX chunk as equal to each other to indicate a "length 0" filter; that is, that the filter occupies zero bytes on disk. This has heretofore been fine, since the commit-graph machinery has no need to care about these filters with too few or too many changed paths. Both cases act like no filter has been generated at all, and so there is no need to store them. In a subsequent commit, however, the commit-graph machinery will learn to only compute Bloom filters for some commits in the current commit-graph layer. This is a change from the current implementation which computes Bloom filters for all commits that are in the layer being written. Critically for this patch, only computing some of the Bloom filters means adding a third state for length 0 Bloom filters: zero entries, too many entries, or "hasn't been computed". It will be important for that future patch to distinguish between "not representable" (i.e., zero or too-many changed paths), and "hasn't been computed". In particular, we don't want to waste time recomputing filters that have already been computed. To that end, change how we store Bloom filters in the "computed but not representable" category: - Bloom filters with no entries are stored as a single byte with all bits low (i.e., all queries to that Bloom filter will return "definitely not") - Bloom filters with too many entries are stored as a single byte with all bits set high (i.e., all queries to that Bloom filter will return "maybe"). These rules are sufficient to not incur a behavior change by changing the on-disk representation of these two classes. Likewise, no specification changes are necessary for the commit-graph format, either: - Filters that were previously empty will be recomputed and stored according to the new rules, and - old clients reading filters generated by new clients will interpret the filters correctly and be none the wiser to how they were generated. Clients will invoke the Bloom machinery in more cases than before, but this can be addressed by returning a NULL filter when all bits are set high. This can be addressed in a future patch. Note that this does increase the size of on-disk commit-graphs, but far less than other proposals. In particular, this is generally more efficient than storing a bitmap for which commits haven't computed their Bloom filters. Storing a bitmap incurs a penalty of one bit per commit, whereas storing explicit filters as above incurs a penalty of one byte per too-large or empty commit. In practice, these boundary commits likely occupy a small proportion of the overall number of commits, and so the size penalty is likely smaller than storing a bitmap for all commits. See, for example, these relative proportions of such boundary commits (collected by SZEDER Gábor): | Percentage of | commit-graph | | | commits modifying | file size | | ├────────┬──────────────┼───────────────────┤ pct. | | 0 path | >= 512 paths | before | after | change | ┌────────────────┼────────┼──────────────┼─────────┼─────────┼───────────┤ | android-base | 13.20% | 0.13% | 37.468M | 37.534M | +0.1741 % | | cmssw | 0.15% | 0.23% | 17.118M | 17.119M | +0.0091 % | | cpython | 3.07% | 0.01% | 7.967M | 7.971M | +0.0423 % | | elasticsearch | 0.70% | 1.00% | 8.833M | 8.835M | +0.0128 % | | gcc | 0.00% | 0.08% | 16.073M | 16.074M | +0.0030 % | | gecko-dev | 0.14% | 0.64% | 59.868M | 59.874M | +0.0105 % | | git | 0.11% | 0.02% | 3.895M | 3.895M | +0.0020 % | | glibc | 0.02% | 0.10% | 3.555M | 3.555M | +0.0021 % | | go | 0.00% | 0.07% | 3.186M | 3.186M | +0.0018 % | | homebrew-cask | 0.40% | 0.02% | 7.035M | 7.035M | +0.0065 % | | homebrew-core | 0.01% | 0.01% | 11.611M | 11.611M | +0.0002 % | | jdk | 0.26% | 5.64% | 5.537M | 5.540M | +0.0590 % | | linux | 0.01% | 0.51% | 63.735M | 63.740M | +0.0073 % | | llvm-project | 0.12% | 0.03% | 25.515M | 25.516M | +0.0050 % | | rails | 0.10% | 0.10% | 6.252M | 6.252M | +0.0027 % | | rust | 0.07% | 0.17% | 9.364M | 9.364M | +0.0033 % | | tensorflow | 0.09% | 1.02% | 7.009M | 7.010M | +0.0158 % | | webkit | 0.05% | 0.31% | 17.405M | 17.406M | +0.0047 % | (where the above increase is determined by computing a non-split commit-graph before and after this patch). Given that these projects are all "large" by commit count, the storage cost by writing these filters explicitly is negligible. In the most extreme example, android-base (which has 494,848 commits at the time of writing) would have its commit-graph increase by a modest 68.4 KB. Finally, a test to exercise filters which contain too many changed path entries will be introduced in a subsequent patch. Suggested-by: SZEDER Gábor <szeder.dev@gmail.com> Suggested-by: Jakub Narębski <jnareb@gmail.com> Helped-by: Derrick Stolee <dstolee@microsoft.com> Helped-by: SZEDER Gábor <szeder.dev@gmail.com> Helped-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-15commit-graph-format.txt: fix no-parent valueLibravatar Conor Davis1-1/+1
The correct value from commit-graph.c: #define GRAPH_PARENT_NONE 0x70000000 Signed-off-by: Conor Davis <git@conor.fastmail.fm> Reviewed-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-03Merge branch 'jt/lazy-fetch'Libravatar Junio C Hamano1-10/+3
Updates to on-demand fetching code in lazily cloned repositories. * jt/lazy-fetch: fetch: no FETCH_HEAD display if --no-write-fetch-head fetch-pack: remove no_dependents code promisor-remote: lazy-fetch objects in subprocess fetch-pack: do not lazy-fetch during ref iteration fetch: only populate existing_refs if needed fetch: avoid reading submodule config until needed fetch: allow refspecs specified through stdin negotiator/noop: add noop fetch negotiator
2020-08-27doc: add document for capability report-status-v2Libravatar Jiang Xin2-6/+52
Add ABNF notation for capability 'report-status-v2' which extends capability 'report-status' by adding additional option lines. Signed-off-by: Jiang Xin <zhiyou.jx@alibaba-inc.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-08-19Merge branch 'ds/sha256-leftover-bits'Libravatar Junio C Hamano2-3/+13
midx and commit-graph files now use the byte defined in their file format specification for identifying the hash function used for object names. * ds/sha256-leftover-bits: multi-pack-index: use hash version byte commit-graph: use the "hash version" byte t/README: document GIT_TEST_DEFAULT_HASH
2020-08-19Merge branch 'ma/sha-256-docs'Libravatar Junio C Hamano4-23/+30
Further update of docs to adjust to the recent SHA-256 work. * ma/sha-256-docs: shallow.txt: document SHA-256 shallow format protocol-capabilities.txt: clarify "allow-x-sha1-in-want" re SHA-256 index-format.txt: document SHA-256 index format http-protocol.txt: document SHA-256 "want"/"have" format
2020-08-19Merge branch 'bc/sha-256-doc-updates'Libravatar Junio C Hamano2-16/+22
Further update of docs to adjust to the recent SHA-256 work. * bc/sha-256-doc-updates: docs: fix step in transition plan docs: document SHA-256 pack and indices
2020-08-19Merge branch 'jb/commit-graph-doc-fix'Libravatar Junio C Hamano1-3/+3
Docfix. * jb/commit-graph-doc-fix: docs: commit-graph: fix some whitespace in the diagram
2020-08-18promisor-remote: lazy-fetch objects in subprocessLibravatar Jonathan Tan1-10/+3
Teach Git to lazy-fetch missing objects in a subprocess instead of doing it in-process. This allows any fatal errors that occur during the fetch to be isolated and converted into an error return value, instead of causing the current command being run to terminate. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-08-17multi-pack-index: use hash version byteLibravatar Derrick Stolee1-1/+6
Similar to the commit-graph format, the multi-pack-index format has a byte in the header intended to track the hash version used to write the file. This allows one to interpret the hash length without having the context of the repository config specifying the hash length. This was not modified as part of the SHA-256 work because the hash length was automatically up-shifted due to that config. Since we have this byte available, we can make the file formats more obviously incompatible instead of relying on other context from the repository. Add a new oid_version() method in midx.c similar to the one in commit-graph.c. This is specifically made separate from that implementation to avoid artificially linking the formats. The test impact requires a few more things than the corresponding change in the commit-graph format. Specifically, 'test-tool read-midx' was not writing anything about this header value to output. Since the value available in 'struct multi_pack_index' is hash_len instead of a version value, we output "20" or "32" instead of "1" or "2". Since we want a user to not have their Git commands fail if their multi-pack-index has the incorrect hash version compared to the repository's hash version, we relax the die() to an error() in load_multi_pack_index(). This has some effect on 'git multi-pack-index verify' as we need to check that a failed parse of a file that exists is actually a verify error. For that test that checks the hash version matches, we change the corrupted byte from "2" to "3" to ensure the test fails for both hash algorithms. Helped-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-08-17commit-graph: use the "hash version" byteLibravatar Derrick Stolee1-2/+7
The commit-graph format reserved a byte among the header of the file to store a "hash version". During the SHA-256 work, this was not modified because file formats are not necessarily intended to work across hash versions. If a repository has SHA-256 as its hash algorithm, it automatically up-shifts the lengths of object names in all necessary formats. However, since we have this byte available for adjusting the version, we can make the file formats more obviously incompatible instead of relying on other context from the repository. Update the oid_version() method in commit-graph.c to add a new value, 2, for sha-256. This automatically writes the new value in a SHA-256 repository _and_ verifies the value is correct. This is a breaking change relative to the current 'master' branch since 092b677 (Merge branch 'bc/sha-256-cvs-svn-updates', 2020-08-13) but it is not breaking relative to any released version of Git. The test impact is relatively minor: the output of 'test-tool read-graph' lists the header information, so those instances of '1' need to be replaced with a variable determined by GIT_TEST_DEFAULT_HASH. A more careful test is added that specifically creates a repository of each type then swaps the commit-graph files. The important value here is that the "git log" command succeeds while writing a message to stderr. Helped-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-08-17shallow.txt: document SHA-256 shallow formatLibravatar Martin Ågren1-1/+1
Similar to recent commits, document that we list object names rather than SHA-1s. Signed-off-by: Martin Ågren <martin.agren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-08-17protocol-capabilities.txt: clarify "allow-x-sha1-in-want" re SHA-256Libravatar Martin Ågren1-4/+8
Two of our capabilities contain "sha1" in their names, but that's historical. Clarify that object names are still to be given using whatever object format has been negotiated using the "object-format" capability. Signed-off-by: Martin Ågren <martin.agren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-08-17index-format.txt: document SHA-256 index formatLibravatar Martin Ågren1-16/+18
Document that in SHA-1 repositories, we use SHA-1 and in SHA-256 repositories, we use SHA-256, then replace all other uses of "SHA-1" with something more neutral. Avoid referring to "160-bit" hash values. Signed-off-by: Martin Ågren <martin.agren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-08-17http-protocol.txt: document SHA-256 "want"/"have" formatLibravatar Martin Ågren1-2/+3
Document that rather than always naming objects using SHA-1, we should use whatever has been negotiated using the object-format capability. Signed-off-by: Martin Ågren <martin.agren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-08-13docs: fix step in transition planLibravatar brian m. carlson1-1/+1
One of the required steps for the objectFormat extension is to implement the loose object index. However, without support for compatObjectFormat, we don't even know if the loose object index is needed, so it makes sense to move that step to the compatObjectFormat section. Do so. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-08-13docs: document SHA-256 pack and indicesLibravatar brian m. carlson1-15/+21
Now that we have SHA-256 support for packs and indices, let's document that in SHA-256 repositories, we use SHA-256 instead of SHA-1 for object names and checksums. Instead of duplicating this information throughout the document, let's just document that in SHA-1 repositories, we use SHA-1 for these purposes, and in SHA-256 repositories, we use SHA-256. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-08-13docs: commit-graph: fix some whitespace in the diagramLibravatar Johannes Berg1-3/+3
In the merge diagram, some whitespace is missing which makes it a bit confusing, fix that. Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-08-11Merge branch 'bc/sha-256-part-3'Libravatar Junio C Hamano1-1/+29
The final leg of SHA-256 transition. * bc/sha-256-part-3: (39 commits) t: remove test_oid_init in tests docs: add documentation for extensions.objectFormat ci: run tests with SHA-256 t: make SHA1 prerequisite depend on default hash t: allow testing different hash algorithms via environment t: add test_oid option to select hash algorithm repository: enable SHA-256 support by default setup: add support for reading extensions.objectformat bundle: add new version for use with SHA-256 builtin/verify-pack: implement an --object-format option http-fetch: set up git directory before parsing pack hashes t0410: mark test with SHA1 prerequisite t5308: make test work with SHA-256 t9700: make hash size independent t9500: ensure that algorithm info is preserved in config t9350: make hash size independent t9301: make hash size independent t9300: use $ZERO_OID instead of hard-coded object ID t9300: abstract away SHA-1-specific constants t8011: make hash size independent ...
2020-08-10Merge branch 'jk/strvec'Libravatar Junio C Hamano1-2/+2
The argv_array API is useful for not just managing argv but any "vector" (NULL-terminated array) of strings, and has seen adoption to a certain degree. It has been renamed to "strvec" to reduce the barrier to adoption. * jk/strvec: strvec: rename struct fields strvec: drop argv_array compatibility layer strvec: update documention to avoid argv_array strvec: fix indentation in renamed calls strvec: convert remaining callers away from argv_array name strvec: convert more callers away from argv_array name strvec: convert builtin/ callers away from argv_array name quote: rename sq_dequote_to_argv_array to mention strvec strvec: rename files from argv-array to strvec argv-array: rename to strvec argv-array: use size_t for count and alloc
2020-07-30Merge branch 'sg/commit-graph-cleanups' into masterLibravatar Junio C Hamano1-1/+1
The changed-path Bloom filter is improved using ideas from an independent implementation. * sg/commit-graph-cleanups: commit-graph: simplify write_commit_graph_file() #2 commit-graph: simplify write_commit_graph_file() #1 commit-graph: simplify parse_commit_graph() #2 commit-graph: simplify parse_commit_graph() #1 commit-graph: clean up #includes diff.h: drop diff_tree_oid() & friends' return value commit-slab: add a function to deep free entries on the slab commit-graph-format.txt: all multi-byte numbers are in network byte order commit-graph: fix parsing the Chunk Lookup table tree-walk.c: don't match submodule entries for 'submod/anything'
2020-07-30bundle: add new version for use with SHA-256Libravatar brian m. carlson1-1/+29
Currently we detect the hash algorithm in use by the length of the object ID. This is inelegant and prevents us from using a different hash algorithm that is also 256 bits in length. Since we cannot extend the v2 format in a backward-compatible way, let's add a v3 format, which is identical, except for the addition of capabilities, which are prefixed by an at sign. We add "object-format" as the only capability and reject unknown capabilities, since we do not have a network connection and therefore cannot negotiate with the other side. For compatibility, default to the v2 format for SHA-1 and require v3 for SHA-256. In t5510, always use format v3 so we can be sure we produce consistent results across hash algorithms. Since head -n N lists the top N lines instead of the Nth line, let's run our output through sed to normalize it and compare it against a fixed value, which will make sure we get exactly what we're expecting. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Reviewed-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-28strvec: update documention to avoid argv_arrayLibravatar Jeff King1-2/+2
There were a few mentions of argv_array in a non-code file which didn't get picked up in the previous commits (note that even comments in code files were already covered because of the mechanical conversion via perl). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-06Merge branch 'bc/sha-256-part-2'Libravatar Junio C Hamano2-0/+24
SHA-256 migration work continues. * bc/sha-256-part-2: (44 commits) remote-testgit: adapt for object-format bundle: detect hash algorithm when reading refs t5300: pass --object-format to git index-pack t5704: send object-format capability with SHA-256 t5703: use object-format serve option t5702: offer an object-format capability in the test t/helper: initialize the repository for test-sha1-array remote-curl: avoid truncating refs with ls-remote t1050: pass algorithm to index-pack when outside repo builtin/index-pack: add option to specify hash algorithm remote-curl: detect algorithm for dumb HTTP by size builtin/ls-remote: initialize repository based on fetch t5500: make hash independent serve: advertise object-format capability for protocol v2 connect: parse v2 refs with correct hash algorithm connect: pass full packet reader when parsing v2 refs Documentation/technical: document object-format for protocol v2 t1302: expect repo format version 1 for SHA-256 builtin/show-index: provide options to determine hash algo t5302: modernize test formatting ...
2020-06-25Merge branch 'jt/cdn-offload'Libravatar Junio C Hamano2-10/+116
The "fetch/clone" protocol has been updated to allow the server to instruct the clients to grab pre-packaged packfile(s) in addition to the packed object data coming over the wire. * jt/cdn-offload: upload-pack: fix a sparse '0 as NULL pointer' warning upload-pack: send part of packfile response as uri fetch-pack: support more than one pack lockfile upload-pack: refactor reading of pack-objects out Documentation: add Packfile URIs design doc Documentation: order protocol v2 sections http-fetch: support fetching packfiles by URL http-fetch: refactor into function http: refactor finish_http_pack_request() http: use --stdin when indexing dumb HTTP pack
2020-06-12Merge branch 'hn/refs-cleanup'Libravatar Junio C Hamano1-0/+1083
Preliminary clean-ups around refs API, plus file format specification documentation for the reftable backend. * hn/refs-cleanup: reftable: define version 2 of the spec to accomodate SHA256 reftable: clarify how empty tables should be written reftable: file format documentation refs: improve documentation for ref iterator t: use update-ref and show-ref to reading/writing refs refs.h: clarify reflog iteration order
2020-06-10Documentation: add Packfile URIs design docLibravatar Jonathan Tan2-1/+109
Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-06-10Documentation: order protocol v2 sectionsLibravatar Jonathan Tan1-10/+8
The current C Git implementation expects Git servers to follow a specific order of sections when transmitting protocol v2 responses, but this is not explicit in the documentation. Make the order explicit. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-06-09reftable: define version 2 of the spec to accomodate SHA256Libravatar Han-Wen Nienhuys1-37/+45
Version appends a hash ID to the file header, making it slightly larger. This commit also changes "SHA-1" into "object ID" in many places. Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-06-09reftable: clarify how empty tables should be writtenLibravatar Han-Wen Nienhuys1-0/+6
The format allows for some ambiguity, as a lone footer also starts with a valid file header. However, the current JGit code will barf on this. This commit codifies this behavior into the standard. Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-06-09reftable: file format documentationLibravatar Jonathan Nieder1-0/+1069
Shawn Pearce explains: Some repositories contain a lot of references (e.g. android at 866k, rails at 31k). The reftable format provides: - Near constant time lookup for any single reference, even when the repository is cold and not in process or kernel cache. - Near constant time verification if a SHA-1 is referred to by at least one reference (for allow-tip-sha1-in-want). - Efficient lookup of an entire namespace, such as `refs/tags/`. - Support atomic push `O(size_of_update)` operations. - Combine reflog storage with ref storage. This file format spec was originally written in July, 2017 by Shawn Pearce. Some refinements since then were made by Shawn and by Han-Wen Nienhuys based on experiences implementing and experimenting with the format. (All of this was in the context of our work at Google and Google is happy to contribute the result to the Git project.) Imported from JGit[1]'s current version (c217d33ff, "Documentation/technical/reftable: improve repo layout", 2020-02-04) of Documentation/technical/reftable.md and converted to asciidoc by running pandoc -t asciidoc -f markdown reftable.md >reftable.txt using pandoc 2.2.1. The result required the following additional minor changes: - removed the [TOC] directive to add a table of contents, since asciidoc does not support it - replaced git-scm.com/docs links with linkgit: directives that link to other pages within Git's documentation [1] https://eclipse.googlesource.com/jgit/jgit Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-06-08Merge branch 'dl/remote-curl-deadlock-fix'Libravatar Junio C Hamano1-0/+2
On-the-wire protocol v2 easily falls into a deadlock between the remote-curl helper and the fetch-pack process when the server side prematurely throws an error and disconnects. The communication has been updated to make it more robust. * dl/remote-curl-deadlock-fix: stateless-connect: send response end packet pkt-line: define PACKET_READ_RESPONSE_END remote-curl: error on incomplete packet pkt-line: extern packet_length() transport: extract common fetch_pack() call remote-curl: remove label indentation remote-curl: fix typo
2020-06-08commit-graph-format.txt: all multi-byte numbers are in network byte orderLibravatar SZEDER Gábor1-1/+1
The commit-graph format specifies that "All 4-byte numbers are in network order", but the commit-graph contains 8-byte integers as well (file offsets in the Chunk Lookup table), and their byte order is unspecified. Clarify that all multi-byte integers are in network byte order. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-06-02Merge branch 'jx/pkt-line-doc-count-fix'Libravatar Junio C Hamano2-4/+4
Docfix. * jx/pkt-line-doc-count-fix: doc: fix wrong 4-byte length of pkt-line message
2020-05-27Documentation/technical: document object-format for protocol v2Libravatar brian m. carlson1-0/+9
Document the object-format extension for protocol v2. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-05-24stateless-connect: send response end packetLibravatar Denton Liu1-0/+2
Currently, remote-curl acts as a proxy and blindly forwards packets between an HTTP server and fetch-pack. In the case of a stateless RPC connection where the connection is terminated before the transaction is complete, remote-curl will blindly forward the packets before waiting on more input from fetch-pack. Meanwhile, fetch-pack will read the transaction and continue reading, expecting more input to continue the transaction. This results in a deadlock between the two processes. This can be seen in the following command which does not terminate: $ git -c protocol.version=2 clone https://github.com/git/git.git --shallow-since=20151012 Cloning into 'git'... whereas the v1 version does terminate as expected: $ git -c protocol.version=1 clone https://github.com/git/git.git --shallow-since=20151012 Cloning into 'git'... fatal: the remote end hung up unexpectedly Instead of blindly forwarding packets, make remote-curl insert a response end packet after proxying the responses from the remote server when using stateless_connect(). On the RPC client side, ensure that each response ends as described. A separate control packet is chosen because we need to be able to differentiate between what the remote server sends and remote-curl's control packets. By ensuring in the remote-curl code that a server cannot send response end packets, we prevent a malicious server from being able to perform a denial of service attack in which they spoof a response end packet and cause the described deadlock to happen. Reported-by: Force Charlie <charlieio@outlook.com> Helped-by: Jeff King <peff@peff.net> Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>