summaryrefslogtreecommitdiff
path: root/Documentation/technical/protocol-v2.txt
AgeCommit message (Collapse)AuthorFilesLines
2021-09-28Merge branch 'jk/reduce-malloc-in-v2-servers'Libravatar Junio C Hamano1-1/+5
Code cleanup to limit memory consumption and tighten protocol message parsing. * jk/reduce-malloc-in-v2-servers: ls-refs: reject unknown arguments serve: reject commands used as capabilities serve: reject bogus v2 "command=ls-refs=foo" docs/protocol-v2: clarify some ls-refs ref-prefix details ls-refs: ignore very long ref-prefix counts serve: drop "keys" strvec serve: provide "receive" function for session-id capability serve: provide "receive" function for object-format capability serve: add "receive" method for v2 capabilities table serve: return capability "value" from get_capability() serve: rename is_command() to parse_command()
2021-09-23Merge branch 'jk/http-server-protocol-versions'Libravatar Junio C Hamano1-1/+7
Taking advantage of the CGI interface, http-backend has been updated to enable protocol v2 automatically when the other side asks for it. * jk/http-server-protocol-versions: docs/protocol-v2: point readers transport config discussion docs/git: discuss server-side config for GIT_PROTOCOL docs/http-backend: mention v2 protocol http-backend: handle HTTP_GIT_PROTOCOL CGI variable t5551: test v2-to-v0 http protocol fallback
2021-09-15docs/protocol-v2: clarify some ls-refs ref-prefix detailsLibravatar Jeff King1-1/+5
We've never documented the fact that a client can provide multiple ref-prefix capabilities. Let's describe the behavior. We also never discussed the "best effort" nature of the prefixes. The client side of git.git has always treated them this way, filtering the result with local patterns. And indeed any client must do this, because the prefix patterns are not sufficient to express the usual refspecs (and so for "foo" we ask for "refs/heads/foo", "refs/tags/foo", and so on). So this may be considered a change in the spec with respect to client expectations / requirements, but it's mostly codifying existing behavior. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-10docs/protocol-v2: point readers transport config discussionLibravatar Jeff King1-1/+7
We recently added tips for server admins to configure various transports to support v2's GIT_PROTOCOL variable. While the protocol-v2 document is pretty technical and not of interest to most admins, it may be a starting point for them to figure out how to turn on v2. Let's put some pointers from there to the other documentation. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-05upload-pack: document and rename --advertise-refsLibravatar Ævar Arnfjörð Bjarmason1-0/+3
The --advertise-refs documentation in git-upload-pack added in 9812f2136b3 (upload-pack.c: use parse-options API, 2016-05-31) hasn't been entirely true ever since v2 support was implemented in e52449b6722 (connect: request remote refs using v2, 2018-03-15). Under v2 we don't advertise the refs at all, but rather dump the capabilities header. This option has always been an obscure internal implementation detail, it wasn't even documented for git-receive-pack. Since it has exactly one user let's rename it to --http-backend-info-refs, which is more accurate and points the reader in the right direction. Let's also cross-link this from the protocol v1 and v2 documentation. I'm retaining a hidden --advertise-refs alias in case there's any external users of this, and making both options hidden to the bash completion (as with most other internal-only options). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-14*: fix typos which duplicate a wordLibravatar Andrei Rybak1-1/+1
Fix typos in documentation, code comments, and RelNotes which repeat various words. In trivial cases, just delete the duplicated word and rewrap text, if needed. Reword the affected sentence in Documentation/RelNotes/1.8.4.txt for it to make sense. Signed-off-by: Andrei Rybak <rybak.a.v@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-16Merge branch 'jt/push-negotiation'Libravatar Junio C Hamano1-0/+8
"git push" learns to discover common ancestor with the receiving end over protocol v2. * jt/push-negotiation: send-pack: support push negotiation fetch: teach independent negotiation (no packfile) fetch-pack: refactor command and capability write fetch-pack: refactor add_haves() fetch-pack: refactor process_acks()
2021-05-05fetch: teach independent negotiation (no packfile)Libravatar Jonathan Tan1-0/+8
Currently, the packfile negotiation step within a Git fetch cannot be done independent of sending the packfile, even though there is at least one application wherein this is useful. Therefore, make it possible for this negotiation step to be done independently. A subsequent commit will use this for one such application - push negotiation. This feature is for protocol v2 only. (An implementation for protocol v0 would require a separate implementation in the fetch, transport, and transport helper code.) In the protocol, the main hindrance towards independent negotiation is that the server can unilaterally decide to send the packfile. This is solved by a "wait-for-done" argument: the server will then wait for the client to say "done". In practice, the client will never say it; instead it will cease requests once it is satisfied. In the client, the main change lies in the transport and transport helper code. fetch_refs_via_pack() performs everything needed - protocol version and capability checks, and the negotiation itself. There are 2 code paths that do not go through fetch_refs_via_pack() that needed to be individually excluded: the bundle transport (excluded through requiring smart_options, which the bundle transport doesn't support) and transport helpers that do not support takeover. If or when we support independent negotiation for protocol v0, we will need to modify these 2 code paths to support it. But for now, report failure if independent negotiation is requested in these cases. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-20object-info: support for retrieving object infoLibravatar Bruno Albuquerque1-0/+31
Sometimes it is useful to get information of an object without having to download it completely. Add the "object-info" capability that lets the client ask for object-related information with their full hexadecimal object names. Only sizes are returned for now. Signed-off-by: Bruno Albuquerque <bga@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-02-25Merge branch 'js/doc-proto-v2-response-end'Libravatar Junio C Hamano1-2/+2
Docfix. * js/doc-proto-v2-response-end: doc: fix naming of response-end-pkt
2021-02-17doc: fix naming of response-end-pktLibravatar Joey Salazar1-2/+2
Git Protocol version 2[1] defines 0002 as a Message Packet that indicates the end of a response for stateless connections. Change the naming of the 0002 Packet to 'Response End' to match the parsing introduced in Wireshark's MR !1922 for consistency. A subsequent MR in Wireshark will address additional mismatches. [1] kernel.org/pub/software/scm/git/docs/technical/protocol-v2.html [2] gitlab.com/wireshark/wireshark/-/merge_requests/1922 Signed-off-by: Joey Salazar <jgsal@protonmail.com> Reviewed-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-02-05ls-refs: report unborn targets of symrefsLibravatar Jonathan Tan1-1/+10
When cloning, we choose the default branch based on the remote HEAD. But if there is no remote HEAD reported (which could happen if the target of the remote HEAD is unborn), we'll fall back to using our local init.defaultBranch. Traditionally this hasn't been a big deal, because most repos used "master" as the default. But these days it is likely to cause confusion if the server and client implementations choose different values (e.g., if the remote started with "main", we may choose "master" locally, create commits there, and then the user is surprised when they push to "master" and not "main"). To solve this, the remote needs to communicate the target of the HEAD symref, even if it is unborn, and "git clone" needs to use this information. Currently, symrefs that have unborn targets (such as in this case) are not communicated by the protocol. Teach Git to advertise and support the "unborn" feature in "ls-refs" (by default, this is advertised, but server administrators may turn this off through the lsrefs.unborn config). This feature indicates that "ls-refs" supports the "unborn" argument; when it is specified, "ls-refs" will send the HEAD symref with the name of its unborn target. This change is only for protocol v2. A similar change for protocol v0 would require independent protocol design (there being no analogous position to signal support for "unborn") and client-side plumbing of the data required, so the scope of this patch set is limited to protocol v2. The client side will be updated to use this in a subsequent commit. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-11-11docs: new capability to advertise session IDsLibravatar Josh Steadmon1-0/+13
In future patches, we will add the ability for Git servers and clients to advertise unique session IDs via protocol capabilities. This allows for easier debugging when both client and server logs are available. Signed-off-by: Josh Steadmon <steadmon@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-06Merge branch 'bc/sha-256-part-2'Libravatar Junio C Hamano1-0/+9
SHA-256 migration work continues. * bc/sha-256-part-2: (44 commits) remote-testgit: adapt for object-format bundle: detect hash algorithm when reading refs t5300: pass --object-format to git index-pack t5704: send object-format capability with SHA-256 t5703: use object-format serve option t5702: offer an object-format capability in the test t/helper: initialize the repository for test-sha1-array remote-curl: avoid truncating refs with ls-remote t1050: pass algorithm to index-pack when outside repo builtin/index-pack: add option to specify hash algorithm remote-curl: detect algorithm for dumb HTTP by size builtin/ls-remote: initialize repository based on fetch t5500: make hash independent serve: advertise object-format capability for protocol v2 connect: parse v2 refs with correct hash algorithm connect: pass full packet reader when parsing v2 refs Documentation/technical: document object-format for protocol v2 t1302: expect repo format version 1 for SHA-256 builtin/show-index: provide options to determine hash algo t5302: modernize test formatting ...
2020-06-25Merge branch 'jt/cdn-offload'Libravatar Junio C Hamano1-10/+38
The "fetch/clone" protocol has been updated to allow the server to instruct the clients to grab pre-packaged packfile(s) in addition to the packed object data coming over the wire. * jt/cdn-offload: upload-pack: fix a sparse '0 as NULL pointer' warning upload-pack: send part of packfile response as uri fetch-pack: support more than one pack lockfile upload-pack: refactor reading of pack-objects out Documentation: add Packfile URIs design doc Documentation: order protocol v2 sections http-fetch: support fetching packfiles by URL http-fetch: refactor into function http: refactor finish_http_pack_request() http: use --stdin when indexing dumb HTTP pack
2020-06-10Documentation: add Packfile URIs design docLibravatar Jonathan Tan1-1/+31
Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-06-10Documentation: order protocol v2 sectionsLibravatar Jonathan Tan1-10/+8
The current C Git implementation expects Git servers to follow a specific order of sections when transmitting protocol v2 responses, but this is not explicit in the documentation. Make the order explicit. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-05-27Documentation/technical: document object-format for protocol v2Libravatar brian m. carlson1-0/+9
Document the object-format extension for protocol v2. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-05-24stateless-connect: send response end packetLibravatar Denton Liu1-0/+2
Currently, remote-curl acts as a proxy and blindly forwards packets between an HTTP server and fetch-pack. In the case of a stateless RPC connection where the connection is terminated before the transaction is complete, remote-curl will blindly forward the packets before waiting on more input from fetch-pack. Meanwhile, fetch-pack will read the transaction and continue reading, expecting more input to continue the transaction. This results in a deadlock between the two processes. This can be seen in the following command which does not terminate: $ git -c protocol.version=2 clone https://github.com/git/git.git --shallow-since=20151012 Cloning into 'git'... whereas the v1 version does terminate as expected: $ git -c protocol.version=1 clone https://github.com/git/git.git --shallow-since=20151012 Cloning into 'git'... fatal: the remote end hung up unexpectedly Instead of blindly forwarding packets, make remote-curl insert a response end packet after proxying the responses from the remote server when using stateless_connect(). On the RPC client side, ensure that each response ends as described. A separate control packet is chosen because we need to be able to differentiate between what the remote server sends and remote-curl's control packets. By ensuring in the remote-curl code that a server cannot send response end packets, we prevent a malicious server from being able to perform a denial of service attack in which they spoof a response end packet and cause the described deadlock to happen. Reported-by: Force Charlie <charlieio@outlook.com> Helped-by: Jeff King <peff@peff.net> Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-07Documentation: fix a bunch of typos, both old and newLibravatar Elijah Newren1-1/+1
Reported-by: Jens Schleusener <Jens.Schleusener@fossies.org> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-08-11doc: fix repeated wordsLibravatar Mark Rushakoff1-1/+1
Inspired by 21416f0a07 ("restore: fix typo in docs", 2019-08-03), I ran "git grep -E '(\b[a-zA-Z]+) \1\b' -- Documentation/" to find other cases where words were duplicated, e.g. "the the", and in most cases removed one of the repeated words. There were many false positives by this grep command, including deliberate repeated words like "really really" or valid uses of "that that" which I left alone, of course. I also did not correct any of the legitimate, accidentally repeated words in old RelNotes. Signed-off-by: Mark Rushakoff <mark.rushakoff@gmail.com> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-01Documentation/technical/protocol-v2.txt: fix formattingLibravatar SZEDER Gábor1-26/+26
Asciidoctor versions v1.5.7 or later print the following warning while building the documentation: ASCIIDOC technical/protocol-v2.html asciidoctor: WARNING: protocol-v2.txt: line 38: unterminated listing block This highlights an issue (even with older Asciidoctor versions) where the 'Initial Client Request' header is not rendered as a header but in monospace. I'm not sure what exactly causes this issue and why it's an issue only with this particular header, but all headers in 'protocol-v2.txt' are written like this: Initial Client Request ------------------------ i.e. the header itself is indented by a space, and the "underline" is two characters longer than the header. Dropping that indentation and making the length of the underline match the length of the header apparently fixes this issue. While at it, adjust all other headers 'protocol-v2.txt' as well, to match the style we use everywhere else. The page rendered with AsciiDoc doesn't have this formatting issue. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-05Merge branch 'jt/fetch-v2-sideband'Libravatar Junio C Hamano1-0/+10
"git fetch" and "git upload-pack" learned to send all exchange over the sideband channel while talking the v2 protocol. * jt/fetch-v2-sideband: tests: define GIT_TEST_SIDEBAND_ALL {fetch,upload}-pack: sideband v2 fetch response sideband: reverse its dependency on pkt-line pkt-line: introduce struct packet_writer pack-protocol.txt: accept error packets in any context Use packet_reader instead of packet_read_line
2019-01-17{fetch,upload}-pack: sideband v2 fetch responseLibravatar Jonathan Tan1-0/+10
Currently, a response to a fetch request has sideband support only while the packfile is being sent, meaning that the server cannot send notices until the start of the packfile. Extend sideband support in protocol v2 fetch responses to the whole response. upload-pack will advertise it if the uploadpack.allowsidebandall configuration variable is set, and fetch-pack will automatically request it if advertised. If the sideband is to be used throughout the whole response, upload-pack will use it to send errors instead of prefixing a PKT-LINE payload with "ERR ". This will be tested in a subsequent patch. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-15filter-options: expand scaled numbersLibravatar Josh Steadmon1-1/+7
When communicating with a remote server or a subprocess, use expanded numbers rather than numbers with scaling suffix in the object filter spec (e.g. "limit:blob=1k" becomes "limit:blob=1024"). Update the protocol docs to note that clients should always perform this expansion, to allow for more compatibility between server implementations. Signed-off-by: Josh Steadmon <steadmon@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-24Merge branch 'jt/connectivity-check-after-unshallow'Libravatar Junio C Hamano1-1/+27
"git fetch" failed to correctly validate the set of objects it received when making a shallow history deeper, which has been corrected. * jt/connectivity-check-after-unshallow: fetch-pack: write shallow, then check connectivity fetch-pack: implement ref-in-want fetch-pack: put shallow info in output parameter fetch: refactor to make function args narrower fetch: refactor fetch_refs into two functions fetch: refactor the population of peer ref OIDs upload-pack: test negotiation with changing repository upload-pack: implement ref-in-want test-pkt-line: add unpack-sideband subcommand
2018-07-18Merge branch 'bw/protocol-v2'Libravatar Junio C Hamano1-2/+1
Doc fix. * bw/protocol-v2: protocol-v2 doc: put HTTP headers after request
2018-06-28upload-pack: implement ref-in-wantLibravatar Brandon Williams1-1/+27
Currently, while performing packfile negotiation, clients are only allowed to specify their desired objects using object ids. This causes a vulnerability to failure when an object turns non-existent during negotiation, which may happen if, for example, the desired repository is provided by multiple Git servers in a load-balancing arrangement and there exists replication delay. In order to eliminate this vulnerability, implement the ref-in-want feature for the 'fetch' command in protocol version 2. This feature enables the 'fetch' command to support requests in the form of ref names through a new "want-ref <ref>" parameter. At the conclusion of negotiation, the server will send a list of all of the wanted references (as provided by "want-ref" lines) in addition to the generated packfile. Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-06-22protocol-v2 doc: put HTTP headers after requestLibravatar Josh Steadmon1-2/+1
HTTP servers return 400 if you send headers before the GET request. Signed-off-by: Josh Steadmon <steadmon@google.com> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-05-30Merge branch 'jt/partial-clone-proto-v2'Libravatar Junio C Hamano1-0/+9
Transfer protocol v2 learned to support the partial clone. * jt/partial-clone-proto-v2: {fetch,upload}-pack: support filter in protocol v2 upload-pack: read config when serving protocol v2 upload-pack: fix error message typo
2018-05-06{fetch,upload}-pack: support filter in protocol v2Libravatar Jonathan Tan1-0/+9
The fetch-pack/upload-pack protocol v2 was developed independently of the filter parameter (used in partial fetches), thus it did not include support for it. Add support for the filter parameter. Like in the legacy protocol, the server advertises and supports "filter" only if uploadpack.allowfilter is configured. Like in the legacy protocol, the client continues with a warning if "--filter" is specified, but the server does not advertise it. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-04-24serve: introduce the server-option capabilityLibravatar Brandon Williams1-0/+10
Introduce the "server-option" capability to protocol version 2. This enables future clients the ability to send server specific options in command requests when using protocol version 2. Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-15fetch-pack: support shallow requestsLibravatar Brandon Williams1-7/+11
Enable shallow clones and deepen requests using protocol version 2 if the server 'fetch' command supports the 'shallow' feature. Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-15fetch-pack: perform a fetch using v2Libravatar Brandon Williams1-1/+66
When communicating with a v2 server, perform a fetch by requesting the 'fetch' command. Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-15upload-pack: introduce fetch server commandLibravatar Brandon Williams1-0/+125
Introduce the 'fetch' server command. Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-15ls-refs: introduce ls-refs server commandLibravatar Brandon Williams1-0/+31
Introduce the ls-refs server command. In protocol v2, the ls-refs command is used to request the ref advertisement from the server. Since it is a command which can be requested (as opposed to mandatory in v1), a client can sent a number of parameters in its request to limit the ref advertisement based on provided ref-prefixes. Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-15serve: introduce git-serveLibravatar Brandon Williams1-0/+170
Introduce git-serve, the base server for protocol version 2. Protocol version 2 is intended to be a replacement for Git's current wire protocol. The intention is that it will be a simpler, less wasteful protocol which can evolve over time. Protocol version 2 improves upon version 1 by eliminating the initial ref advertisement. In its place a server will export a list of capabilities and commands which it supports in a capability advertisement. A client can then request that a particular command be executed by providing a number of capabilities and command specific parameters. At the completion of a command, a client can request that another command be executed or can terminate the connection by sending a flush packet. Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>