summaryrefslogtreecommitdiff
path: root/t/t5551-http-fetch-smart.sh
AgeCommit message (Collapse)AuthorFilesLines
2018-01-19http: support omitting data from tracesLibravatar Jonathan Tan1-0/+12
GIT_TRACE_CURL provides a way to debug what is being sent and received over HTTP, with automatic redaction of sensitive information. But it also logs data transmissions, which significantly increases the log file size, sometimes unnecessarily. Add an option "GIT_TRACE_CURL_NO_DATA" to allow the user to omit such data transmissions. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-01-19http: support cookie redaction when tracingLibravatar Jonathan Tan1-0/+21
When using GIT_TRACE_CURL, Git already redacts the "Authorization:" and "Proxy-Authorization:" HTTP headers. Extend this redaction to a user-specified list of cookies, specified through the "GIT_REDACT_COOKIES" environment variable. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-10Merge branch 'dt/smart-http-detect-server-going-away'Libravatar Junio C Hamano1-0/+52
When the http server gives an incomplete response to a smart-http rpc call, it could lead to client waiting for a full response that will never come. Teach the client side to notice this condition and abort the transfer. An improvement counterproposal has failed. cf. <20161114194049.mktpsvgdhex2f4zv@sigill.intra.peff.net> * dt/smart-http-detect-server-going-away: upload-pack: optionally allow fetching any sha1 remote-curl: don't hang when a server dies before any output
2016-12-19Merge branch 'jk/http-walker-limit-redirect-2.9'Libravatar Junio C Hamano1-0/+4
Transport with dumb http can be fooled into following foreign URLs that the end user does not intend to, especially with the server side redirects and http-alternates mechanism, which can lead to security issues. Tighten the redirection and make it more obvious to the end user when it happens. * jk/http-walker-limit-redirect-2.9: http: treat http-alternates like redirects http: make redirects more obvious remote-curl: rename shadowed options variable http: always update the base URL for redirects http: simplify update_url_from_redirect
2016-12-06http: always update the base URL for redirectsLibravatar Jeff King1-0/+4
If a malicious server redirects the initial ref advertisement, it may be able to leak sha1s from other, unrelated servers that the client has access to. For example, imagine that Alice is a git user, she has access to a private repository on a server hosted by Bob, and Mallory runs a malicious server and wants to find out about Bob's private repository. Mallory asks Alice to clone an unrelated repository from her over HTTP. When Alice's client contacts Mallory's server for the initial ref advertisement, the server issues an HTTP redirect for Bob's server. Alice contacts Bob's server and gets the ref advertisement for the private repository. If there is anything to fetch, she then follows up by asking the server for one or more sha1 objects. But who is the server? If it is still Mallory's server, then Alice will leak the existence of those sha1s to her. Since commit c93c92f30 (http: update base URLs when we see redirects, 2013-09-28), the client usually rewrites the base URL such that all further requests will go to Bob's server. But this is done by textually matching the URL. If we were originally looking for "http://mallory/repo.git/info/refs", and we got pointed at "http://bob/other.git/info/refs", then we know that the right root is "http://bob/other.git". If the redirect appears to change more than just the root, we punt and continue to use the original server. E.g., imagine the redirect adds a URL component that Bob's server will ignore, like "http://bob/other.git/info/refs?dummy=1". We can solve this by aborting in this case rather than silently continuing to use Mallory's server. In addition to protecting from sha1 leakage, it's arguably safer and more sane to refuse a confusing redirect like that in general. For example, part of the motivation in c93c92f30 is avoiding accidentally sending credentials over clear http, just to get a response that says "try again over https". So even in a non-malicious case, we'd prefer to err on the side of caution. The downside is that it's possible this will break a legitimate but complicated server-side redirection scheme. The setup given in the newly added test does work, but it's convoluted enough that we don't need to care about it. A more plausible case would be a server which redirects a request for "info/refs?service=git-upload-pack" to just "info/refs" (because it does not do smart HTTP, and for some reason really dislikes query parameters). Right now we would transparently downgrade to dumb-http, but with this patch, we'd complain (and the user would have to set GIT_SMART_HTTP=0 to fetch). Reported-by: Jann Horn <jannh@google.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-11-18upload-pack: optionally allow fetching any sha1Libravatar David Turner1-0/+22
It seems a little silly to do a reachabilty check in the case where we trust the user to access absolutely everything in the repository. Also, it's racy in a distributed system -- perhaps one server advertises a ref, but another has since had a force-push to that ref, and perhaps the two HTTP requests end up directed to these different servers. Signed-off-by: David Turner <dturner@twosigma.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-11-18remote-curl: don't hang when a server dies before any outputLibravatar David Turner1-0/+30
In the event that a HTTP server closes the connection after giving a 200 but before giving any packets, we don't want to hang forever waiting for a response that will never come. Instead, we should die immediately. One case where this happens is when attempting to fetch a dangling object by its object name. In this case, the server dies before sending any data. Prior to this patch, fetch-pack would wait for data from the server, and remote-curl would wait for fetch-pack, causing a deadlock. Despite this patch, there is other possible malformed input that could cause the same deadlock (e.g. a half-finished pktline, or a pktline but no trailing flush). There are a few possible solutions to this: 1. Allowing remote-curl to tell fetch-pack about the EOF (so that fetch-pack could know that no more data is coming until it says something else). This is tricky because an out-of-band signal would be required, or the http response would have to be re-framed inside another layer of pkt-line or something. 2. Make remote-curl understand some of the protocol. It turns out that in addition to understanding pkt-line, it would need to watch for ack/nak. This is somewhat fragile, as information about the protocol would end up in two places. Also, pkt-lines which are already at the length limit would need special handling. Both of these solutions would require a fair amount of work, whereas this hack is easy and solves at least some of the problem. Still to do: it would be good to give a better error message than "fatal: The remote end hung up unexpectedly". Signed-off-by: David Turner <dturner@twosigma.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-07t5551-http-fetch-smart.sh: use the GIT_TRACE_CURL environment varLibravatar Elia Pinto1-3/+12
Use the new GIT_TRACE_CURL environment variable instead of the deprecated GIT_CURL_VERBOSE. Signed-off-by: Elia Pinto <gitter.spiros@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-05-10submodule: ensure that -c http.extraheader is heededLibravatar Johannes Schindelin1-1/+10
To support this developer's use case of allowing build agents token-based access to private repositories, we introduced the http.extraheader feature, allowing extra HTTP headers to be sent along with every HTTP request. This patch verifies that we can configure these extra HTTP headers via the command-line for use with `git submodule update`, too. Example: git -c http.extraheader="Secret: Sauce" submodule update --init Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-05-10t5551: make the test for extra HTTP headers more robustLibravatar Johannes Schindelin1-1/+2
To test that extra HTTP headers are passed correctly, t5551 verifies that a fetch succeeds when two required headers are passed, and that the fetch does not succeed when those headers are not passed. However, this test would also succeed if the configuration required only one header. As Apache's configuration is notoriously tricky (this developer frequently requires StackOverflow's help to understand Apache's documentation), especially when still supporting the 2.2 line, let's just really make sure that the test verifies what we want it to verify. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Reviewed-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-04-27http: support sending custom HTTP headersLibravatar Johannes Schindelin1-0/+7
We introduce a way to send custom HTTP headers with all requests. This allows us, for example, to send an extra token from build agents for temporary access to private repositories. (This is the use case that triggered this patch.) This feature can be used like this: git -c http.extraheader='Secret: sssh!' fetch $URL $REF Note that `curl_easy_setopt(..., CURLOPT_HTTPHEADER, ...)` takes only a single list, overriding any previous call. This means we have to collect _all_ of the headers we want to use into a single list, and feed it to cURL in one shot. Since we already unconditionally set a "pragma" header when initializing the curl handles, we can add our new headers to that list. For callers which override the default header list (like probe_rpc), we provide `http_copy_default_headers()` so they can do the same trick. Big thanks to Jeff King and Junio Hamano for their outstanding help and patient reviews. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Reviewed-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-06-01Merge branch 'jk/http-backend-deadlock'Libravatar Junio C Hamano1-13/+36
Communication between the HTTP server and http_backend process can lead to a dead-lock when relaying a large ref negotiation request. Diagnose the situation better, and mitigate it by reading such a request first into core (to a reasonable limit). * jk/http-backend-deadlock: http-backend: spool ref negotiation requests to buffer t5551: factor out tag creation http-backend: fix die recursion with custom handler
2015-05-25Merge branch 'jk/http-backend-deadlock-2.3' into jk/http-backend-deadlockLibravatar Junio C Hamano1-13/+36
* jk/http-backend-deadlock-2.3: http-backend: spool ref negotiation requests to buffer t5551: factor out tag creation http-backend: fix die recursion with custom handler
2015-05-25Merge branch 'jk/http-backend-deadlock-2.2' into jk/http-backend-deadlock-2.3Libravatar Junio C Hamano1-13/+36
* jk/http-backend-deadlock-2.2: http-backend: spool ref negotiation requests to buffer t5551: factor out tag creation http-backend: fix die recursion with custom handler
2015-05-25http-backend: spool ref negotiation requests to bufferLibravatar Jeff King1-0/+11
When http-backend spawns "upload-pack" to do ref negotiation, it streams the http request body to upload-pack, who then streams the http response back to the client as it reads. In theory, git can go full-duplex; the client can consume our response while it is still sending the request. In practice, however, HTTP is a half-duplex protocol. Even if our client is ready to read and write simultaneously, we may have other HTTP infrastructure in the way, including the webserver that spawns our CGI, or any intermediate proxies. In at least one documented case[1], this leads to deadlock when trying a fetch over http. What happens is basically: 1. Apache proxies the request to the CGI, http-backend. 2. http-backend gzip-inflates the data and sends the result to upload-pack. 3. upload-pack acts on the data and generates output over the pipe back to Apache. Apache isn't reading because it's busy writing (step 1). This works fine most of the time, because the upload-pack output ends up in a system pipe buffer, and Apache reads it as soon as it finishes writing. But if both the request and the response exceed the system pipe buffer size, then we deadlock (Apache blocks writing to http-backend, http-backend blocks writing to upload-pack, and upload-pack blocks writing to Apache). We need to break the deadlock by spooling either the input or the output. In this case, it's ideal to spool the input, because Apache does not start reading either stdout _or_ stderr until we have consumed all of the input. So until we do so, we cannot even get an error message out to the client. The solution is fairly straight-forward: we read the request body into an in-memory buffer in http-backend, freeing up Apache, and then feed the data ourselves to upload-pack. But there are a few important things to note: 1. We limit the in-memory buffer to prevent an obvious denial-of-service attack. This is a new hard limit on requests, but it's unlikely to come into play. The default value is 10MB, which covers even the ridiculous 100,000-ref negotation in the included test (that actually caps out just over 5MB). But it's configurable on the off chance that you don't mind spending some extra memory to make even ridiculous requests work. 2. We must take care only to buffer when we have to. For pushes, the incoming packfile may be of arbitrary size, and we should connect the input directly to receive-pack. There's no deadlock problem here, though, because we do not produce any output until the whole packfile has been read. For upload-pack's initial ref advertisement, we similarly do not need to buffer. Even though we may generate a lot of output, there is no request body at all (i.e., it is a GET, not a POST). [1] http://article.gmane.org/gmane.comp.version-control.git/269020 Test-adapted-from: Dennis Kaarsemaker <dennis@kaarsemaker.net> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-05-22Merge branch 'jk/skip-http-tests-under-no-curl'Libravatar Junio C Hamano1-6/+0
Test clean-up. * jk/skip-http-tests-under-no-curl: tests: skip dav http-push tests under NO_EXPAT=NoThanks t/lib-httpd.sh: skip tests if NO_CURL is defined
2015-05-20t5551: factor out tag creationLibravatar Jeff King1-13/+21
One of our tests in t5551 creates a large number of tags, and jumps through some hoops to do it efficiently. Let's factor that out into a function so we can make other similar tests. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-05-07t/lib-httpd.sh: skip tests if NO_CURL is definedLibravatar Jeff King1-6/+0
If we built git without curl, we can't actually test against an http server. In fact, all of the test scripts which include lib-httpd.sh already perform this check, with one exception: t5540. For those scripts, this is a noop, and for t5540, this is a bugfix (it used to fail when built with NO_CURL, though it could go unnoticed if you had a stale git-remote-https in your build directory). Noticed-by: Junio C Hamano <junio@pobox.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-26Merge branch 'jk/test-chain-lint'Libravatar Junio C Hamano1-3/+3
People often forget to chain the commands in their test together with &&, leaving a failure from an earlier command in the test go unnoticed. The new GIT_TEST_CHAIN_LINT mechanism allows you to catch such a mistake more easily. * jk/test-chain-lint: (36 commits) t9001: drop save_confirm helper t0020: use test_* helpers instead of hand-rolled messages t: simplify loop exit-code status variables t: fix some trivial cases of ignored exit codes in loops t7701: fix ignored exit code inside loop t3305: fix ignored exit code inside loop t0020: fix ignored exit code inside loops perf-lib: fix ignored exit code inside loop t6039: fix broken && chain t9158, t9161: fix broken &&-chain in git-svn tests t9104: fix test for following larger parents t4104: drop hand-rolled error reporting t0005: fix broken &&-chains t7004: fix embedded single-quotes t0050: appease --chain-lint t9001: use test_when_finished t4117: use modern test_* helpers t6034: use modern test_* helpers t1301: use modern test_* helpers t0020: use modern test_* helpers ...
2015-03-23Merge branch 'jk/test-annoyances'Libravatar Junio C Hamano1-5/+14
Test fixes. * jk/test-annoyances: t5551: make EXPENSIVE test cheaper t5541: move run_with_cmdline_limit to test-lib.sh t: pass GIT_TRACE through Apache t: redirect stderr GIT_TRACE to descriptor 4 t: translate SIGINT to an exit
2015-03-20t: fix trivial &&-chain breakageLibravatar Jeff King1-1/+1
These are tests which are missing a link in their &&-chain, but during a setup phase. We may fail to notice failure in commands that build the test environment, but these are typically not expected to fail at all (but it's still good to double-check that our test environment is what we expect). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-20t: fix moderate &&-chain breakageLibravatar Jeff King1-1/+1
These are tests which are missing a link in their &&-chain, but in a way that probably does not effect the outcome of the test. Most of these are of the form: some_cmd >actual test_cmp expect actual The main point of the test is to verify the output, and a failure in some_cmd would probably be noticed by bogus output. But it is good for the tests to also confirm that "some_cmd" does not die unexpectedly after producing its output. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-20t: fix severe &&-chain breakageLibravatar Jeff King1-1/+1
These are tests which are missing a link in their &&-chain, in a location which causes a significant portion of the test to be missed (e.g., the test effectively does nothing, or consists of a long string of actions and output comparisons, and we throw away the exit code of at least one part of the string). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-12t5551: make EXPENSIVE test cheaperLibravatar Jeff King1-5/+14
We create 50,000 tags to check that we don't overflow the command-line of fetch-pack. But by using run_with_cmdline_limit, we can get the same effect with a much smaller number of tags. This makes the test fast enough that we can drop the EXPENSIVE prereq, which means people will actually run it. It was not documented to do so, but this test was also the only test of a clone-over-http that requires multiple POSTs during the conversation. We can continue to test that by dropping http.postbuffer to its minimum size, and checking that we get two POSTs. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-12upload-pack: fix transfer.hiderefs over smart-httpLibravatar Jeff King1-0/+11
When upload-pack advertises the refs (either for a normal, non-stateless request, or for the initial contact in a stateless one), we call for_each_ref with the send_ref function as its callback. send_ref, in turn, calls mark_our_ref, which checks whether the ref is hidden, and sets OUR_REF or HIDDEN_REF on the object as appropriate. If it is hidden, mark_our_ref also returns "1" to signal send_ref that the ref should not be advertised. If we are not advertising refs, (i.e., the follow-up invocation by an http client to send its "want" lines), we use mark_our_ref directly as a callback to for_each_ref. Its marking does the right thing, but when it then returns "1" to for_each_ref, the latter interprets this as an error and stops iterating. As a result, we skip marking all of the refs that come lexicographically after it. Any "want" lines from the client asking for those objects will fail, as they were not properly marked with OUR_REF. To solve this, we introduce a wrapper callback around mark_our_ref which always returns 0 (even if the ref is hidden, we want to keep iterating). We also tweak the signature of mark_our_ref to exclude unnecessary parameters that were present only to conform to the callback interface. This should make it less likely for somebody to accidentally use it as a callback in the future. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-20Merge branch 'jc/test-lazy-prereq' (early part)Libravatar Junio C Hamano1-2/+0
* 'jc/test-lazy-prereq' (early part): t3419: drop unnecessary NOT_EXPENSIVE pseudo-prerequisite t3302: drop unnecessary NOT_EXPENSIVE pseudo-prerequisite t3302: do not chdir around in the primary test process t3302: coding style updates test: turn USR_BIN_TIME into a lazy prerequisite test: turn EXPENSIVE into a lazy prerequisite
2014-06-20Merge branch 'tb/t5551-clone-notice-to-stderr'Libravatar Junio C Hamano1-2/+1
* tb/t5551-clone-notice-to-stderr: t5551: fix the 50,000 tag test
2014-02-10test: rename http fetch and push test filesLibravatar Nguyễn Thái Ngọc Duy1-0/+252
Make clear which one is for dumb protocol, which one is for smart from their file name. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>