src/transports/http.c


Log

Author Commit Date CI Message
Patrick Steinhardt 5265b31c 2019-01-23T15:00:20 streams: fix callers potentially only writing partial data Similar to the write(3) function, implementations of `git_stream_write` do not guarantee that all bytes are written. Instead, they return the number of bytes that actually have been written, which may be smaller than the total number of bytes. Furthermore, due to an interface design issue, we cannot ever write more than `SSIZE_MAX` bytes at once, as otherwise we cannot represent the number of bytes written to the caller. Unfortunately, no caller of `git_stream_write` ever checks the return value, except to verify that no error occurred. Due to this, they are susceptible to the case where only partial data has been written. Fix this by introducing a new function `git_stream__write_full`. In contrast to `git_stream_write`, it will always return either success or failure, without returning the number of bytes written. Thus, it is able to write all `SIZE_MAX` bytes and loop around `git_stream_write` until all data has been written. Adjust all callers except the BIO callbacks in our mbedtls and OpenSSL streams, which already do the right thing and require the amount of bytes written.
Edward Thomson f673e232 2018-12-27T13:47:34 git_error: use new names in internal APIs and usage Move to the `git_error` name in the internal API for error-related functions.
Jason Haslam 35d86c77 2019-01-14T10:14:36 proxy: fix crash on remote connection with GIT_PROXY_AUTO but no proxy is detected
Edward Thomson 30ac46aa 2018-11-28T10:12:43 http: reset replay_count upon connection Reset the replay_count upon a successful connection. It's possible that we could encounter a situation where we connect successfully but need to replay a request - for example, a connection and initial request succeeds without authentication but a subsequent call does require authentication. Reset the replay count upon any successful request to afford subsequent replays room to manuever.
Edward Thomson 52478d7d 2018-11-18T19:54:49 http: don't allow SSL connections to a proxy Temporarily disallow SSL connections to a proxy until we can understand the valgrind warnings when tunneling OpenSSL over OpenSSL.
Edward Thomson 41f620d9 2018-11-18T19:10:50 http: only load proxy configuration during connection Only load the proxy configuration during connection; we need this data when we're going to connect to the server, however we may mutate it after connection (connecting through a CONNECT proxy means that we should send requests like normal). If we reload the proxy configuration but do not actually reconnect (because we're in a keep-alive session) then we will reload the proxy configuration that we should have mutated. Thus, only load the proxy configuration when we know that we're going to reconnect.
Edward Thomson 0467606f 2018-11-18T11:00:11 http: disallow repeated headers from servers Don't allow servers to send us multiple Content-Type, Content-Length or Location headers.
Edward Thomson 21142c5a 2018-10-29T10:04:48 http: remove cURL We previously used cURL to support HTTP proxies. Now that we've added this support natively, we can remove the curl dependency.
Edward Thomson 5d4e1e04 2018-10-28T21:27:56 http: use CONNECT to talk to proxies Natively support HTTPS connections through proxies by speaking CONNECT to the proxy and then adding a TLS connection on top of the socket.
Edward Thomson 43b592ac 2018-10-25T08:49:01 tls: introduce a wrap function Introduce `git_tls_stream_wrap` which will take an existing `stream` with an already connected socket and begin speaking TLS on top of it. This is useful if you've built a connection to a proxy server and you wish to begin CONNECT over it to tunnel a TLS connection. Also update the pluggable TLS stream layer so that it can accept a registration structure that provides an `init` and `wrap` function, instead of a single initialization function.
Edward Thomson b2ed778a 2018-11-18T22:20:10 http transport: reset error message on cert failure Store the error message from the underlying TLS library before calling the certificate callback. If it refuses to act (demonstrated by returning GIT_PASSTHROUGH) then restore the error message. Otherwise, if the callback does not set an error message, set a sensible default that implicates the callback itself.
Edward Thomson 2ce2315c 2018-10-22T17:33:45 http transport: support cert check for proxies Refactor certificate checking so that it can easily be called for proxies or the remote server.
Edward Thomson 74c6e08e 2018-10-22T14:56:53 http transport: provide proxy credentials
Edward Thomson 496da38c 2018-10-22T12:48:45 http transport: refactor storage Create a simple data structure that contains information about the server being connected to, whether that's the actual remote endpoint (git server) or an intermediate proxy. This allows for organization of streams, authentication state, etc.
Edward Thomson 6af8572c 2018-10-22T11:29:01 http transport: cap number of authentication replays Put a limit on the number of authentication replays in the HTTP transport. Standardize on 7 replays for authentication or redirects, which matches the behavior of the WinHTTP transport.
Edward Thomson 22654812 2018-10-22T11:24:05 http transport: prompt for proxy credentials Teach the HTTP transport how to prompt for proxy credentials.
Edward Thomson 0328eef6 2018-10-22T11:14:06 http transport: further refactor credential handling Prepare credential handling to understand both git server and proxy server authentication.
Edward Thomson 32cb56ce 2018-10-22T10:16:54 http transport: refactor credential handling Factor credential handling into its own function. Additionally, add safety checks to ensure that we are in a valid state - that we have received a valid challenge from the server and that we have configuration to respond to that challenge.
Edward Thomson e6e399ab 2018-10-22T09:49:54 http transport: use HTTP proxies when requested The HTTP transport should understand how to apply proxies when configured with `GIT_PROXY_SPECIFIED` and `GIT_PROXY_SPECIFIED`. When a proxy is configured, the HTTP transport will now connect to the proxy (instead of directly to the git server), and will request the properly-formed URL of the git server endpoint.
Edward Thomson e6f1931a 2018-10-22T00:09:24 http: rename http subtransport's `io` to `gitserver_stream` Rename `http_subtransport->io` to `http_subtransport->gitserver_stream` to clarify its use, especially as we might have additional streams (eg for a proxy) in the future.
Edward Thomson c07ff4cb 2018-10-21T14:17:06 http: rename `connection_data` -> `gitserver_data` Rename the `connection_data` struct member to `gitserver_data`, to disambiguate future `connection_data`s that apply to the proxy, not the final server endpoint.
Edward Thomson ed72465e 2018-10-13T19:16:54 proxy: propagate proxy configuration errors
Patrick Steinhardt c97d302d 2018-11-28T13:45:41 Merge pull request #4879 from libgit2/ethomson/defer_cert_cred_cb Allow certificate and credential callbacks to decline to act
Edward Thomson a2e6e0ea 2018-11-06T14:15:43 transport: allow cred/cert callbacks to return GIT_PASSTHROUGH Allow credential and certificate checking callbacks to return GIT_PASSTHROUGH, indicating that they do not want to act. Introduce this to support in both the http and ssh callbacks. Additionally, enable the same mechanism for certificate validation. This is most useful to disambiguate any meaning in the publicly exposed credential and certificate functions (`git_transport_smart_credentials` and `git_transport_smart_certificate_check`) but it may be more generally useful for callers to be able to defer back to libgit2.
Rick Altherr 83b35181 2018-10-19T10:54:38 transport/http: Include non-default ports in Host header When the port is omitted, the server assumes the default port for the service is used (see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Host). In cases where the client provided a non-default port, it should be passed along. This hasn't been an issue so far as the git protocol doesn't include server-generated URIs. I encountered this when implementing Rust registry support for Sonatype Nexus. Rust's registry uses a git repository for the package index. Clients look at a file in the root of the package index to find the base URL for downloading the packages. Sonatype Nexus looks at the incoming HTTP request (Host header and URL) to determine the client-facing URL base as it may be running behind a load balancer or reverse proxy. This client-facing URL base is then used to construct the package download base URL. When libgit2 fetches the index from Nexus on a non-default port, Nexus trusts the incorrect Host header and generates an incorrect package download base URL.
Carlos Martín Nieto 9b6e4081 2018-10-15T17:08:38 Merge commit 'afd10f0' (Follow 308 redirects)
Zander Brown afd10f0b 2018-10-13T09:31:20 Follow 308 redirects (as used by GitLab)
Anders Borum 475db39b 2018-10-06T12:58:06 ignore unsupported http authentication schemes auth_context_match returns 0 instead of -1 for unknown schemes to not fail in situations where some authentication schemes are supported and others are not. apply_credentials is adjusted to handle auth_context_match returning 0 without producing authentication context.
Etienne Samson 1c949ce1 2018-08-21T02:11:32 transport/http: do not return success if we failed to get a scheme Otherwise we return a NULL context, which will get dereferenced in apply_credentials.
Patrick Steinhardt ecf4f33a 2018-02-08T11:14:48 Convert usage of `git_buf_free` to new `git_buf_dispose`
Edward Thomson ee6be190 2018-01-31T08:36:19 http: standardize user-agent addition The winhttp and posix http each need to add the user-agent to their requests. Standardize on a single function to include this so that we do not get the version numbers we're sending out of sync. Assemble the complete user agent in `git_http__user_agent`, returning assembled strings. Co-authored-by: Patrick Steinhardt <ps@pks.im>
Etienne Samson e9369856 2017-03-21T00:25:15 stream: Gather streams to src/streams
Patrick Steinhardt 0c7f49dd 2017-06-30T13:39:01 Make sure to always include "common.h" first Next to including several files, our "common.h" header also declares various macros which are then used throughout the project. As such, we have to make sure to always include this file first in all implementation files. Otherwise, we might encounter problems or even silent behavioural differences due to macros or defines not being defined as they should be. So in fact, our header and implementation files should make sure to always include "common.h" first. This commit does so by establishing a common include pattern. Header files inside of "src" will now always include "common.h" as its first other file, separated by a newline from all the other includes to make it stand out as special. There are two cases for the implementation files. If they do have a matching header file, they will always include this one first, leading to "common.h" being transitively included as first file. If they do not have a matching header file, they instead include "common.h" as first file themselves. This fixes the outlined problems and will become our standard practice for header and source files inside of the "src/" from now on.
tyler wanek 39e76bb3 2017-01-27T16:05:20 Do not discard proxy_options that have been set when auto is specified
Carlos Martín Nieto a6d833a2 2017-01-13T17:05:58 Merge pull request #4049 from libgit2/ethomson/error_msgs giterr_set: consistent error messages
Etienne Samson 9a64e62f 2016-12-21T21:24:33 http: check certificate validity before clobbering the error variable
Edward Thomson 909d5494 2016-12-29T12:25:15 giterr_set: consistent error messages Error messages should be sentence fragments, and therefore: 1. Should not begin with a capital letter, 2. Should not conclude with punctuation, and 3. Should not end a sentence and begin a new one
Carlos Martín Nieto fafafb1f 2016-12-20T16:19:30 http: bump the pretend git version in the User-Agent We want to keep the git UA in order for services to recognise that we're a Git client and not a browser. But in order to stop dumb HTTP some services have blocked UAs that claim to be pre-1.6.6 git. Thread these needles by using the "git/2.0" prefix which is still close enough to git's yet distinct enough that you can tell it's us.
Patrick Steinhardt b1453601 2016-08-17T11:38:26 transports: http: reset `connected` flag when closing transport
Patrick Steinhardt c4cba4e9 2016-08-17T11:00:05 transports: http: reset `connected` flag when re-connecting transport When calling `http_connect` on a subtransport whose stream is already connected, we first close the stream in case no keep-alive is in use. When doing so, we do not reset the transport's connection state, though. Usually, this will do no harm in case the subsequent connect will succeed. But when the connection fails we are left with a substransport which is tagged as connected but which has no valid stream attached. Fix the issue by resetting the subtransport's connected-state when closing its stream in `http_connect`.
wildart bdec62dc 2016-07-06T13:06:25 remove conditions that prevent use of custom TLS stream
David Brooks 6c9eb86f 2016-06-19T11:46:43 HTTP authentication scheme name is case insensitive.
Carlos Martín Nieto 0d72f67f 2016-03-14T17:36:04 proxy: don't specify the protocol in the type We leave this up to the scheme in the url field. The type should only tell us about whether we want a proxy and whether we want to auto-detect it.
Carlos Martín Nieto b373e9a6 2015-09-21T22:38:50 net: use proxy options struct in the stream config
Carlos Martín Nieto 94bac76c 2015-10-02T03:46:34 http: use a custom user-agent if the user has set it We still prefix it with "git/1.0" since that's required in many situations, but we replace the area which mentions libgit2.
Matt Burke d7375662 2015-09-25T10:16:16 Copy custom_headers insteach of referencing the caller's copy
Matt Burke 35969c68 2015-09-10T08:58:23 Ignore NULL headers
Matt Burke 276f6aa0 2015-09-08T14:00:37 Hook up the custom_headers to the http transport
Matt Burke 24f5b4e1 2015-09-08T13:34:42 Drop extra_http_headers from git_remote
Matt Burke 59d6128e 2015-09-04T09:36:50 Allow the world to set HTTP headers for remotes
Matt Burke ac9b5127 2015-09-04T09:20:45 Pull extra_http_headers from the git_remote
Matt Burke 6af6e690 2015-09-04T09:18:32 Put the extra headers on the connection_data instead
Matt Burke c3733e56 2015-09-04T08:56:26 Add more headers to HTTP requests
Carlos Martín Nieto 11bca2d2 2015-08-15T18:15:23 http: propagate the credentials callback's error code When we ask for credentials, the user may choose to return EUSER to indicate that an error has happened on its end and it wants to be given back control. We must therefore pass that back to the user instead of mentioning that it was on_headers_complete() that returned an error code. Since we can, we return the exact error code from the user (other than PASSTHROUGH) since it doesn't cost anything, though using other error codes aren't recommended.
Anders Borum 31a76374 2015-07-29T22:23:00 case-insensitive check for WWW-Authenticate header Fixes issue #3338
Matthew Plough 768f8be3 2015-06-30T19:00:41 Fix #3094 - improve use of portable size_t/ssize_t format specifiers. The header src/cc-compat.h defines portable format specifiers PRIuZ, PRIdZ, and PRIxZ. The original report highlighted the need to use these specifiers in examples/network/fetch.c. For this commit, I checked all C source and header files not in deps/ and transitioned to the appropriate format specifier where appropriate.
Pierre-Olivier Latour 1630981e 2015-06-30T09:03:23 http: fixed leak when asking for credentials again t->cred might have been allocated the previous time and needs to be freed before asking caller for credentials again.
Carlos Martín Nieto 8b380060 2015-06-29T21:12:44 http: don't give up on auth on the first try When the server rejects an authentication request, ask the caller for the credentials again, instead of giving up on the first try.
Carlos Martín Nieto 8443f492 2015-06-11T16:57:04 curl: remove the encrypted param to the constructor We do not want libcurl to perform the TLS negotiation for us, so we don't need to pass this option.
Carlos Martín Nieto f97d5d09 2015-06-11T16:54:48 http: ask for the curl stream for non-encrypted connections The TLS streams talk over the curl stream themselves, so we don't need to ask for it explicitly. Do so in the case of the non-encrypted one so we can still make use proxies in that case.
Carlos Martín Nieto 8762d721 2015-06-07T14:51:10 http: set the proxy if the stream supports it Of the built-in ones, only cURL support it, but there's no reason a user-provided stream wouldn't support it.
Carlos Martín Nieto 8dea1c21 2015-06-05T11:02:11 Implement a curl stream cURL has a mode in which it acts a lot like our streams, providing send and recv functions and taking care of the TLS and proxy setup for us. Implement a new stream which uses libcurl instead of raw sockets or the TLS libraries directly. This version does not support reporting certificates or proxies yet.
J Wyman 7dd22538 2015-05-11T10:19:25 centralizing all IO buffer size values
Carlos Martín Nieto 24e53d2f 2015-03-19T09:55:20 Rename GIT_SSL to GIT_OPENSSL This is what it's meant all along, but now we actually have multiple implementations, it's clearer to use the name of the library.
Carlos Martín Nieto 6946a3be 2015-03-19T00:18:03 Abstract away the TLS stream implementation Instead, provide git_tls_stream_new() to ask for the most appropriate encrypted stream and use it in our HTTP transport.
Carlos Martín Nieto 6bb54cbf 2014-11-02T13:23:32 Add a SecureTransport TLS channel As an alternative to OpenSSL when we're on OS X. This one can actually take advantage of stacking the streams.
Edward Thomson c5e07187 2015-03-24T14:03:51 Merge pull request #2990 from leoyanggit/custom_param Add a custom param to git_smart_subtransport_definition
Leo Yang 142e5379 2015-03-17T12:49:33 Add a custom param to git_smart_subtransport_definition The smart transport has already take the payload param. For the sub transport a payload param is useful for the implementer.
Carlos Martín Nieto 84d83b8e 2015-03-16T19:41:50 http: do not try to use the cert callback on unencrypted streams When the user has a certificate check callback set, we still have to check whether the stream we're using is even capable of providing a certificate. In the case of an unencrypted certificate, do not ask for it from the stream, and do not call the callback.
Carlos Martín Nieto e892b6a9 2015-03-03T17:23:01 http: enforce the credential types The user may decide to return any type of credential, including ones we did not say we support. Add a check to make sure the user returned an object of the right type and error out if not.
Carlos Martín Nieto a2fd56ab 2014-12-10T16:22:50 Fix a couple of compiler warnings
Carlos Martín Nieto b6f5464e 2014-11-01T21:35:06 Port HTTP(S) to the new stream API
Carlos Martín Nieto 1ca61bdc 2014-11-19T20:53:25 fetch: clear the connection data on close When we fetch twice with the same remote object, we did not properly clear the connection flags, so we would leak state from the last connection. This can cause the second fetch with the same remote object to fail if using a HTTP URL where the server redirects to HTTPS, as the second fetch would see `use_ssl` set and think the initial connection wanted to downgrade the connection.
Edward Thomson 369b0217 2014-10-13T13:34:15 Clean up various compiler warnings
Sven Strickroth e6e834a1 2014-09-18T12:23:07 Provide host name to certificate_check_cb Signed-off-by: Sven Strickroth <email@cs-ware.de>
Carlos Martín Nieto 0782fc43 2014-09-16T01:47:30 net: use only structs to pass information about cert Instead of spreading the data in function arguments, some of which aren't used for ssh and having a struct only for ssh, use a struct for both, using a common parent to pass to the callback.
Carlos Martín Nieto 41698f22 2014-09-11T10:04:05 net: remove support for outright ignoring certificates This option make it easy to ignore anything about the server we're connecting to, which is bad security practice. This was necessary as we didn't use to expose detailed information about the certificate, but now that we do, we should get rid of this. If the user wants to ignore everything, they can still provide a callback which ignores all the information passed.
Carlos Martín Nieto 23ca0ad5 2014-08-29T21:25:13 Bring certificate check back to the normal return code Returning 0 lets the certificate check succeed. An error code is bubbled up to the user.
Carlos Martín Nieto 17491f6e 2014-08-29T17:18:23 transport: always call the certificate check callback We should let the user decide whether to cancel the connection or not regardless of whether our checks have decided that the certificate is fine. We provide our own assessment to the callback to let the user fall back to our checks if they so desire.
Carlos Martín Nieto ec1ce458 2014-08-10T17:06:53 http: send the DER-encoded cert to the callback Instead of the parsed data, we can ask OpenSSL to give us the DER-encoded version of the certificate, which the user can then parse and validate.
Carlos Martín Nieto 9b940586 2014-07-04T12:45:43 Provide a callback for certificate validation If the certificate validation fails (or always in the case of ssh), let the user decide whether to allow the connection. The data structure passed to the user is the native certificate information from the underlying implementation, namely OpenSSL or WinHTTP.
Carlos Martín Nieto 294c6f29 2014-08-16T22:12:13 http: make sure we can consume the data we request The recv buffer (parse_buffer) and the buffer have independent sizes and offsets. We try to fill in parse_buffer as much as possible before passing it to the http parser. This is fine most of the time, but fails us when the buffer is almost full. In those situations, parse_buffer can have more data than we would be able to put into the buffer (which may be getting full if we're towards the end of a data sideband packet). To work around this, we check if the space we have left on our buffer is smaller than what could come from the network. If this happens, we make parse_buffer think that it has as much space left as our buffer, so it won't try to retrieve more data than we can deal with. As the start of the data may no longer be at the start of the buffer, we need to keep track of where it really starts (data_offset) and use that in our calculations for the real size of the data we received from the network. This fixes #2518.
Edward Thomson 23135afa 2014-08-14T11:52:20 Introduce proper http authentication API
Edward Thomson 315cb38e 2014-07-31T18:43:20 Add GSSAPI support for SPNEGO/Kerberos auth over HTTP
Edward Thomson e003f83a 2014-07-31T15:14:56 Introduce git_buf_decode_base64 Decode base64-encoded text into a git_buf
Edward Thomson 40867266 2014-07-31T18:39:58 Perform HTTP keep-alive
Carlos Martín Nieto 9c3e4e97 2014-06-13T02:35:33 http: fix typo in credentials logic We want to check whether the credentials callback is NULL, not whether the payload is.
Carlos Martín Nieto bc0a6198 2014-04-19T15:52:58 transports: allow the creds callback to say it doesn't exist Allow the credentials callback to return GIT_PASSTHROUGH to make the transports code behave as though none was set. This should make it easier for bindings to behave closer to the C code when there is no credentials callback set at their level.
Russell Belfer 9cfce273 2013-12-12T12:11:38 Cleanups, renames, and leak fixes This renames git_vector_free_all to the better git_vector_free_deep and also contains a couple of memory leak fixes based on valgrind checks. The fixes are specifically: failure to free global dir path variables when not compiled with threading on and failure to free filters from the filter registry that had not be initialized fully.
Russell Belfer 25e0b157 2013-12-06T15:07:57 Remove converting user error to GIT_EUSER This changes the behavior of callbacks so that the callback error code is not converted into GIT_EUSER and instead we propagate the return value through to the caller. Instead of using the giterr_capture and giterr_restore functions, we now rely on all functions to pass back the return value from a callback. To avoid having a return value with no error message, the user can call the public giterr_set_str or some such function to set an error message. There is a new helper 'giterr_set_callback' that functions can invoke after making a callback which ensures that some error message was set in case the callback did not set one. In places where the sign of the callback return value is meaningful (e.g. positive to skip, negative to abort), only the negative values are returned back to the caller, obviously, since the other values allow for continuing the loop. The hardest parts of this were in the checkout code where positive return values were overloaded as meaningful values for checkout. I fixed this by adding an output parameter to many of the internal checkout functions and removing the overload. This added some code, but it is probably a better implementation. There is some funkiness in the network code where user provided callbacks could be returning a positive or a negative value and we want to rely on that to cancel the loop. There are still a couple places where an user error might get turned into GIT_EUSER there, I think, though none exercised by the tests.
Ben Straub 1b02baf4 2013-09-26T16:25:05 Adjust to new utility signature
Ben Straub 83fbd368 2013-09-26T15:58:41 Deploy gitno_connection_data into transport
Ben Straub 8988688c 2013-09-25T20:41:56 Migrate redirect URL handling to common utility
Russell Belfer 46fbc88e 2013-09-24T14:50:06 Prevent HTTPS to HTTP redirect
Russell Belfer eb0ff130 2013-09-24T14:07:08 Disconnect path string to preserve after redirect The subtransport path was relying on pointing to data owned by the remote which meant that after a redirect, the updated path was getting lost for future requests. This updates the http transport to strdup the path and maintain its own lifetime. This also pulls responsibility for parsing the URL back into the http transport and isolates the functions that parse and free that connection data so that they can be reused between the initial parsing and the redirect parsing.
Ben Straub 210d5325 2013-09-23T12:09:23 Allow redirects to use same host
Philip Kelley 35e0f3c6 2013-03-25T17:59:30 Refine the redirect check condition
Philip Kelley 2c7f7a66 2013-03-25T17:35:36 http: Support 302 Found (arrbee did most of the work)
Ben Straub 54ffc1f7 2013-01-31T14:41:01 HTTP: use creds in url if available
Ben Straub cf7038a6 2013-01-31T14:04:21 Enhance url parsing to include passwords