src/streams


Log

Author Commit Date CI Message
Zachary Michaels cdb9f390 2021-08-24T17:37:15 mbedTLS: Fix setting certificate directory fixes #6003
Edward Thomson 9fce506c 2021-08-24T15:53:10 openssl: lazily load libraries when dynamically loading Defer dlopen until it's needed when dynamically loading OpenSSL libraries.
Edward Thomson aad497cb 2021-08-24T14:08:43 openssl: dynamically load on macOS
Edward Thomson 0903cac1 2021-08-11T01:30:38 openssl: dynamically load libssl and symbols (optionally) Provide an interface around OpenSSL to dynamically load the libraries and symbols, so that users can distribute a libgit2 library that is not linked directly against OpenSSL. This enables users to target multiple distributions with a single binary. This mechanism is optional and disabled by default. Configure cmake with -DUSE_HTTPS=OpenSSL-Dynamic to use it.
Edward Thomson 150eddd9 2021-08-10T21:41:05 openssl: separate legacy api Refactor the OpenSSL stream implementation so that the legacy code is better abstracted. This will enable future development.
Edward Thomson 1903cfef 2021-08-19T16:49:41 openssl: don't fail when we can't customize allocators During valgrind runs, we try to swap out the OpenSSL allocators for our own. This allows us to avoid some unnecessary warnings about usage. Unfortunately, many builds of OpenSSL do not allow you to swap allocators; for example FIPS builds and the builds running in CentOS. Try to swap the allocators, but do not fail when they cannot be customized.
Edward Thomson 07a3c992 2020-11-21T23:52:39 streams: use GIT_ASSERT
Edward Thomson e316b0d3 2020-05-15T11:47:09 runtime: move init/shutdown into the "runtime" Provide a mechanism for system components to register for initialization and shutdown of the libgit2 runtime.
Edward Thomson 6554b40e 2020-05-13T10:39:33 settings: localize global data Move the settings global data teardown into its own separate function, instead of intermingled with the global state.
Patrick Steinhardt c6184f0c 2020-06-08T21:07:36 tree-wide: do not compile deprecated functions with hard deprecation When compiling libgit2 with -DDEPRECATE_HARD, we add a preprocessor definition `GIT_DEPRECATE_HARD` which causes the "git2/deprecated.h" header to be empty. As a result, no function declarations are made available to callers, but the implementations are still available to link against. This has the problem that function declarations also aren't visible to the implementations, meaning that the symbol's visibility will not be set up correctly. As a result, the resulting library may not expose those deprecated symbols at all on some platforms and thus cause linking errors. Fix the issue by conditionally compiling deprecated functions, only. While it becomes impossible to link against such a library in case one uses deprecated functions, distributors of libgit2 aren't expected to pass -DDEPRECATE_HARD anyway. Instead, users of libgit2 should manually define GIT_DEPRECATE_HARD to hide deprecated functions. Using "real" hard deprecation still makes sense in the context of CI to test we don't use deprecated symbols ourselves and in case a dependant uses libgit2 in a vendored way and knows it won't ever use any of the deprecated symbols anyway.
Patrick Steinhardt 6e1efcd6 2020-06-08T12:46:04 tree-wide: add missing header includes We're missing some header includes leading to missing function prototypes. While we currently don't warn about these, we should have their respective headers included in order to detect the case where a function signature change results in an incompatibility.
Patrick Steinhardt b43a9e66 2020-05-15T17:46:24 streams: openssl: fix memleak due to us not free'ing certs When creating a `git_cert` from the OpenSSL X509 certificate of a given stream, we do not call `X509_free()` on the certificate, leading to a memory leak as soon as the certificate is requested e.g. by the certificate check callback. Fix the issue by properly calling `X509_free()`.
Patrick Steinhardt 8aa04a37 2020-02-19T12:14:16 Merge pull request #5391 from pks-t/pks/coverity-fixes Coverity fixes
Patrick Steinhardt 0119e57d 2020-02-11T10:37:32 streams: openssl: switch approach to silence Valgrind errors As OpenSSL loves using uninitialized bytes as another source of entropy, we need to mark them as defined so that Valgrind won't complain about use of these bytes. Traditionally, we've been using the macro `VALGRIND_MAKE_MEM_DEFINED` provided by Valgrind, but starting with OpenSSL 1.1 the code doesn't compile anymore due to `struct SSL` having become opaque. As such, we also can't set it as defined anymore, as we have no way of knowing its size. Let's change gears instead by just swapping out the allocator functions of OpenSSL with our own ones. The twist is that instead of calling `malloc`, we just call `calloc` to have the bytes initialized automatically. Next to soothing Valgrind, this approach has the benefit of being completely agnostic of the memory sanitizer and is neatly contained at a single place. Note that we shouldn't do this for non-Valgrind builds. As we cannot set up memory functions for a given SSL context, only, we need to swap them at a global context. Furthermore, as it's possible to call `OPENSSL_set_mem_functions` once only, we'd prevent users of libgit2 to set up their own allocators.
Patrick Steinhardt b3b92e09 2020-02-07T12:56:26 streams: openssl: ignore return value of `git_mutex_lock` OpenSSL pre-v1.1 required us to set up a locking function to properly support multithreading. The locking function signature cannot return any error codes, and as a result we can't do anything if `git_mutex_lock` fails. To silence static analysis tools, let's just explicitly ignore its return value by casting it to `void`.
Edward Thomson cb77423f 2019-11-24T16:22:31 valgrind: add valgrind hints in OpenSSL Provide usage hints to valgrind. We trust the data coming back from OpenSSL to have been properly initialized. (And if it has not, it's an OpenSSL bug, not a libgit2 bug.) We previously took the `VALGRIND` option to CMake as a hint to disable mmap. Remove that; it's broken. Now use it to pass on the `VALGRIND` definition so that sources can provide valgrind hints.
Edward Thomson 2ad3eb3e 2019-11-24T15:59:26 valgrind: add suppressions for undefined use valgrind will warn that OpenSSL will use undefined data in connect/read when talking to certain other TLS stacks. Thankfully, this only seems to occur when gcc is the compiler, so hopefully valgrind is just misunderstanding an optimization. Regardless, suppress this warning.
Edward Thomson 2a4bcf63 2019-06-23T18:24:23 errors: use lowercase Use lowercase for our error messages, per our custom.
Patrick Steinhardt 0ceac0d0 2019-01-23T14:45:19 mbedtls: fix potential size overflow when reading or writing data The mbedtls library uses a callback mechanism to allow downstream users to plug in their own receive and send functions. We implement `bio_read` and `bio_write` functions, which simply wrap the `git_stream_read` and `git_stream_write` functions, respectively. The problem arises due to the return value of the callback functions: mbedtls expects us to return an `int` containing the actual number of bytes that were read or written. But this is in fact completely misdesigned, as callers are allowed to pass in a buffer with length `SIZE_MAX`. We thus may be unable to represent the number of bytes written via the return value. Fix this by only ever reading or writing at most `INT_MAX` bytes.
Patrick Steinhardt 75918aba 2019-01-23T14:43:54 mbedtls: make global variables static The mbedtls stream implementation makes use of some global variables which are not marked as `static`, even though they're only used in this compilation unit. Fix this and remove a duplicate declaration.
Patrick Steinhardt 657197e6 2019-01-23T15:54:05 openssl: fix potential size overflow when writing data Our `openssl_write` function calls `SSL_write` by passing in both `data` and `len` arguments directly. Thing is, our `len` parameter is of type `size_t` and theirs is of type `int`. We thus need to clamp our length to be at most `INT_MAX`.
Patrick Steinhardt 7613086d 2019-01-23T15:49:28 streams: handle short writes only in generic stream Now that the function `git_stream__write_full` exists and callers of `git_stream_write` have been adjusted, we can lift logic for short writes out of the stream implementations. Instead, this is now handled either by `git_stream__write_full` or by callers of `git_stream_write` directly.
Patrick Steinhardt 5265b31c 2019-01-23T15:00:20 streams: fix callers potentially only writing partial data Similar to the write(3) function, implementations of `git_stream_write` do not guarantee that all bytes are written. Instead, they return the number of bytes that actually have been written, which may be smaller than the total number of bytes. Furthermore, due to an interface design issue, we cannot ever write more than `SSIZE_MAX` bytes at once, as otherwise we cannot represent the number of bytes written to the caller. Unfortunately, no caller of `git_stream_write` ever checks the return value, except to verify that no error occurred. Due to this, they are susceptible to the case where only partial data has been written. Fix this by introducing a new function `git_stream__write_full`. In contrast to `git_stream_write`, it will always return either success or failure, without returning the number of bytes written. Thus, it is able to write all `SIZE_MAX` bytes and loop around `git_stream_write` until all data has been written. Adjust all callers except the BIO callbacks in our mbedtls and OpenSSL streams, which already do the right thing and require the amount of bytes written.
Patrick Steinhardt 193e7ce9 2019-01-23T15:42:07 streams: make file-local functions static The callback functions that implement the `git_stream` structure are only used inside of their respective implementation files, but they are not marked as `static`. Fix this.
Edward Thomson f1986a23 2019-01-21T09:56:23 streams: don't write more than SSIZE_MAX Our streams implementation takes a `size_t` that indicates the length of the data buffer to be written, and returns an `ssize_t` that indicates the length that _was_ written. Clearly no such implementation can write more than `SSIZE_MAX` bytes. Ensure that each TLS stream implementation does not try to write more than `SSIZE_MAX` bytes (or smaller; if the given implementation takes a smaller size).
Edward Thomson e09f0c10 2019-01-23T10:21:42 deprecation: don't use deprecated stream cb Avoid the deprecated `git_stream_cb` typedef since we want to compile the library without deprecated functions or types. Instead, we can unroll the alias to its actual type.
Sven Strickroth bff7aed2 2019-01-24T16:44:04 Don't use deprecated constants Follow up for PR #4917. Signed-off-by: Sven Strickroth <email@cs-ware.de>
Edward Thomson f673e232 2018-12-27T13:47:34 git_error: use new names in internal APIs and usage Move to the `git_error` name in the internal API for error-related functions.
lhchavez 7b453e7e 2019-01-05T22:12:48 Fix a bunch of warnings This change fixes a bunch of warnings that were discovered by compiling with `clang -target=i386-pc-linux-gnu`. It turned out that the intrinsics were not necessarily being used in all platforms! Especially in GCC, since it does not support __has_builtin. Some more warnings were gleaned from the Windows build, but I stopped when I saw that some third-party dependencies (e.g. zlib) have warnings of their own, so we might never be able to enable -Werror there.
Edward Thomson 02bb39f4 2018-11-22T08:49:09 stream registration: take an enum type Accept an enum (`git_stream_t`) during custom stream registration that indicates whether the registration structure should be used for standard (non-TLS) streams or TLS streams.
Edward Thomson df2cc108 2018-11-18T10:29:07 stream: provide generic registration API Update the new stream registration API to be `git_stream_register` which takes a registration structure and a TLS boolean. This allows callers to register non-TLS streams as well as TLS streams. Provide `git_stream_register_tls` that takes just the init callback for backward compatibliity.
Edward Thomson 21142c5a 2018-10-29T10:04:48 http: remove cURL We previously used cURL to support HTTP proxies. Now that we've added this support natively, we can remove the curl dependency.
Edward Thomson 2878ad08 2018-10-29T08:59:33 streams: remove unused tls functions The implementations of git_openssl_stream_new and git_mbedtls_stream_new have callers protected by #ifdefs and are never called unless compiled in. There's no need for a dummy implementation. Remove them.
Edward Thomson 43b592ac 2018-10-25T08:49:01 tls: introduce a wrap function Introduce `git_tls_stream_wrap` which will take an existing `stream` with an already connected socket and begin speaking TLS on top of it. This is useful if you've built a connection to a proxy server and you wish to begin CONNECT over it to tunnel a TLS connection. Also update the pluggable TLS stream layer so that it can accept a registration structure that provides an `init` and `wrap` function, instead of a single initialization function.
Etienne Samson 03994912 2018-08-29T01:57:24 openssl: only say we're connected if the connection succeeded ssl_close uses this boolean to know if SSL_shutdown should be called. It turns out OpenSSL auto-shutdowns on failure, so if the call to SSL_connect fails, it will complain about "shutdown while in init", trampling the original error.
Etienne Samson caee0a66 2018-08-29T01:57:22 openssl: set the error class to GITERR_SSL
Patrick Steinhardt 2613fbb2 2018-10-18T11:58:14 global: replace remaining use of `git__strtol32` Replace remaining uses of the `git__strtol32` function. While these uses are all safe as the strings were either sanitized or from a trusted source, we want to remove `git__strtol32` altogether to avoid future misuse.
Etienne Samson 21496c30 2018-03-30T00:20:23 stransport: fix a warning on iOS "warning: values of type 'OSStatus' should not be used as format arguments; add an explicit cast to 'int' instead [-Wformat]"
Patrick Steinhardt d4198d4d 2018-07-26T12:11:34 mbedtls: remove unused variable "cacert" In commit 382ed1e87 (mbedtls: load default CA certificates, 2018-03-29), the function `git_mbedtls_stream_global_init` was refactored to call out to `git_mbedtls__set_cert_location` instead of setting up the certificates itself. The conversion forgot to remove the now-unused "cacert" variable, which is now only getting declared to be free'd at the end of the function. Remove it.
Edward Thomson 2fabb622 2018-07-21T01:36:46 mbedtls: free stream on shutdown
Edward Thomson 9e002cd5 2018-07-21T01:11:58 mbedtls: make ciphers_list a static array Instead of allocating the ciphers_list, make it a static array. This prevents us from leaking it or having to manage its memory.
Edward Thomson 4e62d26f 2018-07-21T00:45:24 mbedtls: free ciphers_list
Edward Thomson defa9709 2018-07-21T00:41:38 mbedtls: check allocations
Patrick Steinhardt d19381e2 2018-06-25T14:57:07 mbedtls: fix `inline` being used in mbedtls headers The mbedtls headers make direct use of the `inline` attribute to instruct the compiler to inline functions. As this function is not C90 compliant, this can cause the compiler to error as soon as any of these files is included and the `-std=c90` flag is being added. The mbedtls headers declaring functions as inline always have a prelude which define `inline` as a macro in case it is not yet defined. Thus, we can easily replace their define with our own define, which simply copies the logic of our own `GIT_INLINE` macro.
Patrick Steinhardt 75395c87 2018-06-29T13:35:14 streams: report OpenSSL errors if global init fails In case when the global initialization of the OpenSSL stream fails, the user is left without any hint as to what went wrong as we do not provide any error message at all. This commit refactors the init function to have a common error path, which now also sets an error message including the error string provided by OpenSSL.
Nikita Leshenko 05d89e72 2018-06-25T17:30:04 streams: openssl: Handle error in SSL_CTX_new SIGSEGV otherwise...
Quentin Minster b1cab70b 2018-05-30T02:15:09 streams: openssl: add missing check on OPENSSL_LEGACY_API The `CRYPTO_THREADID` type is no longer available in OpenSSL ≥ 1.1.0 with deprecated features disabled, and causes build failures. Since the `threadid_cb()` function is only ever called by `git_openssl_set_locking()` when `defined(OPENSSL_LEGACY_API)`, only define it then.
Patrick Steinhardt ba5e39ac 2018-05-04T15:25:11 streams: openssl: fix bogus warning on unused parameter Our provided callback function `threadid_cb(CRYPTO_THREADID *threadid)` sets up a unique thread ID by asking pthread for the current thread ID. Since openssl version 1.1, `CRYPTO_THREADID_set_numeric` is simply a no-op macro, leaving the `threadid` argument unused after the preprocessor has processed the macro. GCC does not account for that situation and will thus complain about `threadid` being unused. Silence this warning by using `GIT_UNUSED(threadid)`.
Carlos Martín Nieto 26a09a93 2018-04-30T21:34:36 Merge pull request #4608 from pks-t/pks/openssl-api-cleanup OpenSSL legacy API cleanups
Etienne Samson 173a0375 2018-02-08T23:50:14 openssl: remove leftover #ifdef This is the "OpenSSL available" global init function after all
Etienne Samson b3e0280d 2018-03-29T22:14:11 mbedtls: display error codes as hex for consistency with mbedTLS docs Remaining parts of https://github.com/JuliaLang/julia/blob/8d47a314537779c8fb86642c54925613628a91b0/deps/patches/libgit2-mbedtls-fixup.patch
Etienne Samson 382ed1e8 2018-03-29T22:14:09 mbedtls: load default CA certificates
Etienne Samson 1edde0bc 2018-03-29T22:14:08 mbedtls: use mbedTLS certificate verification Taken from https://github.com/JuliaLang/julia/blob/8d47a314537779c8fb86642c54925613628a91b0/deps/patches/libgit2-mbedtls-verify.patch, with some modifications.
Etienne Samson 4165bb7f 2018-03-29T22:14:06 mbedtls: use our own certificate validation Otherwise REQUIRED means that `git_stream_certificate` will always error. We're doing the mbedtls check in verify_server_cert though.
Etienne Samson ec79b0fd 2018-03-29T22:14:04 mbedtls: fix libgit2 hanging due to incomplete writes
Etienne Samson 2419cccd 2018-03-29T22:14:02 mbedtls: default cipher list support
Etienne Samson 60e1ad92 2018-03-29T22:14:01 mbedtls: add global initialization
Etienne Samson 1a1875f3 2018-03-29T22:13:58 mbedtls: proper certificate verification
Etienne Samson ca3b2234 2018-03-29T22:13:56 mbedtls: initial support
Patrick Steinhardt dd9de88a 2018-04-03T11:57:45 streams: openssl: provide `OPENSSL_init_ssl` for legacy API In order to further avoid using ifdef's in our code flow, provide the function `OPENSSL_init_ssl` in case we are using the legacy OpenSSL API.
Patrick Steinhardt ede63b99 2018-04-03T11:45:00 streams: openssl: unify version checks into single define By now, we have several locations where we are checking the version of OpenSSL to determine whether we can use the new "modern" API or need to use the pre-1.1 legacy API. As we have multiple implementations of OpenSSL with the rather recent libressl implementation, these checks need to honor versions of both implementations, which is rather tedious. Instead, we can just check once for the correct versions and define `OPENSSL_LEGACY_API` in case we cannot use the modern API.
Patrick Steinhardt 2505cbfc 2018-04-03T11:40:39 streams: openssl: move OpenSSL compat layer into implementation OpenSSL version 1.1 has broken its API in quite a few ways. To avoid having to use ifdef's everywhere, we have implemented the BIO functions added in version 1.1 ourselves in case we are using the legacy API. We were implementing them in the header file, though, which doesn't make a lot of sense, since these functions are only ever being used the the openssl stream implementation. Move these functions to the implementation file and mark them static.
Bernard Spil 7490d449 2018-04-02T20:00:07 Fix build with LibreSSL 2.7 LibreSSL 2.7 adds OpenSSL 1.1 API Signed-off-by: Bernard Spil <brnrd@FreeBSD.org>
Patrick Steinhardt 2022b004 2018-02-28T12:06:59 curl: explicitly initialize and cleanup global curl state Our curl-based streams make use of the easy curl interface. This interface automatically initializes and de-initializes the global curl state by calling out to `curl_global_init` and `curl_global_cleanup`. Thus, all global state will be repeatedly re-initialized when creating multiple curl streams in succession. Despite being inefficient, this is not thread-safe due to `curl_global_init` being not thread-safe itself. Thus a multi-threaded programing handling multiple curl streams at the same time is inherently racy. Fix the issue by globally initializing and cleaning up curl's state.
Patrick Steinhardt 84f03b3a 2018-02-16T10:48:55 streams: openssl: fix use of uninitialized variable When verifying the server certificate, we do try to make sure that the hostname actually matches the certificate alternative names. In cases where the host is either an IPv4 or IPv6 address, we have to compare the binary representations of the hostname with the declared IP address of the certificate. We only do that comparison in case we were successfully able to parse the hostname as an IP, which would always result in the memory region being initialized. Still, GCC 6.4.0 was complaining about usage of non-initialized memory. Fix the issue by simply asserting that `addr` needs to be initialized. This shuts up the GCC warning.
Edward Thomson a223bae5 2018-01-03T14:57:25 Merge pull request #4437 from pks-t/pks/openssl-hash-errors hash: openssl: check return values of SHA1_* functions
Patrick Steinhardt ba56f781 2018-01-03T12:54:42 streams: openssl: fix thread-safety for OpenSSL error messages The function `ERR_error_string` can be invoked without providing a buffer, in which case OpenSSL will simply return a string printed into a static buffer. Obviously and as documented in ERR_error_string(3), this is not thread-safe at all. As libgit2 is a library, though, it is easily possible that other threads may be using OpenSSL at the same time, which might lead to clobbered error strings. Fix the issue by instead using a stack-allocated buffer. According to the documentation, the caller has to provide a buffer of at least 256 bytes of size. While we do so, make sure that the buffer will never get overflown by switching to `ERR_error_string_n` to specify the buffer's size.
Etienne Samson 8be2a790 2017-12-05T23:21:05 openssl: free the peer certificate Per SSL_get_peer_certificate docs: ``` The reference count of the X509 object is incremented by one, so that it will not be destroyed when the session containing the peer certificate is freed. The X509 object must be explicitly freed using X509_free(). ```
Etienne Samson 2518eb81 2017-11-24T14:04:10 openssl: merge all the exit paths of verify_server_cert This makes it easier to cleanup allocated resources on exit.
Etienne Samson 1b2e83a9 2017-12-13T00:19:41 stransport: provide error message on trust failures Fixes #4440
Patrick Steinhardt 2d2e70f8 2017-11-30T18:10:28 openssl: fix thread-safety on non-glibc POSIX systems While the OpenSSL library provides all means to work safely in a multi-threaded application, we fail to do so correctly. Quoting from crypto_lock(3): OpenSSL can safely be used in multi-threaded applications provided that at least two callback functions are set, locking_function and threadid_func. We do in fact provide the means to set up the locking function via `git_openssl_set_locking()`, where we initialize a set of locks by using the POSIX threads API and set the correct callback function to lock and unlock them. But what we do not do is setting the `threadid_func` callback. This function is being used to correctly locate thread-local data of the OpenSSL library and should thus return per-thread identifiers. Digging deeper into OpenSSL's documentation, the library does provide a fallback in case that locking function is not provided by the user. On Windows and BeOS we should be safe, as it simply "uses the system's default thread identifying API". On other platforms though OpenSSL will fall back to using the address of `errno`, assuming it is thread-local. While this assumption holds true for glibc-based systems, POSIX in fact does not specify whether it is thread-local or not. Quoting from errno(3p): It is unspecified whether errno is a macro or an identifier declared with external linkage. And in fact, with musl there is at least one libc implementation which simply declares `errno` as a simple `int` without being thread-local. On those systems, the fallback threadid function of OpenSSL will not be thread-safe. Fix this by setting up our own callback for this setting. As users of libgit2 may want to set it themselves, we obviously cannot always set that function on initialization. But as we already set up primitives for threading in `git_openssl_set_locking()`, this function becomes the obvious choice where to implement the additional setup.
Etienne Samson 22317057 2017-03-21T00:36:32 https: Prevent OpenSSL from namespace-leaking
Etienne Samson e9369856 2017-03-21T00:25:15 stream: Gather streams to src/streams