azure-pipelines


Log

Author Commit Date CI Message
lhchavez 6a917c04 2020-06-28T15:51:43 Add CI support for Memory and UndefinedBehavior Sanitizers This change adds two new build targets: MSan and UBSan. This is because even though OSS-Fuzz is great and adds a lot of coverage, it only does that for the fuzz targets, so the rest of the codebase is not necessarily run with the Sanitizers ever :( So this change makes sure that MSan/UBSan warnings don't make it into the codebase. As part of this change, the Ubuntu focal container is introduced. It builds mbedTLS and libssh2 as debug libraries into /usr/local and as MSan-enabled libraries into /usr/local/msan. This latter part is needed because MSan requires the binary and all its dependent libraries to be built with MSan support so that memory allocations and deallocations are tracked correctly to avoid false positives.
Edward Thomson 47fb33ba 2020-06-07T00:39:27 Introduce CI with GitHub Actions Add CI using GitHub Actions and GitHub Packages: * This moves our Linux build containers into GitHub Packages; we will identify the most recent commit that updated the docker descriptions, and then look for a docker image in libgit2's GitHub Packages registry for a container with the tag corresponding to that description. If there is not one, we will build the container and then push it to GitHub Packages. * We no longer need to manage authentication with our own credentials or PAT tokens. GitHub Actions provides a GITHUB_TOKEN that can publish artifacts, packages and commits to our repository within a workflow run. * We will use a matrix to build our various CI steps. This allows us to keep configuration in a single place without multiple YAML files.
Edward Thomson 4852d8da 2020-06-10T22:18:13 docker: don't take BASE as an argument The xenial image depends on ubuntu:xenial; the bionic one on ubuntu:bionic. No need for this to be a variable, that's just additional (unnecessary) state to manage in the CI setup(s).
Edward Thomson c16ba496 2020-06-07T00:37:55 yo
Edward Thomson 80c6b0a2 2020-06-07T00:16:57 Update bionic
Edward Thomson 4a95ee8d 2020-06-07T00:16:39 Update xenial
Edward Thomson c2ac139c 2020-06-07T00:12:47 Update xenial
Edward Thomson 0d82e49f 2020-06-07T00:12:17 Update bionic
Edward Thomson 153199ae 2020-03-17T09:42:41 ci: don't use --insecure mbedTLS has fixed their certificate. 🎉
Patrick Steinhardt 8621bdda 2020-03-13T22:42:51 azure: docker: use insecure flag to fix curl downloads We currently hve some problems with our curl downloads when building Docker images. It's not quite obvious what the problem is and they seem to occur semi-randomly. To unblock our CI, let's add the "--insecure" flag to curl to ignore any certificate errors. This is intended as a temporary solution only.
Patrick Steinhardt 95f329b4 2020-03-10T21:07:34 azure: upgrade to newer hosted VM images Azure is phasing out old images on March 23rd 2020, but we're currently still using them. So let's upgrade images as following: - Ubuntu 16.04 -> ubuntu-18.04 - macOS 10.13 -> macOS-10.15 - Hosted Windows machines -> vs2017-win2016 Each of them is currently the latest version. As the new Microsoft Windows machine has upgraded to MSVS2017, we need to also adjust our CMake generators to "Visual Studio 15 2017". As this CMake generator doesn't accept the target platform name anymore, we instead need to set it up via either "-A Win32" or "-A x64". [1]: https://devblogs.microsoft.com/devops/removing-older-images-in-azure-pipelines-hosted-pools/
Patrick Steinhardt 5ac33ced 2020-03-10T21:39:39 azure: docurium: fix build failure due to bumped CMake requirements Our Docurium builds currently depend on Debian Jessie, which has CMake v3.0 available. As rugged has bumped its CMake requirements to need at least v3.5 now, the documentation build is thus failing. Fix this by converting our Docurium Docker image to be based on Ubuntu Bionic. We already do base all of our images on Ubuntu, so I don't see any sense in using Debian here. If this was only to speed up builds, we should just go all the way and use some minimal container like Alpine anyway. Also remove cache busters. As we're rebuilding the image every time, it's we really don't need them at all.
Patrick Steinhardt c76c1e87 2020-03-10T20:39:09 azure: docker: consistently silence curl but show errors We currently pass the "--silent" flag to most invocations of curl, but in fact this does not only suppress the progress meter, but also any errors. So let's also pass "--show-error", too.
Edward Thomson dc55d0e8 2020-03-02T10:22:54 ci: provide globalsign certs for bionic tls.mbed.org has neglected to send their full certificate chain. Add their intermediate cert manually. 🙄
Edward Thomson 0f316d59 2020-03-01T14:42:03 ci: provide globalsign certs tls.mbed.org has neglected to send their full certificate chain. Add their intermediate cert manually. 🙄
Patrick Steinhardt 6efe3d35 2020-02-19T11:34:55 azure: docker: set up HOME variable to fix Coverity builds In commit 01a834066 (azure: docker: fix ARM builds by replacing gosu(1), 2020-02-18), we've switched our entrypoint from gosu(1) to use sudo(1) instead to fix our ARM builds. The switch introduced an incompatibility that now causes our Coverity builds to fail, as the "--preserve-env" switch will also keep HOME at its current value. As a result, Coverity now tries to set up its configuration directory in root's home directory, which it naturally can't write to. Fix the issue by adding the "--set-home" flag to sudo(1).
Patrick Steinhardt 01a83406 2020-02-18T15:20:43 azure: docker: fix ARM builds by replacing gosu(1) Our nightly builds are currently failing due to our ARM-based jobs. These jobs crash immediately when entering the Docker container with a exception thrown by Go's language runtime. As we're able to successfully builds the Docker images in previous steps, it's unlikely to be a bug in Docker itself. Instead, this exception is thrown by gosu(1), which is a Go-based utility to drop privileges and run by our entrypoint. Fix the issue by dropping gosu(1) in favor of sudo(1).
Patrick Steinhardt 76b49caf 2020-02-18T15:20:08 azure: docker: synchronize Xenial/Bionic build instructions Our two Docker build instructions for Xenial and Bionic have diverged a bit. Let's re-synchronize them with each other to make them as similar as possible.
Patrick Steinhardt f9985688 2020-02-18T15:17:45 azure: docker: detect errors when building images The build step for our Docker images currently succeeds even if building the Docker image fails due to missing && chains in the build script. Fix this by adding them in.
Patrick Steinhardt 68bfacb1 2020-02-18T15:17:17 azure: remove unused Linux setup script Since migrating to Docker containings for our build and test infrastructure, we do not use the "setup-linux.sh" script anymore. Remove it to avoid any confusion.
Patrick Steinhardt 49bb4237 2020-02-07T14:04:07 azure: test: silence termination message when killing git-daemon(1) In order to properly tear down the test environment, we will kill git-daemon(1) if we've exercised it. As git-daemon(1) is spawned as a background process, it is still owned by the shell and thus killing it later on will print a termination message to the shell's stderr, causing Azure to report it as an error. Fix this by disowning the background process.
Patrick Steinhardt fb03f02a 2020-02-07T13:56:36 azure: docker: avoid re-creating libgit2 home directory The Docker entrypoint currently creates the libgit2 user with "useradd --create-home". As we start the Docker container with two volumes pointing into "/home/libgit2/", the home directory will already exist. While useradd(1) copes with this just fine, it will print error messages to stderr which end up as failures in our Azure pipelines. Fix this by simply removing the "--create-home" parameter.
Patrick Steinhardt 52cb4040 2020-02-07T13:52:07 azure: test: silence curl to not cause Azure to trop Without the "--silent" parameter, curl will print a progress meter to stderr. Azure has the nice feature of interpreting any output to stderr as errors with a big red warning towards the end of the build. Let's thus silence curl to not generate any misleading messages.
Patrick Steinhardt a3ec07d7 2020-02-07T13:34:18 azure: docker: pipe downloaded archives into tar(1) directly When building dependencies for our Docker images, we first download the sources to disk first, unpack them and finally remove the archive again. This can be sped up by piping the downloading archive into tar(1) directly to parallelize both tasks. Furthermore, let's silence curl(1) to not print to status information to stderr, which tends to be interpreted as errors by Azure Pipelines.
Patrick Steinhardt 86c54cc8 2020-02-04T11:51:56 azure: coverity: fix Coverity builds due to various issues There's several issues with our Coverity builds, like e.g. missing wget in our containers. Simplify our Coverity pipeline and fix these issues.
Patrick Steinhardt ccffea6b 2020-02-04T11:47:16 azure: coverity: convert to use self-built containers Back in commit 5a6740e7f (azure: build Docker images as part of the pipeline, 2019-08-02), we have converted our pipelines to use self-built Docker images to ease making changes to our Dockerfiles. The commit didn't adjust our Coverity pipeline, though, so let's do this now.
Patrick Steinhardt b4eb0282 2020-02-04T11:46:05 azure: coverity: fix invalid syntax for Docker image In commit bbc0b20bd (azure: fix Coverity's build due to wrong container name, 2019-08-02), Coverity builds were fixed to use the correct container names. Unfortunately, the "fix" completely broke our Coverity builds due to using wrong syntax for the Docker task. Let's fix this by using "imageName" instead of the Docker dict.
Patrick Steinhardt c51bd2f2 2020-02-04T12:15:56 azure: tests: reset FAILED status if flaky re-run succeeds While we already do have logic to re-run flaky tests, the FAILED variable currently does not get reset to "0". As a result, successful reruns will still cause the test to be registered as failed. Fix this by resetting the variable accordingly.
Patrick Steinhardt b33ad764 2020-02-04T11:26:57 azure: tests: re-run flaky proxy tests The proxy tests regularly fail in our CI environment. Unfortunately, this is expected due to the network layer. Thus, let's re-try the proxy tests up to five times in case they fail.
Edward Thomson 7fd9b3f5 2020-01-01T20:48:15 ci: add NTLM tests Download poxygit, a debugging git server, and clone from it using NTLM, both IIS-style (with connection affinity) and Apache-style ("broken", requiring constant reauthentication).
Edward Thomson bf55facf 2019-10-25T12:24:34 tests: allow users to use expect/continue
Edward Thomson 9b96adf3 2019-11-22T18:03:50 docker build: add krb5 libraries and command-line tools
Edward Thomson e5fb5fe5 2019-10-20T17:19:01 ci: perform SPNEGO tests Attempt to obtain a Kerberos ticket from LIBGIT2.ORG and then clone the Negotiate-protected site at test.libgit2.org with that ticket.
Edward Thomson b8e00b98 2019-11-23T21:17:15 ci: cache docker layers Our docker builds are getting expensive, let's cache some of this.
Edward Thomson c64b7aaa 2019-11-23T20:38:30 ci: build our own valgrind The valgrind in the PPA is broken and ignores `--exit-errorcode`. Build and install our own.
Edward Thomson fd831275 2019-11-23T12:40:46 ci: build shared libssh2
Edward Thomson 84807884 2019-11-23T12:40:02 ci: break dockerfile into stages Use a multi-stage docker build so that we can cache early stages and not need to download the apt-provided dependencies during every build (when only later stages change).
Edward Thomson 7a3d04dc 2019-11-23T12:14:23 ci: don't delete the apt cache Deleting the apt cache can be helpful for reducing the size of a container, but since we don't push it anywhere, it only hinders our ability to debug problems while working on the container. Keep it.
Edward Thomson f592c737 2019-11-23T11:55:50 ci: don't install libssh2 since we build it
Edward Thomson 767990e9 2019-11-23T11:25:38 ci: show distribution information The lsb-release command is missing on our images; just show the information from the file instead of relying on it.
Edward Thomson 91ba65af 2019-11-23T10:58:38 ci: provide a default for xcode generator Provide a sane default for `CMAKE_GENERATOR` in the build script so that it can be invoked without having to set that in the environment.
Etienne Samson 3e862514 2019-09-21T22:50:27 azure: enable GSS on macOS builds
Patrick Steinhardt 3c884cc3 2019-09-21T15:05:36 azure: avoid building and testing in Docker as root Right now, all tests in libgit2's CI are being executed as root user. As libgit2 will usually not run as a root user in "normal" usecases and furthermore as there are tests that rely on the ability to _not_ be able to create certain paths, let's instead create an unprivileged user "libgit2" and use that across all docker images.
Patrick Steinhardt 48d23a8c 2019-08-02T12:36:19 azure: convert to use Ninja as build tool While we were still supporting Trusty, using Ninja as a build tool would have required us to first setup pip and then use it to install Ninja. As a result, the speedups from using Ninja were drowned out by the time required to install Ninja. But as we have deprecated Trusty now, both Xenial and Bionic have recent versions of Ninja in their repositories and thus we can now use Ninja.
Patrick Steinhardt 4e07a205 2019-08-02T13:29:54 docker: fix Valgrind errors on Xenial by updating to v3.12.0 The Valgrind version shipped with Xenial has some bugs that keep our tests from working due to bad interactions with openssl [1]. Fix this by using the "hola-launchpad/valgrind" PPA that provides a newer version for which the bug has been fixed. [1]: https://bugs.launchpad.net/ubuntu/+source/valgrind/+bug/1574437
Patrick Steinhardt 3c59d451 2019-08-02T12:34:10 docker: use "--no-install-recommends" to reduce build time Pass the flag "--no-install-recommends" to apt-get in order to trim down the number of packages installed, both reducing build time and image size. As this also causes some required packages to not be installed anymore, add these explicitly to the set of packages installed.
Patrick Steinhardt 39b7e8b0 2019-08-02T12:26:30 docker: convert apt-get to use best practices Reformat both Xenial and Bionic's Dockerfiles to use best practices. Most importantly, we now run `apt-get update` and `apt-get install` in one step followed up by removing the package lists to speed up installation and keep down the image size.
Patrick Steinhardt 9f91d57e 2019-08-02T14:25:02 docker: install libssh2 1.8.2 on Xenial While Xenial provides libssh2 in its repositories, it only has version 1.5.0 available. This version will unfortunately not be able to connect to GitHub due to their removal of weak cryptographic standards [1]. To still enable our CI to execute tests against GitHub, we thus have to update the provided libssh2 version to a newer one. Manually install libssh2 1.8.2 on Xenial. There's no need to do the same for Bionic, as it already provides libssh2 1.8.0. [1]: https://github.blog/2018-02-01-crypto-removal-notice/
Patrick Steinhardt 253dbea2 2019-08-02T10:21:32 docker: install mbedTLS on both Bionic and Xenial We're about to phase out support for Trusty, but neither Bionic nor Xenial images provide the mbedTLS library that's available in Trusty. Build them for both to pull them in line with Trusty.
Patrick Steinhardt bbc0b20b 2019-08-02T10:27:24 azure: fix Coverity's build due to wrong container name The Coverity build is still referencing an old "trusty-openssl" container that is not provided by either our own now-inlined images nor by the libgit2/libgit2-docker repository. Convert it to build and use Xenial images instead.
Patrick Steinhardt 76327381 2019-08-02T10:50:11 azure: deprecate Trusty in favor of Xenial Support for the LTS release Ubuntu 14.04 Trusty has been dropped in April 2019, but Azure is still using Trusty as its primary platform to build and test against. Let's deprecate it in favor of Xenial.
Patrick Steinhardt 5a6740e7 2019-08-02T09:58:55 azure: build Docker images as part of the pipeline The Docker images used for our continuous integration builds currently live in the libgit2/libgit2-docker repository. To make any changes in them, one has to make a PR there, get it reviewed, re-build the images and publish them to Docker Hub. This process is slow and tedious, making it harder than necessary to perform any updates to our Docker-based build pipeline. To fix this, we include all Dockerfiles used by Azure from the mentioned repository and inline them into our own repo. Instead of having to manually push them to the CI, it will now build the required containers on each pull request, allowing much greater flexibility.
Patrick Steinhardt 415ee616 2019-07-12T09:40:13 azure-pipelines: make gitdaemon tests work on Win32 On Win32 builds, the PID file created by git-daemon contained in invalid PID that we were not able to kill afterwards. Somehow, it seems like the contained PID was wrapped in braces. Consequentially, kill(1) failed and thus caused the build to error. Fix this by directly grabbing the PID of the spawned git-daemon process.
Patrick Steinhardt f867bfa8 2019-07-20T18:35:46 azure: skip SSH tests on Win32 platforms On Win32 build hosts, we do not have an SSH daemon readily available and thus cannot perform the SSH tests. Let's skip the tests to not let Azure Pipelines fail.
Patrick Steinhardt 0cda5252 2019-07-19T12:09:58 azure: use bash scripts across all platforms Right now, we maintain semantically equivalent build scripts in both Bash and Powershell to support both Windows and non-Windows hosts. Azure Pipelines supports Bash on Windows, too, via Git for Windows, and as such it's not really required to maintain the Powershell scripts at all. Remove them to reduce our own maintenance burden.
Patrick Steinhardt 1be4f896 2019-07-19T12:07:59 azure: avoid executing compiler if there is none Until now, we always had the CC variable defined in the build.sh pipeline. But as we're about to migrate the Windows jobs to Bash, as well, those will not have CC defined and thus we cannot use "$CC" to determine the compiler version. Fix this by only executing "$CC" if the variable was set.
Patrick Steinhardt 8e356f48 2019-07-20T18:35:20 azure: explicitly specify CMake generator We currently specify the CMake generator as part of the CMAKE_OPTIONS variable. This is fine in the current setup, but during the conversion to drop PowerShell scripts this will prove problematic for all generators that have spaces in their names due to quoting issues. Convert to use an explicit CMAKE_GENERATOR variable that makes it easier to get quoting right.
Patrick Steinhardt 443df2df 2019-06-24T16:27:00 azure: replace mingw setup with bash We're about to phase out our Powershell scripts as Azure Pipelines does in fact support Bash scripts on all platforms. As a preparatory step, let's replace our MinGW setup script with a Bash script.
Patrick Steinhardt d8e85d57 2019-06-27T15:01:24 azure: fix building in MinGW via Bash Azure Pipelines supports bash tasks on Windows hosts due to it always having Git for Windows included. To support this, the Git for Window directory is added to the PATH environment to make the bash shell available for execution. Unfortunately, this breaks CMake with the MinGW generator, as it has sanity checks to verify that no bash executable is in the PATH. So we can either remove Git for Windows from the path, but then we're unable to execute bash jobs. Or we can add it to the path, but then we're unable to execute CMake with the MinGW generator. Let's re-model how we set the PATH environment. Instead of setting up PATH for the complete build job, we now set a variable "BUILD_PATH" for the job. This variable is only being used when executing CMake so that it encounters a sanitizied PATH environment without GfW's bash shell.
Patrick Steinhardt ffac520e 2019-06-24T16:19:35 azure: move build scripts into "azure-pipelines" directory Since we have migrated to Azure Pipelines, we have deprecated and subsequentally removed all infrastructure for AppVeyor and Travis. Thus it doesn't make a lot of sense to have the split between "ci/" and "azure-pipelines/" directories anymoer, as "azure-pipelines/" is essentially our only CI. Move all CI scripts into the "azure-pipelines/" directory to have everything centrally located and to remove clutter in the top-level directory.
Patrick Steinhardt d827b11b 2019-06-28T13:20:54 tests: execute leak checker via CTest directly Right now, we have an awful hack in our test CI setup that extracts the test command from CTest's output and then prepends the leak checker. This is dependent on non-machine-parseable output from CMake and also breaks on various ocassions, like for example when we have spaces in the current path or when the path contains backslashes. Both conditions may easily be triggered on Win32 systems, and in fact they do break our Azure Pipelines builds. Remove the awful hack in favour of a new CMake build option "USE_LEAK_CHECKER". If specifying e.g. "-DUSE_LEAK_CHECKER=valgrind", then we will set up all tests to be run under valgrind. Like this, we can again simply execute ctest without needing to rely on evil sourcery.
Patrick Steinhardt 270fd807 2019-07-18T13:44:10 azure: compile one Windows platform with the WinHTTP SHA1 backend We currently have no job that compiles libgit2 with the WinHTTP backend for SHA1. Due to this, a compile error has been introduced and not noticed for several months. Change the x86 MSVC job to use the HTTPS backend for SHA1. The x86 job was chosen with no particular reason.
Etienne Samson 94fc83b6 2019-06-13T16:48:35 cmake: Modulize our TLS & hash detection The interactions between `USE_HTTPS` and `SHA1_BACKEND` have been streamlined. Previously we would have accepted not quite working configurations (like, `-DUSE_HTTPS=OFF -DSHA1_BACKEND=OpenSSL`) and, as the OpenSSL detection only ran with `USE_HTTPS`, the link would fail. The detection was moved to a new `USE_SHA1`, modeled after `USE_HTTPS`, which takes the values "CollisionDetection/Backend/Generic", to better match how the "hashing backend" is selected, the default (ON) being "CollisionDetection". Note that, as `SHA1_BACKEND` is still used internally, you might need to check what customization you're using it for.
Edward Thomson afb04a95 2019-05-21T14:03:04 ci: use a mix of regex backends Explicitly enable the `builtin` regex backend and the PCRE backend for some Linux builds.
Edward Thomson c7e5eca6 2019-02-28T10:46:26 Revert "foo" This reverts commit 1fe3fa5e59818c851d50efc6563db5f8a5d7ae9b.
Edward Thomson 1fe3fa5e 2019-02-28T10:44:34 foo
Edward Thomson 3f823c2b 2019-02-14T00:00:06 ci: enable hard deprecation Enable hard deprecation in our builds to ensure that we do not call deprecated functions internally.
Edward Thomson ef91917f 2019-02-14T09:19:32 ci: skip ssh tests on macOS nightly Like 811c1c0f8f80521dccc746a7bff180cd77a783ff, disable the SSH tests on macOS until we can resolve the newly introduced infrastructure issues.
Edward Thomson 0cf5b6b1 2019-01-28T10:48:49 ci: ignore coverity failures in nightly runs Coverity is back but it's only read-only! Agh. Just allow it to fail and not impact the overall job run.
Edward Thomson 08d71f72 2019-01-27T22:46:07 ci: return coverity to the nightlies
Edward Thomson 4a798a91 2018-10-28T17:57:53 nightly: use latest images, not test images
Edward Thomson 1ebf3a7d 2019-01-19T00:34:55 ci: only run invasive tests during nightly runs
Etienne Samson 3c6d1979 2019-01-11T02:06:41 ci: move coverity in its own pipeline Since Coverity is down for a unspecified timeframe, isolate it from the "hosted" nightlies.
Edward Thomson f195c385 2018-10-26T14:10:13 nightly: the path to yaml templates is relative Don't prefix the path to the yaml templates - the nightly template itself is already in the `azure-pipelines` directory. Instead, just use the relative path.
Edward Thomson be5a2ae2 2018-10-25T23:19:42 ci: run all the jobs during nightly builds Instead of running the oddball builds, run all the builds (the ones that we always run during PR validation and CI) during a nightly build for increased coverage.
Edward Thomson d82800e8 2018-10-21T09:31:42 ci: use bionic for non-amd64 builds Use Bionic so that we have a modern libssh2 (for communicating with GitHub). We've ported fixes to our Trusty-based amd64 images, but maintaining patches for multiple platforms is heinous.
Edward Thomson b244ea79 2018-10-21T09:24:08 ci: introduce nightly x86 linux builds
Edward Thomson 0e521abd 2018-10-21T09:15:24 ci: introduce nightly arm docker builds Use multiarch arm32 and arm64 docker images to run Xenial-based images for those platforms. We can support all the tests on ARM32 and 64 _except_ the proxy-based tests. Our proxy on ARM seems regrettably unstable, either due to some shoddy dependencies (with native code?) or the JREs themselves. Run these platforms as part of our nightly builds; do not run them during pull request or CI validation.
Edward Thomson 4ec597dc 2018-10-21T09:12:43 ci: move configuration yaml to its own directory As the number of each grow, separate the CI build scripts from the YAML definitions.