FIPS is short for "Federal Information Processing Standard", a document which defines certain security practices for a "cryptographic module" (CM). It aims to ensure a certain security floor is met by vendors selling products to U.S. Federal institutions.
GitLab has completed FIPS 140-2 Compliance for the build specified in this documentation. You can find our FIPS 140-2 Attestation in our customer assurance package, specifically the community package.
FIPS compliance at GitLab
To be compliant, all components (GitLab itself, Gitaly, etc) must be compliant, along with the communication between those components, and any storage used by them. Where functionality cannot be brought into compliance, it must be disabled when FIPS mode is enabled.
Leveraged Cryptographic modules
|Cryptographic module name
|Software component used
|Ubuntu 20.04 AWS Kernel Crypto API Cryptographic Module
|Ubuntu 20.04 OpenSSL Cryptographic Module
|Gitaly, Rails (Puma/Sidekiq)
|Ubuntu 20.04 Libgcrypt Cryptographic Module
|Amazon Linux 2 Kernel Crypto API Cryptographic Module
|Amazon Linux 2 OpenSSL Cryptographic Module
|RedHat Enterprise Linux 8 OpenSSL Cryptographic Module
|UBI containers: Workhorse, Pages, Container Registry, Rails (Puma/Sidekiq), Security Analyzers
|RedHat Enterprise Linux 8 Libgcrypt Cryptographic Module
|UBI containers: GitLab Shell,
Supported Operating Systems
The supported hybrid platforms are:
- Omnibus GitLab: Ubuntu 20.04 LTS
- Cloud Native GitLab: Amazon Linux 2 (EKS)
Unsupported features in FIPS mode
Some GitLab features may not work when FIPS mode is enabled. The following features are known to not work in FIPS mode. However, there may be additional features not listed here that also do not work properly in FIPS mode:
- Container Scanning support for scanning images in repositories that require authentication.
- Code Quality does not support operating in FIPS-compliant mode.
- Dependency scanning support for Gradle.
- Dynamic Application Security Testing (DAST) supports a reduced set of analyzers. The proxy-based analyzer and on-demand scanning is not available in FIPS mode today, however browser-based DAST, DAST API, and DAST API Fuzzing images are available.
- Solutions for vulnerabilities for yarn projects.
- Static Application Security Testing (SAST) supports a reduced set of analyzers when operating in FIPS-compliant mode.
- Advanced search is currently not included in FIPS mode. It must not be enabled to be FIPS-compliant.
- Gravatar or Libravatar-based profile images are not FIPS-compliant.
Additionally, these package repositories are disabled in FIPS mode:
FIPS validation at GitLab
Unlike FIPS compliance, FIPS validation is a formal declaration of compliance by an accredited auditor. The requirements needed to pass the audit are the same as for FIPS compliance.
A list of FIPS-validated modules can be found at the NIST (National Institute of Standards and Technology) cryptographic module validation program.
Install GitLab with FIPS compliance
This guide is specifically for public users or GitLab team members with a requirement to run a production instance of GitLab that is FIPS compliant. This guide outlines a hybrid deployment using elements from both Omnibus and our Cloud Native GitLab installations.
- Amazon Web Services account. Our first target environment is running on AWS, and uses other FIPS Compliant AWS resources.
- Ability to run Ubuntu 20.04 machines for GitLab. Our first target environment uses the hybrid architecture.
Set up a FIPS-enabled cluster
You can use the GitLab Environment Toolkit to spin up a FIPS-enabled cluster for development and testing. As mentioned in the prerequisites, these instructions use Amazon Web Services (AWS) because that is the first target environment.
Set up your environment
To get started, your AWS account must subscribe to a FIPS-enabled Amazon Machine Image (AMI) in the AWS Marketplace console.
This example assumes that the
Ubuntu Pro 20.04 FIPS LTS AMI by
Canonical Group Limited has been added your account. This operating
system is used for virtual machines running in Amazon EC2.
The simplest way to get a FIPS-enabled GitLab cluster is to use an Omnibus reference architecture. See the GET Quick Start Guide for more details. The following instructions build on the Quick Start and are also necessary for Cloud Native Hybrid installations.
Terraform: Use a FIPS AMI
GitLab team members can view more information in this internal handbook page on how to use FIPS AMI:
Ansible: Specify the FIPS Omnibus builds
The standard Omnibus GitLab releases build their own OpenSSL library, which is
not FIPS-validated. However, we have nightly builds that create Omnibus packages
that link against the operating system's OpenSSL library. To use this package,
gitlab_repo_script_url fields in the Ansible
GitLab team members can view more information in this internal handbook page on Ansible (AWS):
Cloud Native Hybrid
A Cloud Native Hybrid install uses both Omnibus and Cloud Native GitLab (CNG) images. The previous instructions cover the Omnibus part, but two additional steps are needed to enable FIPS in CNG:
- Use a custom Amazon Elastic Kubernetes Service (EKS) AMI.
- Use GitLab containers built with RedHat's Universal Base Image (UBI).
Build a custom EKS AMI
Because Amazon does not yet publish a FIPS-enabled AMI, you have to build one yourself with Packer.
Amazon publishes the following Git repositories with information about custom EKS AMIs:
This GitHub pull request makes it possible to create an Amazon Linux 2 EKS AMI with FIPS enabled for Kubernetes v1.21. To build an image:
Run the following:
git clone https://github.com/awslabs/amazon-eks-ami cd amazon-eks-ami git fetch origin pull/898/head:fips-ami git checkout fips-ami AWS_DEFAULT_REGION=us-east-1 make 1.21-fips # Be sure to set the region accordingly
If you are using a different version of Kubernetes, adjust the
When the AMI build is done, a new AMI should be created with a message such as the following:
==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs: AMIs were created:
In this example, the AMI ID is
ami-0a25e760cd00b027e, but your value may
Building a RHEL-based system with FIPS enabled should be possible, but there is an outstanding issue preventing the Packer build from completing.
Because this builds a custom AMI based on a specific version of an image, you must periodically rebuild the custom AMI to keep current with the latest security patches and upgrades.
Terraform: Use a custom EKS AMI
GitLab team members can view more information in this internal handbook page on how to use a custom EKS AMI:
Ansible: Use UBI images
CNG uses a Helm Chart to manage which container images to deploy. To use UBI-based containers, edit the Ansible
vars.yml to use custom
charts.yml in the location specified above and specify tags with a
You can also use release tags, but the versioning is tricky because each component may use its own versioning scheme. For example, for GitLab v15.2:
FIPS Performance Benchmarking
The Quality Engineering Enablement team assists these efforts by checking if FIPS-enabled environments perform well compared to non-FIPS environments.
Testing shows an impact in some places, such as Gitaly SSL, but it's not large enough to impact customers.
You can find more information on FIPS performance benchmarking in the following issue:
Setting up a FIPS-enabled development environment
The simplest approach is to set up a virtual machine running Red Hat Enterprise Linux 8.
Red Hat provide free licenses to developers, and permit the CD image to be downloaded from the Red Hat developer's portal. Registration is required.
After the virtual machine is set up, you can follow the GDK
installation instructions, including the advanced instructions for RHEL.
asdf is not used for dependency management because it's essential to
use the RedHat-provided Go compiler and other system dependencies.
Enable FIPS mode
After GDK and its dependencies are installed, run this command (as root) and restart the virtual machine:
You can check whether it's taken effect by running:
In this environment, OpenSSL refuses to perform cryptographic operations forbidden by the FIPS standards. This enables you to reproduce FIPS-related bugs, and validate fixes.
You should be able to open a web browser inside the virtual machine and sign in to the GitLab instance.
You can disable FIPS mode again by running this command, then restarting the virtual machine:
Detect FIPS enablement in code
You can query
Gitlab::FIPS in Ruby code to determine if the instance is FIPS-enabled:
Gitlab::SSHPublicKey.supported_sizes(name).select(&:positive?).min || -1
Omnibus FIPS packages
GitLab has a dedicated repository
for builds of the Omnibus GitLab which are built with FIPS compliance.
These GitLab builds are compiled to use the system OpenSSL, instead of
the Omnibus-embedded version of OpenSSL. These packages are built for:
- RHEL 8 (and compatible)
- AmazonLinux 2
These are consumed by the GitLab Environment Toolkit (GET).
Nightly Omnibus FIPS builds
The Distribution team has created nightly FIPS Omnibus builds, which can be used for testing purposes. These should never be used for production environments.
The following sections describe ways you can verify if FIPS is enabled.
$ cat /proc/sys/crypto/fips_enabled
Ruby (Omnibus images)
irb(main):001:0> require 'openssl'; OpenSSL.fips_mode
Ruby (CNG images)
irb(main):001:0> require 'openssl'; OpenSSL.fips_mode
Google maintains a
dev.boringcrypto branch in the Go compiler
that makes it possible to statically link BoringSSL, a FIPS-validated module forked from OpenSSL.
However, BoringSSL is not intended for public use.
- Using a FIPS-validated, system OpenSSL is straightforward.
- This is the source code used by Red Hat's go-toolset package.
- Unlike go-toolset, this fork appears to keep up with the latest Go releases.
However, cgo must be enabled via
CGO_ENABLED=1 for this to work. There
is a performance hit when calling into C code.
Projects that are compiled with
golang-fips on Linux x86 automatically
get built the crypto routines that use OpenSSL. While the
build tag is automatically present, no extra build tags are actually
needed. There are specific build tags
that disable these crypto hooks.
We can check whether a given binary is using OpenSSL via
go tool nm
and look for symbols named
Cfunc__goboringcrypto. For example:
$ go tool nm nginx-ingress-controller | grep Cfunc__goboringcrypto | tail
2a0b650 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA384_Final
2a0b658 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA384_Init
2a0b660 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA384_Update
2a0b668 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA512_Final
2a0b670 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA512_Init
2a0b678 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA512_Update
2a0b680 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_internal_ECDSA_sign
2a0b688 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_internal_ECDSA_verify
2a0b690 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_internal_ERR_error_string_n
2a0b698 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_internal_ERR_get_error
In addition, LabKit contains routines to check whether FIPS is enabled.
How FIPS builds are created
Many GitLab projects (for example: Gitaly, GitLab Pages) have
standardized on using
FIPS_MODE=1 make to build FIPS binaries locally.
The Omnibus FIPS builds are triggered with the
environment variable set to
true. When this environment variable is
set, the Omnibus recipes dependencies such as
curl, NGINX, and libgit2
will link against the system OpenSSL. OpenSSL will NOT be included in
the Omnibus build.
The Omnibus builds are created using container images that use the
golang-fips compiler. For
example, this job created
registry.gitlab.com/gitlab-org/gitlab-omnibus-builder/centos_8_fips:3.3.1 image used to
build packages for RHEL 8.
Add a new FIPS build for another Linux distribution
First, you need to make sure there is an Omnibus builder image for the desired Linux distribution. The images used to build Omnibus packages are created with Omnibus Builder images.
Review this merge request. A new image can be added by:
- Adding CI jobs with the
_fipssuffix (for example:
- Making sure the
Snippets.new(fips: fips).populateinstead of
After this image has been tagged, add a new CI job to Omnibus GitLab.
Cloud Native GitLab (CNG)
The Cloud Native GitLab CI pipeline generates images using several base images:
UBI images ship with the same OpenSSL package as those used by RHEL. This makes it possible to build FIPS-compliant binaries without needing RHEL. Note that RHEL 8.2 ships a FIPS-validated OpenSSL, but 8.5 is in review for FIPS validation.
This merge request
introduces a FIPS pipeline for CNG images. Images tagged for FIPS have the
-fips suffix. For example,
webservice container has the following tags:
Testing merge requests with a FIPS pipeline
Merge requests that can trigger Package and QA, can trigger a FIPS package and a Reference Architecture test pipeline. The base image used for the trigger is Ubuntu 20.04 FIPS:
e2e:package-and-testjob, if not already triggered.
- On the
gitlab-omnibus-mirrorchild pipeline, manually trigger
- When the package job is complete, manually trigger the