CNCF TAG-Security Harbor Project Self Assessment

March 2020 Primary Author: Michael Michael, Harbor Maintainer ( @michmike, @michmike77) Security Reviewers: Andres Vega, Justin Cappos, Chase Pettet, Vinay Venkataraghavan, Robert Ficaglia, Martin Vrachev, Payam Tarverdyan Chychi, Cameron Seader.

This document details the design goals and security implications of Harbor to aid in the security assessment by CNCF TAG-Security.

• CNCF TAG-Security Harbor Project Self Assessment
• Metadata
• Overview
  • Background
  • Goals
    • Security Goals
  • Non-Goals
  • History
• Intended Use
  • Target Users
  • Use Cases
• Project & Design
  • System Design
    • Identity Provider Integration
    • Components & Dependencies
  • Operations
    • Breakdown of Access, Tokens, and Creds in Harbor
  • Configuration and Set-Up
    • Default Configuration
  • Project Compliance
• Security Analysis
  • Attacker Motivations
  • Predisposing Conditions
  • Expected Attacker Capabilities
  • Attack Risks and Effects
  • Security Degradation
  • Blast Radius and Recovery
  • Compensating Mechanisms
    • Compartmentalization
    • RBAC
    • Forensics
• Secure Development Practices
  • Development Pipeline
    • Tooling
  • Communication Channels
  • Ecosystem
  • Security Issue Resolution
  • Responsible Disclosures Process
  • Incident Response
  • Security Testing
• Roadmap
  • Project Next Steps
• Appendix
  • Known Issues Over Time
  • CII Best Practices
  • Case Studies
  • Competition
  • Missing Features

Metadata

Overview

The Harbor project is an enterprise-grade container registry that enables enterprises to assess the security posture of container images and enforce that images are compliant with being free from vulnerabilities as defined in the configured vulnerability databases.

NOTE

Vulnerabilities are only detected in an artifact or container image as the result of static analysis. This static analysis is performed by a scanner like CoreOS’s Clair or Aqua’s Trivy. If the vulnerabilities are already published and the scanner can pick them up, images are being analyzed against those vulnerabilities. In the case of Zero Day CVEs, these take time to be published in vulnerability databases and may take hours or days to be picked up by Trivy or Clair. Each scanner has their own vulnerability databases that they use for scanning.

The project was started by a group of VMware engineers in 2014. Since then, it has grown over time into a popular open source container image registry that is able to secure images with role-based access control, scan images for vulnerabilities, and sign images for authenticity and traceability. As a CNCF Incubating project, Harbor delivers compliance, performance, and interoperability to help you consistently and securely manage images across cloud native compute platforms like Kubernetes and Docker.

Harbor is a packaged solution, meaning an end user will download Harbor from our GitHub release site and install on their own hardware. The end user is responsible for management, operations, upgrades, and Day 2 operations (Day 2 Operations generally describes the monitoring, logging, maintenance, and troubleshooting that keeps apps, services, and hosts up and running). In that sense, Harbor can be considered a private enterprise registry as it is not a public service offered by the Harbor Project team.

The Harbor project provides the following benefits:

• Compliance - Image compliance is enforced by enabling Harbor project administrators to prevent images from being deployed if they were scanned and found to have vulnerabilities. Compliance is only enforced at a point in time using the available data. Data in this case includes the policy set by the administrator (e.g. Don’t allow images with high severity vulnerabilities to be pulled from Harbor), the vulnerability database the scanner will use, and the scan results against images from Trivy or Clair. Once an image is deployed, Harbor no longer plays a role in compliance with the running image on a Kubernetes cluster (even if 10 minutes after an image is deployed, a Harbor scan is executed and we found this image to be vulnerable. A Harbor administrator can configure webhooks to be notified when such vulnerabilities are found, but they have to execute actions outside of Harbor to enforce compliance. For example, they have to patch or quarantine their Kubernetes pods). Products like Aqua CSP have an integrated solution that can help customers enforce compliance for running images in a cluster.
• Performance - Highly modular architecture (more on that in later sections) and scalable core services enable Harbor to scale and serve hundreds of Kubernetes clusters and Terabytes of storage. Performance and Scalability is covered in this document.
• Interoperability - Harbor integrates with many other registries for replication (more on that in later sections) as well as integrates with popular static analysis scanners (more on that in later sections). These options enable project administrators to choose which tools they wish to use for their use cases.

As you are reading this document, please pay specific attention to three very important sections. These sections break down the different sets of credentials and access in Harbor, what happens if any of them are compromised, and the steps towards recovery and mitigation. In addition, these sections talk about compensating circumstances that limit exposure or risk.

1. Breakdown of Access, Tokens, and Creds in Harbor
2. Blast Radius and Recovery
3. Compensating Mechanisms

Background

Harbor is an open source container image registry that can securely store images, helm charts, and other OCI-compliant cloud native artifacts with role-based access control, scan images for vulnerabilities, and sign images for authenticity and traceability. Harbor signs images by leveraging TUF Notary. Harbor users that chose to utilize the Notary integration and capabilities have some key advantages like Survivable Key Comporose, Freshness Guarantees, Configurable Trust Thresholds, Signing Delegation, Use of Existing Distribution, and Untrusted Mirrors and Transport. Read more about those advantages here.

As of the writing of this assessment, Harbor (latest released version is v1.10) supports only two artifact types, container images and Helm charts. With Harbor v2.0 (to be released in April 2020), Harbor will be fully OCI compliant. We will support any OCI compliant artifact type, providing them many of the capabilities that Harbor offers today → RBAC, quotas, retention policy, replication, policy, and more. Scanning will be available to images and CNAB bundles. Some of the artifact types supported include: docker images, docker manifest lists, Helm charts, CNAB bundles, OPA bundles.

Goals

The goal of the Harbor project is to be an all-in-one registry that can be used to store and serve cloud native OCI-compliant artifacts. You can see a comparison with other registry solutions on how Harbor stacks up on what we consider various capabilities of a registry. Harbor is designed as a complementary service to Kubernetes deployments and aligns well with other projects in the cloud native ecosystem.

By design, Harbor is intended to be complementary and well aligned with the cloud native community and other projects in the CNCF ecosystem. As a cloud native registry for docker containers and Helm charts, it supports, utilizes, or depends on the following projects and use cases:

1. Kubernetes
   1. Kubernetes deployments can leverage Harbor as their cloud native registry for Helm charts and docker containers
   2. Kubernetes operators can create their cloud native artifact security policy on Harbor and create the guard rails that allow developers to use Harbor in a self service way
2. Helm - Harbor can store and serve Helm charts by integrating with the Helm Chart Repository Server ChartMuseum
3. Notary - Harbor depends on Notary for signing images and establishing trust and ownership for image creators and consumers. This is an optional component in Harbor. Administrators may not want to take a dependency on Notary or may not need image signing or any of the Notary advantages mentioned earlier, so Harbor offers a choice in enabling this feature or not.
4. CoreOS Clair - The built-in vulnerability & static analysis scanner in Harbor. This is a hard dependency for Harbor. Harbor is bundled and includes Clair for release v1.10 and v2.0.
5. Aqua Trivy - A first-generation optional pluggable scanning adapter that Harbor users can utilize to assess the security posture of docker containers. This is a hard dependency for Harbor starting with version 2.0 that will ship in April 2020.
6. Anchore Engine - A first-generation optional pluggable scanning adapter that Harbor users can utilize to assess the security posture of docker containers. This is an optional component if the user wants to leverage Anchore’s security scanners.
7. DoSec (Beijing Xiaoyou Technology Co., Ltd) - A first-generation optional pluggable scanning adapter that was independently built by a security company which focuses on cloud-native and container security tooling. This is an optional component if the user wants to leverage DoSec’s security scanner.

For more information on the pluggable scanners in Harbor, visit our pluggable scanner compatibility list.

The Harbor project also plans to integrate, store, or make use of other cloud native projects like Prometheus, CNAB bundles (coming with Harbor v2.0 in April 2020), Kubernetes Operators, OCI Registry (coming with Harbor v2.0 in April 2020), Dragonfly (coming in summer 2020), and Kraken (coming in summer 2020).

Security Goals

• Protect artifacts in the registry from being replaced with malicious content by preventing unauthorized access. (e.g insert a backdoor in a container image)
  • Mitigation: Harbor enables enterprises to integrate with OIDC and LDAP for authentication. Read the “Identity Provider Integration” section for more details. Authorization is protected in the Harbor Database. SSL is enabled by default for access to Harbor API and UI and for core service to service communication in Harbor (the SSL for core service to service communication is a feature of Harbor v.2.0 that’s going to be released in late April 2020)
• Protect from bad actors gaining access to Harbor and modifying policies that would allow unsigned images or images with vulnerabilities to be pulled
  • Mitigation: Harbor has robust RBAC with specific actions that each user is entitled to do. RBAC and roles are fully contained within Harbor’s DB regardless of the authentication scheme used. IdP integration tells us reliably who the account is (pure auth) and Harbor will dictate what they are entitled to do. You can also associate LDAP or OIDC groups with a Harbor role to make management easier.
• Protect from bad actors to be able to pull an artifact they don’t have access to
  • Mitigation: Harbor auth will prevent someone with no access from connecting to Harbor, and project RBAC will control permissions in a project for Harbor authenticated users
• Protect from bad actors high-jacking the pluggable scanner and returning incorrect data about the vulnerability status of an artifact
  • We talk about the circumstances of this situation in the blast radius section below. An additional protection offered by Harbor includes the fact that the credentials pluggable scanners use to communicate with Harbor are rotated per scan job (Harbor engine rotates the accounts before it kicks off a scan job and the credentials are only valid for the duration of the job). This means an attack where the credentials are stolen will be short lived as those credentials are also short lived.
• Protect from bad actors high-jacking the webhook pipeline and returning incorrect data to subscribers
  • Mitigation: Harbor auth will prevent someone with no access from connecting to Harbor, and project RBAC will control permissions for webhook config. In addition, the webhook calls can be protected by SSL and Harbor can verify the remote certificate.

Non-Goals

• Harbor is not meant to be a content store for all existing artifacts such as legacy data types (Java WAR files, JARs, any other data type that’s not OCI-Compliant) unsuitable for the cloud-native world or those that are not OCI-compliant. The OCI specs are located at https://github.com/opencontainers/runtime-spec/blob/master/spec.md and at https://github.com/opencontainers/image-spec
• Harbor cannot protect itself from bad actors that want to DDOS Harbor in such a way that Kubernetes clusters can’t pull the images they need for workloads to run. If a bad actor is successful, end users will not know about it until Kubernetes workloads stop deploying because images can’t be pulled. Health monitoring and outside-in monitoring and synthetic transactions against Harbor and the firewall solution will help network operations discover there is an issue.
  • We rely on network administrators to protect Harbor using a hardware/software firewall or other global traffic manager tools. This is actually the recommended practice for both Harbor and Kubernetes environments
  • Harbor is also generally only available on the intranet
  • The Harbor administrator can also configure rate-limiting on nginx, protecting the Harbor API and User Interface from bad actors.

History

Harbor Through the Years

• In June 2014, Harbor started as a project within VMware’s China R&D organization, where it was leveraged for a handful of internal projects to manage container images. To allow more developers in the community to use and contribute to the project, VMware open-sourced Harbor in March of 2016 and it has steadily gained traction since.
• Harbor has been integrated into three commercial VMware products; vSphere Integrated Containers (VIC), Pivotal Container Services (aka PKS or VMware Enterprise PKS), and VMware Essential PKS.
• Many companies include Harbor in their own cloud native solutions and Harbor has 20+ Product Implementations (some of these products include VMware Tanzu Kubernetes Grid, Pivotal Container Service, Rancher and others)
• In April 2018, Harbor passed 4000 stars on GitHub and currently has 59 community contributors worldwide, 30 of which have made non-trivial contributions to the project.
• In July 2018, Harbor became a sandbox project of CNCF
• In Nov 2018, Harbor became an incubating project of CNCF
• In Oct 2019, Harbor has 7600 stars on GitHub with 200+ contributors worldwide.
• Harbor v1.10, released in Dec 2019, enabled the integration with third party pluggable scanners, opening up the ecosystem of cloud native security solutions that work with Harbor. Aqua Security, Anchore Engine & Enterprise and DoSec are the first security solutions to integrate with Harbor via our pluggable scanner service.
• Harbor v2.0, estimated to release in April 2020, will make Harbor fully OCI compliant, supporting any OCI-compliant cloud native artifact

Intended Use

Target Users

Harbor has three main personas. For a detailed listing of all personas and their permissions, we have documented them in managing users and user permissions by role.

• End User/Developer
  • A developer that has access to consume artifacts from Harbor. Typically they push and pull containers from Harbor using a docker-compliant client. They also have access to the Harbor user interface and API for container management, but their permissions are restricted.
• Project Administrator
  • A scoped version of a Harbor user with elevated permissions that has full access and permissions for a project. This includes project configuration, project policy, and user management.
• Enterprise Operator / Security Administrator
  • The Harbor administrator, responsible for the health and maintenance of the entire system. They install, configure, and maintain Harbor and its policy engine after installation. You can learn more about administration here

Use Cases

Harbor can be used as a repository for OCI-compliant artifacts. The only other artifact that we will support is Helm Charts through ChartMuseum. However, over time we will deprecate that capability as Helm compliance with OCI matures. Until then, we will enable users to push Helm Charts into Harbor both through ChartMuseum and as an OCI artifact. No other non-OCI-compliant file type can be pushed to Harbor. OCI-compliance is defined by the OCI working group as having adhered to both the OCI image spec as well the OCI runtime spec.

Specific uses cases include:

• Store docker container images and Helm charts within Harbor that can be pulled down to your environment at any given time
• The Harbor registry can be a key component of your CI/CD pipeline. You can use Harbor to push your artifacts so that Harbor can scan them for vulnerabilities before deploying those artifacts to your Kubernetes clusters. On top of that, Harbor policies like image retention and quota management can help make sure storage requirements don’t balloon. Retention and quota management policies are both configured at the project level, meaning they are applied consistently across all artifacts within a project. You can set different policies for each project.
• We understand that sometimes workloads are deployed at the edge, in branch offices, in other public cloud provider regions, etc. It is important that a local cache exists with images needed by those workloads. Harbor can operate in a hub-and-spoke model, where you use a central Harbor instance to scan all images and then replicate them to the locations where the Kubernetes clusters are located. Harbor can replicate hosted images and charts within Harbor to / from another Harbor instance or a 3rd-party on-premise or hosted registry (Dockerhub, GCR, ACR, ECR, Quay) through scheduled tasking
• Scan images for the latest vulnerabilities directly from within Harbor using one of our many vetted 3rd party OSS (Aqua Trivy, Anchor Engine, CoreOS Clair) or commercial scanners (Aqua CSP, Anchore Enterprise, DoSec). The plug-and-play nature of the scanning framework means you can use the default scanner, CoreOS Clair, that comes with the Harbor installer or pair one of our supported pluggable scanners with Harbor.
• Host CNABs, OPAs, or arbitrary cloud native artifacts that follow the OCI specifications and further gate access through whitelisting specific mediaTypes.
• Automatically garbage collect images in a non-blocking manner so that does not interfere with ongoing operations and does not have to put the registry into read-only mode

Project & Design

System Design

Harbor consists of a series of core proprietary components as well as several 3rd party open source software (Docker distribution, Clair, Notary, Chartmuseum, Helm etc.) running as containers to stand up a registry instance, capable of running on a wide array of popular Linux distros. You can find a complete listing of all Harbor dependencies here as well as a summary in the section directly below. Harbor can run as Docker compose as well as a Kubernetes deployment via Helm charts and has recently seen a 3rd release version in the form of an Operator (see our announcement here) This service is exposed to the end user via a web console and a rich set of APIs that allows you to use Docker CLI and other OCI-compatible client tooling to push and pull images. It has namespace management through a construct called ‘project’ and users gain access to their projects through full RBAC support. Policies enforcing image scanning and vulnerabilities (leveraging OSS project Trivy) and image signing (TUF Notary) are applied across these projects to enforce artifacts can only be pulled from the Harbor registry if they meet the policy requirements.

Harbor Architecture

Identity Provider Integration

Harbor supports different modes for authenticating users and managing user accounts. For production deployments, we recommend leveraging an Identity Provider (IdP). In cases where Harbor use cases require authenticating against many IdPs, we recommend using Dex as a connectivity component to other identity providers through Dex’s connectors.

• Database Authentication: You create and manage user accounts directly in Harbor. The user accounts are stored in the Harbor database.
  • User email and optional group information are used by Harbor to identify a user in the IdP
• LDAP/Active Directory Authentication: You connect Harbor to an external LDAP/Active Directory server. The user accounts are created and managed by your LDAP/AD provider.
  • User email and optional group information are used by Harbor to identify a user in the IdP. The username is also from the DN (according to the UID setting in Harbor)
• OIDC Provider Authentication: You connect Harbor to an external OIDC provider. The user accounts are created and managed by your ODIC provider.

RBAC and roles are fully contained within Harbor’s Database regardless of the authentication scheme used. IdP integration tells us reliably who the account is (pure auth) and Harbor will dictate what they are entitled to do. This means that IdP’s only perform authentication. Authorization is always performed by Harbor and the authorization configuration is in the Harbor Database.

For logging and forensics with IdPs, please visit the appropriate section.

Components & Dependencies

Complete details into the Harbor architecture were included in the Harbor due diligence document for Graduation.

Operations

Areas where keys / tokens need to be safely managed internally in Harbor are interaction with Docker Distribution (Harbor under the covers uses the Docker Distribution at https://github.com/docker/distribution as the toolset to pack, ship, store, and deliver content), image scanning, signing, and replication. The table in the section below breaks down all the keys and certs used in Harbor.

Harbor uses a proprietary token service to generate a token for the Docker client to push and pull images from the Docker distribution, please see this. This is the same token service used to generate tokens for robot accounts.

Using a proprietary token service was a legacy implementation detail of Harbor. We needed a token service that supports customized claims (for example see below) and data so we used our own. At the time we created this, we were not able to find a 3rd party token service that we could use. The token we create is a JWT token

Token Claims Example

"access": [
{
"type": "repository",
"name": "library/ubuntu",
"actions": [
"push",
"*",
"pull"
]}}

Image signing is done with Notary, which requires a certificate for HTTPS connection to Harbor. Customer provides the certificate (can be generated with OpenSSL) during installation which can then be updated at any time by replacing the .crt and .key in the designated paths and then restarting Harbor. The notary service within Harbor needs a separate cert for the actual signing of the image and this cert is generated by Harbor itself. There are 2 processes running in notary, notary-server and notary-signer. Notary-server is the only client for notary-signer. The certificate for notary-signer is generated each time the user installs Harbor, by install I mean both first-time installation and upgrade. In the event an administrator would like to replenish the notary-signer certificate the easiest way is to do it is by upgrading or re-installing Harbor. Notary handles the cryptographic signature of images, utilizing the notary-signer component. This page has more details on the Notary architecture.

Image scanning is achieved either via the built-in scanner (Clair is the only built-in scanner up to v1.10. With Harbor 2.0, Trivy and Clair are included as built-in scanners) or by pairing Harbor with a 3rd party image scanner that is exposed as an API endpoint to Harbor. The scanner is authorized for use with Harbor through a generated bearer token which allows the scanner to pull images across from Harbor. The bearer token cannot be refreshed with Harbor v1.10, however, with version Harbor v2.0 (to be released in April 2020), we are replacing the bearer token with robot accounts whose tokens can be refreshed at any time. Here is an overview on how robot accounts will be used in this scenario for Harbor 2.0. You can also read more about robot accounts here.

1. Harbor can run either on-demand or scheduled scan jobs. When a scan job is started, its job is to scan a collection of artifacts for vulnerabilities. The scanner will receive a pointer to these artifacts via API
2. The first step Harbor does before sending a scan request to the scanner (Trivy adaptor or Clair adaptor for example), is creating a set of credentials so that scanners can get the artifacts from Harbor. In order to create these credentials, Harbor will call an internal interface to create a robot account. The creds for the robot account are sent to the scanner to allow them to pull content from Harbor (using the pointers mentioned in step 1). So scanners are not using Harbor API to generate robot accounts, one is generated for them by Harbor engine.
3. This robot account has access scope for pull and scanner_pull which is a special access privilege created just for scanners, i.e. it will allow for skip the policy checker (CVEs blocking pulls etc) when called through the scanner pull.
   1. Only Harbor system administrators can add a new scanner.
   2. Scanners only get a robot account for projects they are enabled to run and only for the duration of the scan job
   3. Nobody else can access the scanner robot account besides scan jobs as other users would not know the robot account token (the token is not saved anywhere and for more info see the table below)
4. The actual authN/authZ for the robot accounts happens in harbor core components
5. Once the scan request is completed, the robot account is deleted by Harbor. The lifetime of these credentials would be the same as that of the scan job. This process is followed to control the lifetime of the credentials and tie them to the specific job they are trying to accomplish. A new robot account is created for the next scan job. We create a new robot account for each scan job since those accounts are shared with an external system, the scanning adapter. It is more secure to recreate them than depend on the scanning adapter to securely store them.

When replication is enabled in Harbor, the security administrator has to provide Harbor with credentials to be able to connect with the target registry. The target registry is any external registry that Harbor supports, including another Harbor instance, Google Registry, Azure Registry, AWS Registry, etc. In order to replicate artifacts to a target registry, the system needs an Access ID as well as the Access Secret (think of these as the username and password) for that target endpoint which is encrypted by AES and then stored securely in Harbor DB and encrypted via aes/base64. This access key will hold all necessary permissions for the target registry including access to storage etc. security administrators can update the Access ID and Access secret at any time.

Breakdown of Access, Tokens, and Creds in Harbor

• If a Harbor instance is compromised and the keys are leaked or stolen, the directive is to replace all the certificates using the methods outlined in the table below. This is the secure way to recover
• There is fate sharing involved in protecting some of these secrets. It really boils down to where they are stored. If they are stored in the file system, they can be compromised together. If stored in Kubernetes secrets, the same. If stored in a Database, compromising the database compromised all secrets stored in it. The Harbor team will explore integration with popular Vault solutions in the near future. That integration will ensure that in a Kubernetes environment, our secrets are securely stored in a Vault.
• In cases where credentials are rotated there is a secure way for a device with credential X to change to having credential Y such that all parties that trusted X now trust Y instead (and no longer trust X)
• All rotations of credentials are manual-driven by an action
• If the JWT tokens mentioned below are forged / incorrect / invalidated by the issuer (Harbor is the issuer), the consumer of these will get an access denied. Harbor can recreate them and the consumer would get a new secret token.
• Recommendations are made such that when deploying Harbor in production the security best practices to harden a Kubernetes cluster are followed. This way limitations of Kubernetes secrets are mitigated.
• For internal TLS, we leverage net/http lib of golang to establish TLS connection, which doesn’t support CRL. CA hierarchy can be supported per our understanding as long as the root and intermediate CA are pasted into one file, but we have not tested that. mutual auth is not fully supported. We had explored that in the past but found it unfriendly to users so decided to disable it in v2.0.0

Configuration and Set-Up

For detailed installation and configuration steps, the documents can be found here. In a nutshell, Harbor supports two deployment methods: Docker-Compose and a Helm Chart installation method for Kubernetes environments. We are also working on an operator for Harbor which is currently a work in progress (see our announcement here). The installation documentation linked above has the up-to-date information on installing Harbor.

By default, Harbor does not ship with certificates. Users must supply their own certificates for HTTPs access. In production environments, it is recommended practice to always use HTTPS (enabled by default). Ultimately, the Harbor installer has configuration options for http and https. It is up to the admin configuring installation to fill in the right set of details and supply the certificate. In addition, if you enable Content Trust with Notary for signing images, you must use HTTPS, you don’t have a choice. HTTPS requires use of certificates that are either signed by a trusted 3rd party CA or self signed. The steps to enable HTTPS in Harbor are detailed here. The project does support deploying with HTTP enabled for UI/API access but this is usually only for air-gapped non-production environments that do not have a connection to the external internet so it is less prone to man-in-the-middle attacks.

Furthermore, the latest release of Harbor v2.0, coming in April 2020, will optionally allow for all internal components of Harbor (core, jobservice, registry, registry controller) to communicate with each other via HTTPS-based RESTful API. If the administrator does not want to use SSL, all core services of Harbor (think pod to pod traffic) will communicate over http. We also updated our installers to make it easier for operators to configure these certificates:

• Docker-Compose: We added an option in the docker-compose installer (through the use of a script) for users to generate self-signed certificates. Users can also bring their own certificates and keys and configure them during Harbor installation.
• Helm Chart: Users will need to generate the certificates and keys and configure the secrets in the values.yaml configuration of the Chart.

We have not tested Harbor with a service mesh like Istio. Some of our users view Istio as a key component of application workloads versus an infrastructure solution like Harbor.

Default Configuration

• By default, Harbor does not ship with certificates. Users must supply their own certificates for HTTPs access. Since the default and recommended installation of Harbor uses HTTPS for secure connection to the registry and secure transmission of data, the installer has to provide the certificates.
• Default deployment also disables self-registration which would have allowed any user to sign up for an account with a valid email address. By disabling this feature, administrators have to manage end user accounts in Harbor and add them to the projects to which they shall be entitled. Only an administrator can enable self-registration again. To read more about RBAC and entitlement, look in the “Compensating Mechanisms” section.
• On the next release of Harbor v2.0, we will also be removing the ability for a user to query a password reset to prevent accidentally disclosing information when used maliciously for phishing.
• Harbor 2.0 will also allow for enforcing TLS communication between all internal components as described above.

Project Compliance

Harbor does not meet any compliance standards today. However, as we interact with more service providers that are putting Harbor in production (example by OVH Cloud here) and delivering it as a service, we understand there may be a need to acquire some of these certifications. We will continue exploring this vertical and their needs of Harbor and prioritize any requests. Currently, there haven’t been any regarding compliance certifications.

NOTE

There is only 1 public reference of a service provider today. However we are working privately with a few more service providers that don’t want to be named publicly yet. There are also some enterprises that are internally acting as service providers and have the same use cases.

Harbor has achieved the Silver level criteria for CII Best Practices, https://bestpractices.coreinfrastructure.org/en/projects/2095?criteria_level=1. The silver badge is displayed at https://github.com/goharbor/harbor/blob/master/README.md, Harbor’s primary repository.

Security Analysis

Attacker Motivations

There are multiple reasons why hackers would want to attack the Harbor registry as it is a critical component of the container delivery supply chain. Also refer to the “Security Goals” section above.

• Attackers can try to gain entry to hosted images and charts and either alter the content of these artifacts (a series of stacked layers) or inject vulnerabilities into the artifacts. This is the most critical vulnerability as it affects potentially production applications that are deployed from images hosted in Harbor. The motivation is to get applications running in Kubernetes to run arbitrary or vulnerable code.
• Attackers can try to gain access to projects outside of their permissions or elevate their rights to use advanced policies and features outside of their role-based access level.
• Another attacker vector includes DOS’ing the Harbor infrastructure, preventing artifacts from being pushed or pulled by developers. This essentially stalls the software delivery pipeline at the CI/CD layer and in Kubernetes environments. This can be a great thing for the attack chain since I can move laterally exploiting known vulns in deployed images while blocking devs from deploying patched images
• Harbor also holds sensitive information about its users such as credentials and image entitlements that would be of interest to hackers.
• In certain multi-tenant deployments where a Harbor instance is shared amongst different organizations or different customers, certain members have restricted access even within their own projects as they cannot view other members in the same project and logs of historical operations.
• Attackers can use package and image info for recon itself. They don’t have to change anything but just learn from one single source all the targets to go after in their attack planning

Predisposing Conditions

1. The keys are stored as plaintext, some credentials are set via environment variables in docker-compose deployment. So if an attacker gains root access of the Docker host, sensitive data will be at risk. This is an issue with docker-compose only.
   1. Recommendations are made such that when deploying Harbor in production the security best practices to harden a cluster are followed. For example we recommended a Kubernetes deployment in production, with all the proper security measures (node security, etcd SSL, encryption at rest, RBAC policies, namespaces, and more). We have not tested with any vault solutions (even though Kubernetes interoperates well with many vault vendors), but this is on our roadmap so that we can make recommendations in that area as well. In addition, many customers are deploying a single enterprise app per cluster. This means they are deploying an entire Kubernetes cluster for an app like Harbor or Harbor in conjunction with other infrastructure services.This reduces the attack surface as the cluster has no other tenants and can be properly hardened.
2. Harbor supports authn integration with external services such as LDAP, OIDC provider, so compromising the external IDP will give attackers access to Harbor. Compromise of the IDP results in loss of access control integrity for Harbor.

Expected Attacker Capabilities

There are currently no known vulnerabilities and operators have the ability to use TLS for communication between all internal components. TLS is not enforced, but the Harbor team has documented the risks of an insecure setup and made it harder for operators to deploy insecurely. Areas that are potentially susceptible to attacks would be credential handling, robot accounts, replication to other registries, and token-based single sign-on. However, all of these use some form of token based authentication and we assume the attacker cannot easily break existing cryptographic algorithms like AES or SHA256

Attack Risks and Effects

Harbor allows for secure authentication through either local DB, LDAP/AD, or OIDC connection via a vetted external vendor that you must select during the initial phase of installation. If the IdP supports it, you can also have the additional security of MFA, but Harbor does not provide MFA natively. It does not allow cutting across these after deployment, ie. you cannot change to LDAP or OIDC after you have created local users in Harbor DB. There are also different types of users (system administrator, project administrator, developer, limited guest, and guest) with different privileges to minimize the effects of accidental operations.

We currently don’t have an option for multi-signature trust or admission webhook that can kick off an approval workflow in a service management tool. We understand those options would make our user accounts more powerful and secure and we can look into it in the future.

Security Degradation

Exploiting vulnerabilities in user management and authentication can lead to maliciously changing users’ permission levels, accessing restricted artifacts, modifying artifacts to introduce malware/vulnerabilities and deleting artifacts or bringing the registry down. If an attacker gains access to artifacts on the registry, he can pull these down to their environment and alter the content before re-pushing to the registry. If an attacker intercepts tokens used for communicating with docker clients, he can break artifact pushing and pulling and render the registry unusable. If hackers gain access to the Harbor database, he could manipulate critical metadata such as adding or deleting existing users or change the authentication mode or information related to the OIDC endpoint or LDAP server. This is why there are multiple roles with varying access levels and previous exploits around self registration and permissions elevation have all been patched.

Blast Radius and Recovery

In this section, we will discuss the blast radius when bad things happen. Cases where an attacker is likely to be able to gain additional access after a compromise are listed below. A key component of this section is to also identify the steps a Harbor administrator should take in such a situation.

If a customer has enabled Notary, image provenance is not broken unless in the case where the notary-signer component of Notary is compromised (this case is outlined in the table below). In all other cases, images that are signed by Notary will continue to assure their users of the origin of the image. In the early sections of this document we also talked about other benefits Notary brings to Harbor.

Mandatory reading prior to this section is the “Breakdown of Access, Tokens, and Creds in Harbor” section. You can view an index overview of the blast radius and recovery table below. It is color coded in terms of risk during a successful attack.

Blast Radius Index

Compensating Mechanisms

Harbor only uses proven cryptographic protocols without known serious weaknesses and tries to adhere to NIST recommended practices where applicable. For example, all instances of SHA-1 in the project have either been removed or replaced with SHA-2/SHA-3 in Harbor v1.10. Harbor v2.0 can now use TLS for all communication between internal components so that no cryptographic hashes are transmitted over plain HTTP at any point during its usage. Admin can also mandate image scan on push of artifacts to Harbor. In addition, policy can further prevent any images that are either unscanned or scanned with vulnerabilities of a certain criticality or higher from being pulled down.

Note: Image digests qualify as sha256. We depend on docker on how image digests are calculated.

• Harbor uses TLS v1.2
  • Notary, clair, docker registry, trivy can be configured to require TLSv1.2.
  • Chartmuseum is not configurable, and we are going to deprecate it in the next 2 releases
• Nginx ciphers: ssl_ciphers ‘!aNULL:kECDH+AESGCM:ECDH+AESGCM:RSA+AESGCM:kECDH+AES:ECDH+AES:RSA+AES:’;
• Password Hash: SHA256 with PDKDF2
• Credentials Encryption: AES128
• Signing Token: RS256
• Harbor is using SHA-2/SHA-3 throughout

Compartmentalization

Cure53, our security pentester also noted this in their report:“It is worth pointing out that even if one part of the application is compromised, it does not necessarily mean that the server is also directly prone to successful attacks. The separation of duty principle is implemented in a praiseworthy manner”

Different components and services in Harbor have their own tasks and areas of scope. For example, in the case of robot accounts, because the permissions are encoded in the robot token (used as password), if an attacker hacked the private key they can modify the permission of the robot account. But the attacker is not able to compromise another area of Harbor without successfully performing another attack. Meaning the attacker cannot gain project admin permissions by leveraging this attack. In another example, if a local Harbor user’s credentials were leaked (when not using an IdP), that does not enable the attacker to elevate its privilege.

In the Blast Radius and Recovery section we also talk about what happens when a successful attack is performed.

RBAC

Harbor has implemented its own RBAC. Today, this is not a pluggable component of our architecture, but in the future we can look into a dependency to OPA or other policy enforcement libraries.

• Users have different access and ability to do certain actions depending on the role they are assigned to in a project. The user permissions per role are explained here.
• Entitlement: Each user assigned to a project can have one of many Roles in Harbor (limited guest, guest, developer, master, project admin). Roles are scoped to a project, so in a project a user can only have one role. User permissions per role are documented here and are the foundation of project RBAC in Harbor

Forensics

• When an external IdP is used, logging and forensics around failed auth is logged in the IdP
• When Harbor built-in authentication is used instead of an IdP (this is rare in production environments), Harbor will log failed authentication attempts
• When the Harbor server returns 403 (forbidden), it is included in the Harbor project’s access log

Secure Development Practices

Development Pipeline

All new features must pass human review as well as automated testing. The project implements continuous integration through the use of Travis CI and Github actions, which runs on every single commit to DEV and Master https://travis-ci.org/goharbor/harbor. PRs also have to go through unit testing for backend services and UI tests.

Tooling

• Golint and Govet for managing compiler warnings, coding style, and correctness
• Gosec is used before each release as part of the internal security review
• Black Duck Binary analysis is run every night for application security testing used to find security vulnerabilities that can make an application susceptible to attack

Communication Channels

The recommended channel for community support is via Slack.

• CNCF Slack
  • The #Harbor channel has over 700 members.
  • The #Harbor-Dev channel has over 200 members.
  • Multiple working group channels exist for engineering discussions or special projects.
• WeChat
  • There are 3 Harbor groups with membership of 476, 486, and 46 people, respectively.
• CNCF Lists
  • Harbor users has 88 members with 68 threads.
  • Harbor developers has 71 members with 56 threads.
  • A separate mailing list exists for security related discussions and it is documented in a later section of this document.
• New vulnerability issues can be reported to a private mailing list that includes only the Harbor Security Team as per our policy at https://github.com/goharbor/harbor/security/policy

Harbor also has a dedicated community manager, Jonas Rosland from VMware.

Bugs, Issues, and Feature requests can be reported via GitHub issues. Most of the issues get replies within a week. Feature prioritization and severity of issues help the Harbor team decide when to fix an issue. For example, high priority bugs (security issues for example) are fixed immediately (in the next release of Harbor).

Ecosystem

Please see the Goals section at the beginning of this document to understand how Harbor aligns with the cloud native ecosystem.

Security Issue Resolution

Responsible Disclosures Process

Harbor has a comprehensive vulnerability and security policy that is outlined at https://github.com/goharbor/harbor/security/policy. We have already battle tested this policy and our Incident Response more than a few times and it has worked very well for both the Harbor team, our users, as well as the security researchers that reported vulnerabilities and attack vectors. You can view our published advisories at https://github.com/goharbor/harbor/security/advisories?state=published.

Our policy states that anyone who finds a vulnerability should report it to the Harbor security team through cncf-harbor-security@lists.cncf.io with the details of the vulnerability. The email will be fielded by the Harbor Security Team, which is made up of Harbor maintainers who have committer and release permissions. Emails will be addressed within 3 business days, including a detailed plan to investigate the issue and any potential workarounds to perform in the meantime.

Incident Response

If a vulnerability is acknowledged and the timeline for a fix is determined, the Security Team will work on a plan to communicate with the appropriate community, including identifying mitigating steps that affected users can take to protect themselves until the fix is rolled out. The Security Team will also create a CVSS using the CVSS Calculator. The Security Team makes the final call on the calculated CVSS; it is better to move quickly than making the CVSS perfect. The CVE will initially be set to private and Security Team will provide early disclosure of the vulnerability by emailing the cncf-harbor-distributors-announce@lists.cncf.io mailing list. A public disclosure date is then negotiated by the Harbor Security Team, the bug submitter, and the distributors list. Once the fix is confirmed, the Security Team will patch the vulnerability in the next patch or minor release, and backport a patch release into all earlier supported releases.

Security Testing

Harbor has undergone two security and penetration testing exercises. The first was funded and executed by VMware in August 2019, and the latest funded by CNCF and executed in October 2019 by external security researcher Cure53.

The Cure53 report is already published in our GitHub repo and we have already addressed and fixed all the critical vulnerabilities in the v1.10 release of Harbor (Dec 2019). At the time we published the report, we issued security advisories, and made sure our users were notified to patch their Harbor instances. The key results of the report were:

• 1 Critical Vulnerability
• 3 High Vulnerabilities
• 1 Medium Vulnerability
• 1 Low Issue

To emphasize the first few sentences from the report’s conclusion (A total budget spent

on the project amounted to eighteen person-days split across 7 security researchers):

• [-ve] Cure53 managed to spot six security-relevant findings. Four were classified as security vulnerabilities and two should be seen as general weaknesses
• [-ve] While the Harbor software made a well-rounded impression, the results in terms of security posture are not yet optimal, calling for more hardening work across various areas
  • [Commentary by Harbor Security Team] We are addressing all high and critical vulnerabilities found in Harbor v1.10
• [-ve] Cure53 demonstrated that Harbor suffers from a number of security issues that are usually found in completely untested applications
  • [Commentary by Harbor Security Team] We are addressing all high and critical vulnerabilities found in Harbor v1.10
• [+ve] Can certainly call Harbor a modern web application that follows various up-to-date and modern security practices
• [+ve] While the words above may sound harsh and the found vulnerabilities are in fact severe, they do not signify that the Harbor complex is in a particularly bad shape. For example, the number of findings is very low and the overall pentest results and general impressions about the codebase are rather positive. Lack of further “Critical” input sanitization issues is definitely a good indicator as well.
• [+ve] Furthermore, Cure53 feels obliged to state that the Harbor code is clean, easy to follow and yields itself well to auditing. Similarly, the platform is built upon containerized microservices that raise the security level in a non-negligible way. It is worth pointing out that even if one part of the application is compromised, it does not necessarily mean that the server is also directly prone to successful attacks. The separation of duty principle is implemented in a praiseworthy manner. It is also important to note that Cure53 did not find flaws in the areas that the Harbor was most worried about as a result of previous vulnerabilities residing there. In fact, these were almost completely issue-free. Access control via RBAC with its different user models and groups is tightly implemented, uploaded files are handled safely and dangerous features such as webhooks are secure and robust
  • The Areas that the Harbor team was most worried were: web UI, injection attacks, code execution attacks, RBAC & multitenancy & privilege escalation, integrity tests, programmatic DoS, service robustness, DB hardening, security of k8s deployment
• [+ve] To conclude, Cure53 feels that Harbor is fully legitimized in calling its software compound secure and trustworthy. The examined scope items exhibit strong security posture, even though this CNFC-funded project revealed that some problems exist

Overall, the security pentest report indicates that with a few fixes (which were already addressed in v1.10) Harbor will achieve limiting the attack surface, taking appropriate care of user-supplied input with security-driven best practices, as well as - to a certain extent - the usage of the Go language ecosystem.

Roadmap

Project Next Steps

The Harbor project roadmap includes plans for better support of additional cloud native artifacts like CNABs and OPAs, better image distribution through proxy cache and integration with p2p tools like Alibaba Dragonfly and Uber Kraken, as well a Harbor operator that deploys in HA mode for better management of multiple Harbor instances.

You can read more about the roadmap and release mechanics in this section, which includes links to our publicly disclosed roadmap. More information on the project’s committed roadmap and key customer requests that are being addressed are located in the up-to-date project board.

There are currently no CNCF Requests. However, we will continue to leverage the CNCF benefit of getting external security researchers to security-test our product on the cadence recommended by CNCF, thus increasing the security of Harbor and the cloud native ecosystem.

Appendix

Known Issues Over Time

Harbor has undergone internal as well external penetration tests by 3rd party security vendors and have since patched all discovered vulnerabilities. Other past vulnerabilities found by our user have all been fixed and tracked here.

CII Best Practices

Harbor has achieved the CII silver badge and is in the process of working towards attaining a gold badge. The silver badge is displayed at https://github.com/goharbor/harbor/blob/master/README.md, Harbor’s primary repository.

Case Studies

We would like to point to the graduation due diligence document that includes a comprehensive list of real-world use cases, adopters, and production customers of Harbor. During the TOC presentation, we also had two production customer testimonials from Hyland Software and vPay. Project adopters are also captured in the Harbor repo at https://github.com/goharbor/harbor/blob/master/ADOPTERS.md

An abbreviated bullet list of adopters and case studies includes:

1. Hyland Software, a leader in content services platforms is using Harbor in production:
   1. Hyland has approximately 2400 Tags spread across 670 images in 175 different projects. Approximately 10% of which store helm charts in Harbor
   2. Hyland’s Harbor instance currently stores about 2.5 Terabytes of container and chart data
   3. Approximately 1000 Developers at Hyland interact with Harbor on a day-to-day basis
2. vPay, a leading payment solutions provider is using Harbor in production with 150+ images and 95GB of total storage utilized. Harbor is currently replicated across two sites in the United States using the Harbor-to-Harbor replication feature
3. OVH, a large cloud provider in France, has built a managed service for Harbor as part of their kubernetes offering. OVH chose Harbor over other competitors due to its capabilities in image scanning, content trust, fine-grain RBAC among others. Specifically, Harbor is deployed as a managed structure built seamlessly on top of other infrastructure services such as Nova and Swift. There are currently 600+ beta testers of the managed service, with 100 day-to-day active ones. OVHcloud also started developing a Harbor operator geared toward more complex, stateful deployments and have since open-sourced this work under the goharbor repo. They have also published a blog post documenting their journey with Harbor and why they chose Harbor as the technology to build their registry as a service capability.
4. Netease (One of the largest internet companies in China) 4. Uses Harbor as the container repository in NetEase Qingzhou Microservice and has it deployed on K8s with an HA deployment. 5. There are more than 2000 nodes in the K8s cluster with one Harbor cluster instance serving 10000-30000 containers. 6. Harbor is used in Qingzhou serving NetEase cloud music, NetEase Kaola and other internal and external services
5. Qihoo 360 (Large internet security company in China) is using Harbor to serve a Kubernetes cluster of 1000 nodes and 10,000 PODs
6. DIDI (Ride sharing company operating in Asia) has 20 TB of container images hosted on Harbor
7. A company that wishes not to be disclosed has Harbor v1.5 in production with 35 thousand images. They plan to upgrade to Harbor v1.9 by the end of October

Competition

There are numerous registries available for developers and platform architecture teams to leverage. We’ve analyzed the various options available and summarized them in the link below in the form of a comparison guide. This guide provides our best estimation of features and functionality available on other container registry platforms. Should you find mistakes please submit a PR to update the table.

https://github.com/goharbor/harbor/blob/master/docs/1.10/build-customize-contribute/registry-landscape.md

As you can see from the table, Harbor scores very well when compared with other cloud native repositories. This speaks to the community of Harbor and its ability to deliver features that are important to our users in a timely manner.

Missing Features

There are a couple of features that Harbor is missing as compared to the competition and we have identified them below. The Harbor community is working on delivering these features in the next 6-12 months. Look for more details in the roadmap section in this document as well as our roadmap on our project board.

1. Proxy Cache support → To be delivered in early H2CY2020 with Harbor v2.1
2. Ability to have a single repository for all cloud native artifacts (from Kubernetes operators, CNAB bundles, to storing other platform libraries for golang, java, etc for development needs) → Being delivered with Harbor v2.0 in April 2020