Skip to main content
Version: Devel

External Secrets

External Secrets is a CNCF Sandbox project, accepted in 2022 under the sponsorship of TAG Security.

About​

The External Secrets Operator (ESO) is a Kubernetes operator that enhances secret management by decoupling the storage of secrets from Kubernetes itself. It enables seamless synchronization between external secret management systems and native Kubernetes Secret resources.

ESO supports a wide range of backends, including:

…and many more. For a full and up-to-date list of supported providers, refer to the official External Secrets documentation.

Integration with PostgreSQL and CloudNativePG​

When it comes to PostgreSQL databases, External Secrets integrates seamlessly with CloudNativePG in two major use cases:

  • Automated password management: ESO can handle the automatic generation and rotation of database user passwords stored in Kubernetes Secret resources, ensuring that applications running inside the cluster always have access to up-to-date credentials.

  • Cross-platform secret access: It enables transparent synchronization of those passwords with an external Key Management Service (KMS) via a SecretStore resources. This allows applications and developers outside the Kubernetes cluster—who may not have access to Kubernetes secrets—to retrieve the database credentials directly from the external KMS.

Example: Automated Password Management with External Secrets​

Let’s walk through how to automatically rotate the password of the app user every 24 hours in the cluster-example Postgres cluster from the quickstart guide.

Important

Before proceeding, ensure that the cluster-example Postgres cluster is up and running in your environment.

By default, CloudNativePG generates and manages a Kubernetes Secret named cluster-example-app, which contains the credentials for the app user in the cluster-example cluster. You can read more about this in the “Connecting from an application” section.

With External Secrets, the goal is to:

  1. Define a Password generator that specifies how to generate the password.
  2. Create an ExternalSecret resource that keeps the cluster-example-app secret in sync by updating only the password and pgpass fields.

Creating the Password Generator​

The following example creates a Password generator resource named pg-password-generator in the default namespace. You can customize the name and properties to suit your needs:

apiVersion: generators.external-secrets.io/v1alpha1
kind: Password
metadata:
  name: pg-password-generator
spec:
  length: 42
  digits: 5
  symbols: 5
  symbolCharacters: "-_$@"
  noUpper: false
  allowRepeat: true

This specification defines the characteristics of the generated password, including its length and the inclusion of digits, symbols, and uppercase letters.

Creating the External Secret​

The example below creates an ExternalSecret resource named cluster-example-app-secret, which refreshes the password every 24 hours. It uses a Merge policy to update only the specified fields (password, pgpass, jdbc-uri and uri) in the cluster-example-app secret.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: cluster-example-app-secret
spec:
  refreshInterval: "24h"
  target:
    name: cluster-example-app
    creationPolicy: Merge
    template:
      metadata:
        labels:
          cnpg.io/reload: "true"
      type: kubernetes.io/basic-auth
      data:
        password: "{{ .password }}"
        pgpass: "cluster-example-rw:5432:app:app:{{ .password }}"
        jdbc-uri: "jdbc:postgresql://cluster-example-rw.default:5432/app?password={{ .password }}&user=app"
        uri: "postgresql://app:{{ .password }}@cluster-example-rw.default:5432/app"
  dataFrom:
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: Password
          name: pg-password-generator

The label cnpg.io/reload: "true" ensures that CloudNativePG triggers a reload of the user password in the database when the secret changes.

Verifying the Configuration​

To check that the ExternalSecret is correctly synchronizing:

kubectl get es cluster-example-app-secret

To observe the password being refreshed in real time, temporarily reduce the refreshInterval to 30s and run the following command repeatedly:

kubectl get secret cluster-example-app \
  -o jsonpath="{.data.password}" | base64 -d

You should see the password change every 30 seconds, confirming that the rotation is working correctly.

Managing the Password of a DatabaseRole​

The example above updates cluster-example-app, a secret CloudNativePG created itself. The same generator can instead own the password of a role you declare with a DatabaseRole, by pairing it with mode: secret: External Secrets generates and rotates the password, and CloudNativePG reads that secret and applies it to PostgreSQL.

The secret must be of type kubernetes.io/basic-auth and hold a username key carrying the role name, alongside password. It must also carry the cnpg.io/reload: "true" label: the instance manager is deliberately not allowed to watch secrets, so the operator watches the labeled ones on its behalf and signals the change through the status of the role. Without that label a rotation is never noticed.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: stevie-password
spec:
  refreshInterval: "24h"
  target:
    name: stevie-eso-password
    creationPolicy: CreateOrMerge
    template:
      metadata:
        labels:
          cnpg.io/reload: "true"
      type: kubernetes.io/basic-auth
      data:
        # Must match the role name in the DatabaseRole below
        username: "stevie"
        password: "{{ .password | toString }}"
        pgpass: "cluster-example-rw:5432:app:stevie:{{ .password }}"
        jdbc-uri: "jdbc:postgresql://cluster-example-rw.default:5432/app?password={{ .password }}&user=stevie"
        uri: "postgresql://stevie:{{ .password }}@cluster-example-rw.default:5432/app"
  dataFrom:
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: Password
          name: pg-password-generator
---
apiVersion: postgresql.cnpg.io/v1
kind: DatabaseRole
metadata:
  name: stevie
spec:
  cluster:
    name: cluster-example
  name: stevie
  login: true
  password:
    mode: secret
    secret: stevie-eso-password
warning

Do not reach for mode: external here. That mode tells the operator to stop managing the password altogether, so the generated secret would never be applied and the role would be left with no password at all.

The same approach extends to any custom secret or PostgreSQL user you create and want to rotate regularly.

Example: Integration with an External KMS​

One of the most widely used Key Management Service (KMS) providers in the CNCF ecosystem is HashiCorp Vault. Although Vault is licensed under the Business Source License (BUSL), a fully compatible and actively maintained open source alternative is available: OpenBao. OpenBao supports all the same interfaces as HashiCorp Vault, making it a true drop-in replacement.

In this example, we'll demonstrate how to integrate CloudNativePG, External Secrets Operator, and HashiCorp Vault to automatically rotate a PostgreSQL password and securely store it in Vault.

Important

This example assumes that HashiCorp Vault is already installed and properly configured in your environment, and that your team has the necessary expertise to operate it. There are various ways to deploy Vault, and detailing them is outside the scope of CloudNativePG. While it's possible to run Vault inside Kubernetes, it is more commonly deployed externally. For detailed instructions, consult the HashiCorp Vault documentation.

Continuing from the previous example, we will now create the necessary SecretStore and PushSecret resources to complete the integration with Vault.

Creating the SecretStore​

In this example, we assume that HashiCorp Vault is accessible from within the namespace at http://vault.vault.svc:8200, and that a Kubernetes Secret named vault-token exists in the same namespace, containing the token used to authenticate with Vault.

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://vault.vault.svc:8200"
      path: "secrets"
      # Specifies the Vault KV secret engine version ("v1" or "v2").
      # Defaults to "v2" if not set.
      version: "v2"
      auth:
        # References a Kubernetes Secret that contains the Vault token.
        # See: https://www.vaultproject.io/docs/auth/token
        tokenSecretRef:
          name: "vault-token"
          key: "token"
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-token
data:
  token: aHZzLioqKioqKio= # hvs.*******

This configuration creates a SecretStore resource named vault-backend.

Important

This example uses basic token-based authentication, which is suitable for testing API, and CLI use cases. While it is the default method enabled in Vault, it is not recommended for production environments. For production, consider using more secure authentication methods. Refer to the External Secrets Operator documentation for a full list of supported authentication mechanisms.

info

HashiCorp Vault must have a KV secrets engine enabled at the secrets path with version v2. If your Vault instance uses a different path or version, be sure to update the path and version fields accordingly.

Creating the PushSecret​

The PushSecret resource is used to push a Kubernetes Secret to HashiCorp Vault. In this simplified example, we'll push the credentials for the app user of the sample cluster cluster-example.

For more details on configuring PushSecret, refer to the External Secrets Operator documentation.

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: pushsecret-example
spec:
  deletionPolicy: Delete
  refreshInterval: 24h
  secretStoreRefs:
    - name: vault-backend
      kind: SecretStore
  selector:
    secret:
      name: cluster-example-app
  data:
    - match:
        remoteRef:
          remoteKey: cluster-example-app

In this example, the PushSecret resource instructs the External Secrets Operator to push the Kubernetes Secret named cluster-example-app to HashiCorp Vault (from the previous example). The remoteKey defines the name under which the secret will be stored in Vault, using the SecretStore named vault-backend.

Verifying the Configuration​

To verify that the PushSecret is functioning correctly, navigate to the HashiCorp Vault UI. In the kv secrets engine at the path secrets, you should find a secret named cluster-example-app, corresponding to the remoteKey defined above.