Kubernetes Secrets
Use Vault secrets in Kubernetes cluster with External Secrets Operator
External Secrets Operator (ESO) lets you call Vault APIs and store Vault secrets as Kubernetes secrets.
This guide provides an example of how to integrate Vault with ESO. You can follow this example in your Kubernetes cluster using your free Pangea account.
Install Helm
An easy way to install the External Secrets Operator (ESO) is using its Helm chart repository. To do this, you'll need to have Helm installed on your system.
To check if Helm is installed, run the following command:
helm version
If you don't have Helm installed, follow the official Helm documentation to install it on your system.
For example, if you're using macOS, install Helm with Homebrew:
brew install helm
Install External Secrets Operator
Once Helm is installed, you can add the ESO chart repository with the following command:
helm repo add external-secrets https://charts.external-secrets.io
"external-secrets" has been added to your repositories
Install ESO in its own namespace. ESO runs as a deployment with elevated privileges and can create, read, and update secrets in all namespaces:
helm install external-secrets external-secrets/external-secrets \
-n external-secrets --create-namespace
NAME: external-secrets
LAST DEPLOYED: Sun Jun 30 14:08:39 2024
NAMESPACE: external-secrets
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
external-secrets has been deployed successfully in namespace external-secrets!
To begin using ExternalSecrets, you will need to set up a SecretStore
or ClusterSecretStore resource (for example, by creating a 'vault' SecretStore).
More information on the different types of SecretStores and how to configure them
can be found in our Github: https://github.com/external-secrets/external-secrets
kubectl get deployments -n external-secrets
NAME READY UP-TO-DATE AVAILABLE AGE
external-secrets 1/1 1 1 20m
external-secrets-cert-controller 1/1 1 1 20m
external-secrets-webhook 1/1 1 1 20m
Integrate External Secrets Operator with Vault
SecretStore and ExternalSecret objects
You can use the ESO SecretStore object with a Webhook backend to request secrets from Vault APIs.
You can specify which secret(s) to request from Vault and how to save them as a Kubernetes secret using the ExternalSecret object.
SecretStore and ExternalSecret are scoped to a namespace.
You can use cluster-scoped ClusterSecretStore as a centralized secret backend that can be referenced by ExternalSecrets from all namespaces in the cluster.
You can also manage namespaced ExternalSecrets centrally by using cluster-scoped ClusterExternalSecret .
The following explains how to use SecretStore and ExternalSecret to save one or more Vault secrets as a Kubernetes secret.
Create example namespace
To keep the resources created with this guide separate from other deployments, create a namespace in your Kubernetes cluster:
kubectl create namespace external-secrets-example
namespace/external-secrets-example created
To follow the code examples in this guide, set the example namespace as the default for all subsequent kubectl commands:
kubectl config set-context --current --namespace external-secrets-example
kubectl config view | grep namespace
namespace: external-secrets-example
Save Vault token in Kubernetes secret
To authorize requests to Vault APIs, copy a Vault service token and save it in a Kubernetes secret:
kubectl create secret generic pangea-vault-credentials \
--from-literal token=pts_gktfz4...v3pgws
You can verify that your secret was created and populated correctly:
kubectl get secrets
NAME TYPE DATA AGE
pangea-vault-credentials Opaque 1 23m
kubectl get secrets pangea-vault-credentials -o yaml
apiVersion: v1
data:
token: cHRzX2...NwZ3dz
kind: Secret
metadata:
name: pangea-vault-credentials
namespace: external-secrets-example
...
type: Opaque
Kubernetes secret values are base64-encoded. On Linux and macOS, you can use the base64 command-line utility to decode the values and display them in readable form:
kubectl get secrets pangea-vault-credentials --template={{.data.token}} \
| base64 --decode
pts_gktfz4...v3pgws
Save Vault secret in Kubernetes secret
The following example shows how to fetch a single Vault secret value and make it available as a Kubernetes secret.
Create SecretStore for Vault /v1/get APIs
You can use Vault's /v1/get API endpoint to fetch a single Vault item.
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: pangea-vault-get-secret-store
namespace: external-secrets-example
spec:
provider:
webhook:
url: "https://vault.aws.us.pangea.cloud/v1/get"
method: POST
body: '{ "id": "{{ .remoteRef.key }}"}'
result:
jsonPath: "$.result"
headers:
Content-Type: application/json
Authorization: "Bearer {{ print .auth.token }}"
secrets:
- name: auth
secretRef:
name: pangea-vault-credentials
This manifest instructs ESO to do the following:
- Request a Vault secret by its ID, provided as a parameter by an ExternalSecret object (L9-11).
- Authorize the request using a Vault service token saved in the
pangea-vault-credentialsKubernetes secret (L16-20). - Return the result of the request back to the ExternalSecret object (L13).
kubectl apply -f ./pangea-vault-get-secret-store.yaml
secretstore.external-secrets.io/pangea-vault-get-secret-store created
You can verify the addition of the new SecretStore by listing the existing SecretStore objects in the namespace:
kubectl get secretstores.external-secrets.io
NAME AGE STATUS CAPABILITIES READY
pangea-vault-get-secret-store 4h9m Valid ReadOnly True
Once the SecretStore is created, you can retrieve specific Vault secrets and save them as Kubernetes secrets.
Create Vault secret in Pangea User Console
Navigate to Vault's Secrets & Keys in the Pangea User Console. Add a secret and save it in my-app folder:
Copy the new secret ID using the Copy button next to it. You will need this ID in the next step.
Create ExternalSecret for Vault secret
This ESO ExternalSecret will use the SecretStore created for Vault's /v1/get API. It will retrieve a single Vault secret from the SecretStore and save it as a Kubernetes secret.
Use your new secret ID copied from the Pangea User Console in the previous step for the key value on line 16 below.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: pangea-vault-get-external-secret
namespace: external-secrets-example
spec:
refreshInterval: "5m"
secretStoreRef:
name: pangea-vault-get-secret-store
kind: SecretStore
target:
name: pangea-vault-get-secret
template:
templateFrom:
- target: Data
literal: |-
{{ $result := .secret | fromJson }}
{{- $result.name -}}: "{{ $result.current_version.secret }}"
data:
- secretKey: secret
remoteRef:
key: pvi_k6zrjhvcncyy2mdeonq3kh3u3vvf3prv
This manifest instructs ESO to:
- Specify a Vault secret by its ID (L22).
- Pass the Vault ID to the
pangea-vault-get-secret-storeSecretStore (L8-10) as the.remoteRef.keyparameter (L21-22). - Save the Vault secret returned from the SecretStore into
pangea-vault-get-secretKubernetes secret (L11-12). - Save the Vault secret name and value as a key/value pair in the
datafield of thepangea-vault-get-secretsecret (L13-18). - Re-obtain the Vault secret every five minutes (L7).
kubectl apply -f ./pangea-vault-get-external-secret.yaml
externalsecret.external-secrets.io/pangea-vault-get-external-secret created
Verify ExternalSecret
Verify that your ExternalSecret is working correctly by checking its status:
kubectl get externalsecrets.external-secrets.io
NAME STORE REFRESH INTERVAL STATUS READY
pangea-vault-get-external-secret pangea-vault-get-secret-store 5m SecretSynced True
If STATUS is not SecretSync or READY is not True, you are not saving Vault data in the Kubernetes secret. This might occur during setting up the ExternalSecret or due to changes you made in Vault.
For example, if the Vault item cannot be fetched or the SecretStore is not found, the STATUS field will show SecretSyncedError and the READY field will be populated with False:
... STATUS READY
... SecretSyncedError False
When you make changes in your ExternalSecret manifest, you can apply them with:
kubectl apply -f ./<external-secret-file-name>.yaml
However, to immediately verify changes made in the SecretStore or Vault, you might need to delete the ExternalSecret object and re-create it:
kubectl delete externalsecrets.external-secrets.io \
<external-secret-name> \
&& kubectl apply -f ./<external-secret-file-name>.yaml
Verify Kubernetes secret created by ExternalSecret
kubectl get secrets
NAME TYPE DATA AGE
pangea-vault-credentials Opaque 1 23m
pangea-vault-get-secret Opaque 1 33m
kubectl get secrets pangea-vault-get-secret \
--template='{{index .data "my-app-secret"}}' \
| base64 --decode
my app secret value
To check if the secret value is being synced, select the secret on Vault's Secrets & Keys page in the Pangea User Console, click Rotate Now, and update the value:
After the period specified in the ExternalSecret's refreshInterval, the value of the Kubernetes secret should be updated:
kubectl get secrets pangea-vault-get-secret \
--template='{{index .data "my-app-secret"}}' \
| base64 --decode
my app secret value updated
To expedite the update process, you can recreate the ExternalSecret.
Save multiple Vault secrets in Kubernetes secret
The following example shows how to fetch multiple Vault secrets and save them as key/value pairs in a Kubernetes secret.
Create SecretStore for Vault /v1/list APIs
You can use Vault's /v1/list API endpoint to fetch multiple Vault items that meet specific filter criteria.
In this guide, we will use a folder in Vault to group secrets.
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: pangea-vault-list-secret-store
namespace: external-secrets-example
spec:
provider:
webhook:
url: "https://vault.aws.us.pangea.cloud/v1/list"
method: POST
body: '{ "filter": { "folder": "{{ .remoteRef.key }}" }, "include_secrets": true }'
result:
jsonPath: "$.result"
headers:
Content-Type: application/json
Authorization: "Bearer {{ print .auth.token }}"
secrets:
- name: auth
secretRef:
name: pangea-vault-credentials
This manifest instructs ESO to do the following:
- Request a list of Vault secrets located in a folder, specified as a parameter by an ExternalSecret (L9-11).
- Authorize the request using a Vault service token saved in the
pangea-vault-credentialsKubernetes secret (L16-20). - Return the entire
resultobject from the/v1/listendpoint response (L13).
kubectl apply -f ./pangea-vault-list-secret-store.yaml
secretstore.external-secrets.io/pangea-vault-list-secret-store created
You can verify that the new SecretStore was added by listing the existing SecretStore objects in the namespace:
kubectl get secretstores.external-secrets.io
NAME AGE STATUS CAPABILITIES READY
pangea-vault-get-secret-store 3h38m Valid ReadOnly True
pangea-vault-list-secret-store 2m26s Valid ReadOnly True
Once the SecretStore is created, you can use it to retrieve specific Vault secrets and save them as Kubernetes secrets.
Add secrets to Vault folder in Pangea User Console
Navigate to Vault's Secrets & Keys in the Pangea User Console. Save multiple secrets in the my-app folder. For example:
-
As of 07/03/2024, ESO does not preserve
/in parameters passed to external APIs via a Webhook. Thus, to obtain multiple Vault secrets by a folder name, you must use a folder created at the Root level of Vault's Secrets & Keys in the Pangea User Console. -
In this example, Vault secret names will serve as keys in the
datafield of a Kubernetes secret. Due to Constraints on Secret names and data in Kubernetes, you can only use alphanumeric characters, hyphens, underscores, and periods in Vault secret names used in this way.
Create ExternalSecret for multiple Vault items
A single ESO ExternalSecret can save multiple Vault secrets as key/value pairs in the data field of a Kubernetes secret.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: pangea-vault-list-external-secret
namespace: external-secrets-example
spec:
refreshInterval: "5m"
secretStoreRef:
name: pangea-vault-list-secret-store
kind: SecretStore
target:
name: pangea-vault-list-secret
template:
templateFrom:
- target: Data
literal: |-
{{ $result := .secrets | fromJson }}
{{ range $result.items }}
{{- .name -}}: "{{ .current_version.secret }}"
{{ end }}
data:
- secretKey: secrets
remoteRef:
key: my-app
This manifest instructs ESO to do the following:
- Specify a Vault folder to retrieve secrets from (L24).
- Pass the folder name to the
pangea-vault-list-secret-storeSecretStore as the.remoteRef.keyparameter (L8-10, L23-24). - Save Vault secret values returned from the SecretStore as key/value pairs in the
datafield ofpangea-vault-list-secretKubernetes secret (L11-20). - Re-obtain the Vault secret every five minutes (L7).
When a template is used to transform the external API data before it is stored in a Kubernetes secret, the secret key specified in the data field (L21-22) is not added to the Kubernetes secret.
Optionally, you can save the entire content received from the SecretStore under the specified key by changing the template MergePolicy as described in the ESO Advanced Templating documentation.
In this example, the data field in the Kubernetes secret is replaced with key/value pairs defined in the template (L18-20).
kubectl apply -f ./pangea-vault-list-external-secret.yaml
externalsecret.external-secrets.io/pangea-vault-list-external-secret created
Verify ExternalSecret
kubectl get externalsecrets.external-secrets.io
NAME STORE REFRESH INTERVAL STATUS READY
pangea-vault-get-external-secret pangea-vault-get-secret-store 5m SecretSynced True
pangea-vault-list-external-secret pangea-vault-list-secret-store 5m SecretSynced True
If STATUS is not SecretSync or READY is not True, you are not saving Vault data in the Kubernetes secret. This might occur during setting up the ExternalSecret or due to changes you made in Vault.
For example, if the Vault item cannot be fetched or the SecretStore is not found, the STATUS field will show SecretSyncedError and the READY field will be populated with False:
... STATUS READY
... SecretSyncedError False
When you make changes in your ExternalSecret manifest, you can apply them with:
kubectl apply -f ./<external-secret-file-name>.yaml
However, to immediately verify changes made in the SecretStore or Vault, you might need to delete the ExternalSecret object and re-create it:
kubectl delete externalsecrets.external-secrets.io \
<external-secret-name> \
&& kubectl apply -f ./<external-secret-file-name>.yaml
Verify Kubernetes secret created by ExternalSecret
kubectl get secrets
NAME TYPE DATA AGE
pangea-vault-credentials Opaque 1 6h30m
pangea-vault-list-secret Opaque 3 9m47s
pangea-vault-get-secret Opaque 1 3h16m
You can use the following command to output the key/value pairs saved in the pangea-vault-list-secret data field. The command uses the base64 utility, which assumes you are running it on Linux or macOS:
kubectl get secrets pangea-vault-list-secret \
--template='{{range $k, $v := .data}}{{printf "%s: %s\n" $k (base64decode $v)}}{{end}}'
my-app-secret: my app secret value updated
my.third.secret: my third secret value
my_second_secret: my second secret value
Use Kubernetes secrets
Vault secrets imported with ESO can be injected in your deployments as environment variables or mounted as volumes, like any other Kubernetes Secrets . Follow the best practice for Safely Handling Secrets in Kubernetes Clusters when using them in your cluster.
Was this article helpful?