Introduction

In this guide we will learn how to inject Vault secrets into Kubernetes pods from a HashiCorp Vault server that runs outside the cluster, on its own Ubuntu VM. Rather than installing Vault itself in Kubernetes, we will add only one small component inside the cluster: the Vault Agent Injector. Once it is in place, the injector fetches secrets from Vault and places them inside an app pod as a file. This is the simplest way to inject Vault secrets into Kubernetes without running Vault inside the cluster.

Everything in Kubernetes lives in one namespace called vault.

Before you start, note these two values, because you will use them many times:

  • VAULT_ADDR: the URL of your Vault VM, for example http://192.168.1.50:8200 .
  • K8S_API: the URL of your Kubernetes API server, for example https://192.168.1.60:6443 .

To find your kubernetes API server URL, simply run kubectl cluster-info . For example:

$ kubectl cluster-info
Kubernetes control plane is running at https://192.168.10.10:6443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

This guide will follow these parts:

  • Part 1: Prepare the Kubernetes Cluster to Inject Vault Secrets
  • Part 2: Configure Vault to Inject Secrets into Kubernetes
  • Part 3: Install the Vault Agent Injector to Inject Secrets into Kubernetes
  • Part 4: Inject Vault Secrets into Kubernetes Pods
  • Part 5: Refresh Vault Secrets in Kubernetes Automatically
  • Part 6: Inject Dynamic Database Secrets from Vault into Kubernetes

Procedure

Part 1: Prepare the Kubernetes Cluster to Inject Vault Secrets

1.1 Create the namespace

$ kubectl create namespace vault

1.2 Create a service account for Vault

Vault needs a way to check that tokens coming from your pods are real. For this purpose, it calls the Kubernetes TokenReview API. Therefore, we create a service account with the right permissions:

$ kubectl -n vault create serviceaccount vault-auth

$ kubectl create clusterrolebinding vault-auth-tokenreview \
  --clusterrole=system:auth-delegator \
  --serviceaccount=vault:vault-auth

However, on Kubernetes 1.24 and newer, tokens are no longer created automatically. As a result, we create a long lived token secret by hand:

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth-token
  namespace: vault
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
EOF

1.3 Collect three values for the Vault UI

Run these commands and save the output somewhere, because you will paste these values into the Vault web UI in Part 2.

# 1) Token reviewer JWT
$ kubectl -n vault get secret vault-auth-token \
  -o go-template='{{ .data.token | base64decode }}'; echo

# 2) Kubernetes CA certificate (PEM)
$ kubectl -n vault get secret vault-auth-token \
  -o go-template='{{ index .data "ca.crt" | base64decode }}'

# 3) Kubernetes API address
$ kubectl cluster-info

One important network check before moving on: the Vault VM must be able to reach the Kubernetes API, and in the other direction, pods in the cluster must be able to reach Vault. Since both machines sit on the same network, a quick curl -k K8S_API from the VM and curl VAULT_ADDR/v1/sys/health from a pod will confirm this.

Part 2: Configure Vault to Inject Secrets into Kubernetes

First, open your Vault address in a browser and log in with your root token or an admin user.

2.1 Enable a KV secrets engine

In the left sidebar, click Secrets Engines. Next, click Enable new engine +. Choose KV and click Next. Then set the Path to kv and leave Version 2 selected. Finally, click Enable engine.

2.2 Create the Vault Secrets to Inject

Inside the new kv engine, click Create secret +. Set the path for this secret to webapp/config. After that, add two key and value pairs under Secret data:

  • username with value user
  • password with value s3cr3t-p4ssw0rd

Once both are in, click Save.

2.3 Create a policy for the app

In the sidebar go to Policies, then ACL Policies, then Create ACL policy +. Name it webapp, and afterwards paste this into the policy editor. Notice the data/ part in the path, because KV version 2 requires it.

path "kv/data/webapp/config" {
  capabilities = ["read"]
}

Finally, click Create policy.

2.4 Enable and configure the Kubernetes auth method

First, in the sidebar go to Access, then Auth Methods, then Enable new method +. Choose Kubernetes, click Next, keep the path as kubernetes, and click Enable method.

After that, you will land on the method page. Click Configure (or Configuration, then Edit configuration), and then fill in the form with the values from Step 1.3:

  • Kubernetes host: your K8S_API value, for example https://192.168.1.60:6443
  • Kubernetes CA Certificate: paste the full PEM block, including the BEGIN and END CERTIFICATE lines
  • Token Reviewer JWT: paste the long JWT string. It may be hidden inside a collapsed section called Kubernetes Options, so expand it if needed.

Once everything is filled in, click Save.

Now, an important warning about a known UI quirk. The Token Reviewer JWT field is write only, so the UI always shows it as blank. Moreover, some Vault versions silently drop it when you save the form. If that happens, logins will later fail with 403 permission denied. In addition, if you run vault monitor -log-level=trace on the VM, you will see Vault fall back to the client’s own token with an error like: tokenreviews.authentication.k8s.io is forbidden: User “system:serviceaccount:vault:webapp” cannot create resource “tokenreviews”.

Fortunately, there is a reliable fix: write this one piece of config with the CLI on the Vault VM. Everything else can stay in the UI. Note that this command rewrites the whole auth config, so you must include the host and CA again:

$ export VAULT_ADDR=https://127.0.0.1:8200
$ export VAULT_SKIP_VERIFY=true   # only if using a self-signed cert
$ vault login                     # paste your root or admin token

# save the values from Step 1.3 into two files first: reviewer.jwt and ca.crt
$ vault write auth/kubernetes/config \
    kubernetes_host="https://<K8S_API_IP>:6443" \
    kubernetes_ca_cert=@ca.crt \
    token_reviewer_jwt="$(cat reviewer.jwt)"

No pod restart is needed after this, because the agent retries on its own. You may still see a failed to read local service account token debug line in Vault’s logs. However, that is harmless. It only means Vault is not running inside the cluster.

2.5 Create a role that binds the app to the policy

Still on the kubernetes/ auth method page, open the Roles tab and click Create role +. Then fill in:

  • Name: webapp
  • Bound service account names: webapp
  • Bound service account namespaces: vault
  • Generated Token’s Policies (under the Tokens section): webapp
  • Generated Token’s Initial TTL: 24h (optional)

Finally, click Save. Vault is now fully configured.

Part 3: Install the Vault Agent Injector to Inject Secrets into Kubernetes

We only install the injector, which is the component that will inject Vault secrets into Kubernetes pods, and it points at your external Vault VM.

$ helm repo add hashicorp https://helm.releases.hashicorp.com
$ helm repo update

$ helm install vault hashicorp/vault \
  --namespace vault \
  --set "injector.externalVaultAddr=VAULT_ADDR"

Use the same scheme (http:// or https://) as your VAULT_ADDR, because it must match how the Vault listener on the VM is really configured:

  • No TLS on the listener. This is common for lab and demo setups where the Vault config has tls_disable = 1. In that case use http://, for example --set "injector.externalVaultAddr=http://192.168.1.50:8200". As a quick check: if the web UI opens at an http:// address, this is your case.
  • TLS enabled. Use https://, for example --set "injector.externalVaultAddr=https://192.168.1.50:8200". However, if the certificate is self signed or comes from a private CA, the injected agents will not trust it by default. In that case you have two options. The first option is to add these annotations to the app in Part 4, next to the existing ones:
vault.hashicorp.com/tls-secret: "vault-ca"        # a secret in vault holding your CA
vault.hashicorp.com/ca-cert: "/vault/tls/ca.crt"

Additionally, create that secret with kubectl -n vault create secret generic vault-ca --from-file=ca.crt=/path/to/ca.pem. The second option, for a throwaway demo only, is to skip verification with vault.hashicorp.com/tls-skip-verify: "true".

Afterwards, verify the injector is running:

$ kubectl -n vault get pods
# vault-agent-injector-xxxxx   1/1   Running

Part 4: Inject Vault Secrets into Kubernetes Pods

Now we deploy a sample app and inject Vault secrets into Kubernetes as a file under /vault/secrets/. First, save the following as webapp.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: webapp
  namespace: vault
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  namespace: vault
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "webapp"
        # Demo only: skip TLS verification (needed when Vault's cert lacks an IP SAN
        # or is self-signed). Remove this and use the tls-secret and ca-cert
        # annotations from Part 3 for anything beyond a throwaway demo.
        vault.hashicorp.com/tls-skip-verify: "true"
        vault.hashicorp.com/agent-inject-secret-config.txt: "kv/data/webapp/config"
        vault.hashicorp.com/agent-inject-template-config.txt: |
          {{- with secret "kv/data/webapp/config" -}}
          username={{ .Data.data.username }}
          password={{ .Data.data.password }}
          {{- end -}}
    spec:
      serviceAccountName: webapp
      containers:
        - name: webapp
          image: nginx:alpine
          ports:
            - containerPort: 80

Next, apply it and check the result:

$ kubectl apply -f webapp.yaml

$ kubectl -n vault get pods
# webapp-xxxxx   2/2   Running   (2/2 means app plus vault-agent sidecar)

$ kubectl -n vault exec deploy/webapp -c webapp -- cat /vault/secrets/config.txt

Expected output:

username=user
password=s3cr3t-p4ssw0rd

Part 5: Refresh Vault Secrets in Kubernetes Automatically

The vault-agent sidecar does not fetch the secret just once. Instead, it keeps running and rewrites /vault/secrets/config.txt whenever the secret changes in Vault. For static KV secrets it checks again on a polling interval. The default interval is 5 minutes, which is too slow for testing. Therefore, speed it up to 30 seconds by adding one annotation to webapp.yaml, right next to the other vault.hashicorp.com/* annotations:

        vault.hashicorp.com/template-static-secret-render-interval: "30s"

Then apply it. Because the annotation changed, a new pod rolls out:

$ kubectl apply -f webapp.yaml
$ kubectl -n vault rollout status deploy/webapp

Now run the live test. First, in one terminal, watch the injected file:

$ kubectl -n vault exec deploy/webapp -c webapp -- \
  sh -c 'while true; do date; cat /vault/secrets/config.txt; echo; echo ---; sleep 5; done'

Meanwhile, in the Vault UI, open Secrets Engines, then kv, then webapp/config, then Create new version +. Change the password value to something new and save.

Within about 30 seconds, the terminal output changes to the new password. No redeploy, no restart, no kubectl: once you inject Vault secrets into Kubernetes this way, rotating them in the UI is enough.

Finally, a note for real apps. The file refreshes on its own; however, a process that read the secret only at startup will not notice the change. Fortunately, the injector can run a command after each render using the annotation vault.hashicorp.com/agent-inject-command-config.txt, for example to send a reload signal to the app. That needs shareProcessNamespace: true in the pod spec, and in addition the app must run as a user that the agent (uid 100) is allowed to signal. In contrast, apps that read the file again on each use, or that watch the file for changes, need nothing extra.

Part 6: Inject Dynamic Database Secrets from Vault into Kubernetes

So far we have used static values; now we inject Vault secrets into Kubernetes that Vault generates on demand. This is where Vault gets really impressive. With the database secrets engine, Vault does not store a database password at all. Instead, Vault holds admin access to the database, and every time an app authenticates, it creates a brand new database user with a random password and a time to live (TTL). Meanwhile, the agent sidecar keeps renewing the lease. When it cannot renew anymore, it fetches fresh credentials and rewrites the injected file, and Vault removes the old user from the database. As a result, no two pods ever share credentials, and a leaked credential dies on its own.

We will use PostgreSQL running inside the cluster. However, one networking note before we start: Vault on the VM must be able to reach the database in order to create users in it, while the app reaches it from inside the cluster. Therefore the database gets a NodePort. The VM connects through <any-node-ip>:30432, whereas pods connect through the normal service DNS name.

6.1 Deploy a demo PostgreSQL

First, save this as postgres.yaml and apply it:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: vault
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          env:
            - name: POSTGRES_USER
              value: postgres
            - name: POSTGRES_PASSWORD
              value: rootpassword
            - name: POSTGRES_DB
              value: appdb
          ports:
            - containerPort: 5432
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: vault
spec:
  type: NodePort
  selector:
    app: postgres
  ports:
    - port: 5432
      targetPort: 5432
      nodePort: 30432
$ kubectl apply -f postgres.yaml
$ kubectl -n vault rollout status deploy/postgres

Afterwards, do a sanity check from the Vault VM, because this is the exact path Vault will use:

# any tool works; for example with psql installed:
$ psql "postgresql://postgres:rootpassword@<NODE_IP>:30432/appdb" -c "select 1;"
# or just check that the port answers:
$ nc -zv <NODE_IP> 30432

<NODE_IP> is the IP of any cluster node. To see the node IPs, run kubectl get nodes -o wide.

6.2 Enable the Database secrets engine

In the Vault UI sidebar go to Secrets Engines, then Enable new engine +. Choose Databases, click Next, keep the path database, and finally click Enable engine.

6.3 Create the database connection

Inside the database engine, click Create connection + (or use the Connections tab). Then fill in:

  • Database plugin: PostgreSQL
  • Connection name: appdb
  • Connection URL: postgresql://{{username}}:{{password}}@<NODE_IP>:30432/appdb?sslmode=disable (type {{username}} and {{password}} exactly as written; Vault fills them in with the credentials below)
  • Username: postgres
  • Password: rootpassword
  • Allowed roles: webapp-db, but only if this field appears. Many Vault versions do not show it in the connection form, so if it is not there, just skip it. When you create the role through the UI in Step 6.4, Vault adds it to the allowed roles automatically.

Next, click Create database. When the UI offers to rotate the root credentials, choose Enable without rotating. Rotating means only Vault knows the postgres password afterward, yet we still want to log in directly in Step 6.6 to show off the generated users. In production, of course, you would rotate.

6.4 Create a dynamic role

Still inside the database engine, click Create role +, and then fill in:

  • Role name: webapp-db
  • Database name or connection: appdb
  • Type of role: dynamic
  • Generated credentials’ TTL: 3m (short on purpose so the test shows rotation quickly; use 1h or more normally)
  • Max TTL: 6m
  • Creation statements:
CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";

Finally, click Create role. This SQL is the template that Vault runs against Postgres for every credential request. In other words, {{name}} and {{password}} become the generated username and password.

6.5 Let the app read the new path

Next, edit the policy. Go to Policies, then ACL Policies, then webapp, then Edit, and change it to:

path "kv/data/webapp/config" {
  capabilities = ["read"]
}

path "database/creds/webapp-db" {
  capabilities = ["read"]
}

Save it. Note that there is no data/ segment in the database path, because that extra segment is a KV version 2 quirk only.

6.6 Inject the Dynamic Vault Secrets into the App

Add two more annotations to webapp.yaml, next to the existing ones:

        vault.hashicorp.com/agent-inject-secret-db-creds.txt: "database/creds/webapp-db"
        vault.hashicorp.com/agent-inject-template-db-creds.txt: |
          {{- with secret "database/creds/webapp-db" -}}
          postgresql://{{ .Data.username }}:{{ .Data.password }}@postgres.vault.svc.cluster.local:5432/appdb
          {{- end -}}

Two details are worth noticing here. First, dynamic secrets use .Data.username, not the KV version 2 form .Data.data.*. Second, the URL the app uses points to the in-cluster service DNS name, not the NodePort.

$ kubectl apply -f webapp.yaml
$ kubectl -n vault rollout status deploy/webapp

Now the show:

  1. See the generated credentials. The file contains a full connection URL. The username sits after the //, while the password sits between the : and the @:
$ kubectl -n vault exec deploy/webapp -c webapp -- cat /vault/secrets/db-creds.txt

postgresql://v-kubernet-webapp-db-x7TqR2sO:A1b2C3-d4E5f6G7h8@postgres.vault.svc.cluster.local:5432/appdb
  1. Next, prove it is a real database user. List the roles inside Postgres and point at the ones Vault created, together with their expiry time:
$ kubectl -n vault exec deploy/postgres -- psql -U postgres -d appdb -c '\du'
  1. After that, prove the credentials work. Paste the username and password from the file:
$ kubectl -n vault exec deploy/postgres -- \
  psql "postgresql://v-kubernet-webapp-db-x7TqR2sO:A1b2C3-d4E5f6G7h8@localhost:5432/appdb" -c 'select current_user;'
  1. Finally, watch rotation happen. With the 3 minute TTL and 6 minute max TTL, keep this loop running. Within a few minutes the whole connection string changes to a brand new user, while \du in Postgres shows the old one disappear:
$ kubectl -n vault exec deploy/webapp -c webapp -- \
  sh -c 'while true; do date; cat /vault/secrets/db-creds.txt; echo; echo ---; sleep 15; done'

You can also watch the leases live in the UI under Secrets Engines, then database. Moreover, you can revoke a lease by hand to kill a credential on the spot.

Summary

In conclusion, the guide showed how to inject Vault secrets into Kubernetes from an external HashiCorp Vault. You now have a full external Vault working in sync with Kubernetes: static secrets injected as files, automatic refresh when a secret changes, and dynamic database credentials that rotate on their own and clean up after themselves.

References

Deploying a Fully Offline Kubernetes Cluster with Kubespray
How to Install HashiCorp Vault.
Run Vault on Kubernetes.

Tags: