Inject secrets directly into Pods from Vault revisited

A key part of the Banzai Cloud Pipeline platform, has always been our strong focus on security. We incorporated Vault into our architecture early on in the design process, and we have developed a number of support components to be easily used with Kubernetes. We love what Vault enables us to do, but, as with many things security-related, strengthening one part of our system exposed a weakness elsewhere. For us, that weakness was K8s secrets, which is the standard way in which applications consume secrets and credentials on Kubernetes. Any secret that is securely stored in Vault and then unsealed for consumption will eventually end up as a K8s secret, and with much less protection and security than we'd like. K8s secrets use base64 encoding that, while better than nothing, does not satisfy our standards, and likely fails to satisfy the standards of most enterprise clients as well. As a result, we've developed a solution wherein we can bypass the K8s secrets mechanism and inject secrets directly into Pods from Vault.

Vault -> Kubernetes secrets -> Pod

If you are familiar with Kubernetes secrets, you know that these secrets are stored in etcd. When we say that we intend to bypass K8s security, we mean that we don't intend to touch etcd at all; the problem with etcd is that when data is encrypted at rest, it is encrypted with a global key (see the relevant documentation). That's less than ideal in a multi-tenant cluster, where independent and unrelated users might potentially gain access to the secrets of others. Also, if you already have a security team that's operating a certified Vault installation, they're probably not going to be happy about placing an unencrypted secret in an intermediary location (Kubernetes secrets, i.e. etcd). Banzai Cloud's Pipeline platform already used Kubernetes webhooks to provide a range of advanced features (security scans, spot instance scheduling, annotating webhooks, etc.), and it occured to us that using a webhook to inject secrets directly into Kubernetes containers from Vault would be a good way of bypassing etcd. Let's dive into how it works.

Kubernetes mutating webhook for injecting secrets

Our mutating admission webhook injects an executable into containers (in a non-intrusive way) inside Pods, which then request secrets from Vault through special environment variable definitions. This project was inspired by a number of other projects (e.g. channable/vaultenv, hashicorp/envconsul), but one thing that makes it unique is that it is a daemonless solution. First, the Kubernetes webhook checks if a container has environment variables with values that correspond to a specific schema. Then it reads the values for those variables directly from Vault at start-up:

env: - name:
AWS_SECRET_ACCESS_KEY value:
"vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY"

After that, the init-container is injected into the Pod, and a small binary called vault-env is attached to it as an in-memory volume. That volume is mounted to all containers with the appropriate environment variable definitions. The init-container also changes the command of the container to run vault-env, instead of running the application directly. vault-env starts up, connects to Vault (using the Kubernetes Auth method), checks that the environment variables have a reference to a value stored in Vault (vault:secret/....) and replaces that with a corresponding value from Vault's secrets backend. Afterward, vault-env executes the original process (with syscall.Exec()), which uses the secret that was originally stored in Vault. Using this solution prevents secrets stored in Vault from landing in Kubernetes secrets (and in etcd). vault-env was designed to work on Kubernetes, but there's nothing stopping it from being used outside of Kubernetes as well. It can be configured with the standard Vault client's environment variables, since there's a standard Go Vault client underneath. Currently, the Kubernetes Service Account-based Vault authentication mechanism is used by vault-env, which requests a Vault token in return for the Service Account of the container it's being injected into. But our implementation is going to change in order to allow the use of the Vault Agent's Auto-Auth feature very soon. This will allow users to request tokens in init-containers with all the authentication mechanisms supported by Vault Agent, so they won't be handcuffed to the Kubernetes Service Account-based method.

Why is this more secure than using Kubernetes secrets or using any other custom sidecar container?

Our solution is particularly lightweight and uses only existing Kubernetes constructs like annotations and environment variables. No confidential data ever persists on the disk - not even temporarily - or in etcd. All secrets are stored in memory, and only visible to the process that requests them. If you want to make this solution even more robust, you can disable kubectl exec-ing in running containers. If you do so, no one will be able to hijack injected environment variables from a process. Additionally, there is no persistent connection with Vault, and any Vault token used to read environment variables is flushed from memory before the application starts, in order to minimize attack surface.

A complete example

This example will guide you through setting up a fully functional Vault installation with the Banzai Cloud Vault operator, and help you to create an example deployment that will be mutated by the webhook so that environment variables can be injected into Pods:

# These examples require Helm 3 and kubectl:

# Add the Banzai Cloud Helm repository
helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com

# Create a namespace for the bank-vaults components called vault-infra
# Namespace labeling is required, because the webhook's mutation is based on label selectors
kubectl create namespace vault-infra
kubectl label namespace vault-infra name=vault-infra

# Install the vault-operator to the vault-infra namespace
helm upgrade --namespace vault-infra --install vault-operator banzaicloud-stable/vault-operator --wait

# Clone the bank-vaults project
git clone git@github.com:banzaicloud/bank-vaults.git
cd bank-vaults

# Create a Vault instance with the operator which has the Kubernetes auth method configured
kubectl apply -f operator/deploy/rbac.yaml
kubectl apply -f operator/deploy/cr.yaml

# Now you have a fully functional Vault installation on top of Kubernetes,
# orchestrated by the `banzaicloud/vault-operator` and `banzaicloud/bank-vaults`.

# Next, install the mutating webhook with Helm into its own namespace (to bypass the catch-22 situation of self mutation)
helm upgrade --namespace vault-infra --install vault-secrets-webhook banzaicloud-stable/vault-secrets-webhook --wait

# Set the Vault token from the Kubernetes secret
# (strictly for demonstrative purposes, we have K8s unsealing in cr.yaml)

export VAULT_TOKEN=$(kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root} | base64 --decode)

# Tell the CLI that the Vault Cert is signed by a custom CA

kubectl get secret vault-tls -o jsonpath="{.data.ca\.crt}" | base64 --decode > $PWD/vault-ca.crt
export VAULT_CACERT=$PWD/vault-ca.crt

# Tell the CLI where Vault is listening (the certificate has 127.0.0.1 as well as alternate names)

export VAULT_ADDR=https://127.0.0.1:8200

# Forward the TCP connection from your Vault pod to localhost (in the background)

kubectl port-forward service/vault 8200 &

# Write a secret into Vault, which will be injected as an environment variable

vault kv put secret/accounts/aws AWS_SECRET_ACCESS_KEY=s3cr3t

# Apply the deployment with special environment variables
# It will be mutated by the webhook

kubectl apply -f deploy/test-deployment.yaml

The deployment will be mutated by the webhook, because it has at least one environment variable that has a value that is a reference to a path in Vault. Here's what the original deployment looks like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-secrets
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-secrets
  template:
    metadata:
      labels:
        app: hello-secrets
      annotations:
        vault.security.banzaicloud.io/vault-addr: "https://vault:8200"
        vault.security.banzaicloud.io/vault-tls-secret: "vault-tls"
    spec:
      serviceAccountName: default
      containers:
        - name: alpine
          image: alpine
          command:
            [
              "sh",
              "-c",
              "echo $AWS_SECRET_ACCESS_KEY && echo going to
              sleep... && sleep 10000",
            ]
          env:
            - name: AWS_SECRET_ACCESS_KEY
              value: "vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY"

It produces Pods like so (only the relevant parts are shown here, Pods are mutated directly):

apiVersion: v1
kind: Pod
metadata:
  name: hello-secrets-575554499f-26894
  labels:
    app: hello-secrets
  annotations:
    vault.banzaicloud.io/vault-addr: "https://vault:8200"
    vault.security.banzaicloud.io/vault-tls-secret: "vault-tls"
spec:
  initContainers:
    - name: copy-vault-env
      command:
        - sh
        - -c
        - cp /usr/local/bin/vault-env /vault/
      image: banzaicloud/vault-env:latest
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - mountPath: /vault/
          name: vault-env
  containers:
    - name: alpine
      command:
        - /vault/vault-env
      args:
        - sh
        - -c
        - echo $AWS_SECRET_ACCESS_KEY $ && echo going to
          sleep... && sleep 10000
      image: alpine
      imagePullPolicy: Always
      env:
        - name: AWS_SECRET_ACCESS_KEY
          value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY
        - name: VAULT_ADDR
          value: https://vault:8200
        - name: VAULT_SKIP_VERIFY
          value: "false"
        - name: VAULT_PATH
          value: kubernetes
        - name: VAULT_ROLE
          value: default
        - name: VAULT_IGNORE_MISSING_SECRETS
          value: "false"
        - name: VAULT_CACERT
          value: /vault/tls/ca.crt
      volumeMounts:
        - mountPath: /vault/
          name: vault-env
        - mountPath: /vault/tls/ca.crt
          name: vault-tls
          subPath: ca.crt
  volumes:
    - emptyDir:
        medium: Memory
      name: vault-env
    - name: vault-tls
      secret:
        secretName: vault-tls

As you can see, none of the original environment variables in the definiition have been touched, and the sensitive value of the AWS_SECRET_ACCESS_KEY variable is only visible inside the alpine container.

Using charts without an explicit container.command and container.args

The Webhook is now capable of determining the container's entry point and command with the help of image metadata queried from the image registry. This data is cached until the webhook Pod is restarted. If the registry is publicly accessible (without authentication), you don't need to do anything, but, if the registry requires authentication, the necessary credentials have to be made available in the Pod's imagePullSecrets section, or in the Pod's ServiceAccount.

NOTE: Future improvement: on AWS and GKE and other cloud providers get a credential dynamically with the cloud-specific SDK

MySQL example:

# Put the MySQL passwords into Vault
vault kv put secret/mysql MYSQL_ROOT_PASSWORD=s3cr3t MYSQL_PASSWORD=3xtr3ms3cr3t

# Install the MySQL chart with root and a user password sourced from Vault
helm upgrade --install mysql stable/mysql \
  --set mysqlRootPassword=vault:secret/data/mysql#MYSQL_ROOT_PASSWORD \
  --set mysqlPassword=vault:secret/data/mysql#MYSQL_PASSWORD \
  --set "podAnnotations.vault\.security\.banzaicloud\.io/vault-addr"=https://vault:8200 \
  --set "podAnnotations.vault\.security\.banzaicloud\.io/vault-tls-secret"=vault-tls \
  --wait


# Open a connection towards the MySQL Service
kubectl port-forward service/mysql 3306 &

# Read the MySQL user password's secret from Vault
# Make sure you still have the port-forward from the previous example
vault read secret/data/mysql

Key         Value
---         -----
data        map[MYSQL_PASSWORD:3xtr3ms3cr3t MYSQL_ROOT_PASSWORD:s3cr3t]
metadata    map[created_time:2019-09-05T13:03:42.980780517Z deletion_time: destroyed:false version:1]

# Open up the MySQL shell with the root user and its corresponding password from Vault `s3cr3t`
mysql -h 127.0.0.1 -u root -p

Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 84
Server version: 5.7.14 MySQL Community Server (GPL)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>


# And here's the magic: there are still no secrets in the env vars!
kubectl exec -it mysql-749cfddc67-5slcb bash
root@mysql-749cfddc67-5slcb:/\# env | grep MYSQL.*PASSWORD
MYSQL_PASSWORD=vault:secret/data/mysql#MYSQL_PASSWORD
MYSQL_ROOT_PASSWORD=vault:secret/data/mysql#MYSQL_ROOT_PASSWORD

Of course, if you exec into the Pod and rerun vault-env it will fork a new process which will have those environment variables correctly set: 

/vault/vault-env env | grep MYSQL.*PASSWORD
2019/09/05 13:42:09 Received new Vault token
2019/09/05 13:42:09 Initial Vault token arrived
MYSQL_PASSWORD=3xtr3ms3cr3t
MYSQL_ROOT_PASSWORD=s3cr3t

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  namespace: default
  name: pod-execer
rules:
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]

apiVersion: v1
kind: Secret
metadata:
  name: sample-secret
  annotations:
    vault.security.banzaicloud.io/vault-addr: "https://vault.default.svc.cluster.local:8200"
    vault.security.banzaicloud.io/vault-role: "default" # In case of Secrets the webhook's ServiceAccount
is used
    vault.security.banzaicloud.io/vault-skip-verify: "true"
    vault.security.banzaicloud.io/vault-path: "kubernetes"
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: eyJhdXRocyI6eyJodHRwczovL2RvY2tlci5pbyI6eyJ1c2VybmFtZSI6InZhdWx0OnNlY3JldC9kYXRhL2RvY

kubectl create secret docker-registry dockerhub --docker-username="vault:secret/data/dockerrepo#DOCKER_REPO_USER" --docker-password="vault:secret/data/dockerrepo#DOCKER_REPO_PASSWORD"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-addr="https://vault.default.svc.cluster.local:8200"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-role="default"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-skip-verify="true"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-path="kubernetes"