Application secrets¶

The Vault service provided by Caascad is configured such as Kubernetes applications can read secrets in it.

Applications can authenticate through Vault using the associated token with their ServiceAccount and read secrets. For applications that don't have a native integration with Vault it is possible to use the Vault injector service (optionally deployed in clusters by Caascad).

Kubernetes secrets vs Vault secrets¶

For providing secrets to applications, Kubernetes features a native secret API which allows users to specify secrets and use their contents in the application Pod environment as files or environment variables.

Kubernetes secrets are simple to use because of their native integration to the Kubernetes ecosystem but they have some drawbacks:

usually Kubernetes resource manifests are stored in code repositories. Secrets which are by nature confidential should be encrypted in theses code repositories and decrypted when manifests are applied to the cluster. Therefore supplementary tooling must be put in use
access to Kubernetes secrets by applications outside their namespace is forbidden. If multiple applications need access to the same secret it needs to be duplicated in multiple namespaces
secrets are always static
with Kubernetes RBAC it is not possible to configure fine grained access to secrets to the users of the cluster
depending on the Kubernetes control plane configuration the API resources including secrets may not be encrypted at rest on disk

Using Vault can solve theses different problems:

secrets are not stored in files and are written directly in Vault
Vault has fine grained access policies to secrets
no namespace, multiple applications can read the same secret regardless their method of deployment: Kubernetes application, VM, etc...
Vault usage is not limited to Kubernetes applications
Vault can provide dynamic secrets (AWS temporary AK/SK, certificates, ...)
Vault DB is encrypted at rest

However since Vault is not natively integrated to the Kubernetes ecosystem some integration is needed between applications and Vault. The following methods need to be considered:

if the application is capable to talk directly to Vault the connection to Vault is made by the application itself
if the application has no native integration with Vault a third party solution like Vault agent can be used to read secrets from Vault before and/or while the application is running

Secrets hierarchy¶

Caascad's Vault integration provides a generic organization model to cover most of use cases.

Secrets for Kubernetes applications must be stored in specific places in the KV store depending on the access scope you want:

kv/k8s/global/all/: access by all applications on all clusters
kv/k8s/global/<NAMESPACE>: access by applications of a specific namespace on all clusters
kv/k8s/<CLUSTER_NAME>/all: access by all applications on a specific cluster
kv/k8s/<CLUSTER_NAME>/<NAMESPACE>: access by applications of a specific namespace on a specific cluster

Note

By default you won't see any of theses paths until some secret has been provisioned.

When creating a secret make sure to write it in a correct path like described above.

To write secrets in kv/k8s/global you need the caascad-devops role (or the caascad-vault-k8s-secrets which is included in caascad-devops). Users that have a caascad-devops-<CLUSTER_NAME> role (or caascad-vault-k8s-secrets-<CLUSTER_NAME>) are allowed to write secrets in kv/k8s/<CLUSTER_NAME> only.

Native integration of applications with Vault¶

This is the recommended method if you develop yourself the applications because the secrets won't be easily visible inside the cluster (even with a kubectl exec in the Pod).

Easy to use libraries are available for most commonly used programming languages.

Example of Kubernetes authentication and secret reading with the HVAC library (Python):

import hvac
# Read the Kubernetes token in the Pod
f = open('/var/run/secrets/kubernetes.io/serviceaccount/token')
jwt = f.read()
# Vault client creation
client = hvac.Client(url='https://vault.<ZONE_NAME>.caascad.com')
# Login with `caascad-secret-reader` role and the Pod token
# Depending on the cluster the `mount_point` is different
client.auth_kubernetes('caascad-secret-reader', jwt, mount_point='kubernetes-<CLUSTER_NAME>')
# Read some secret
s = client.secrets.kv.read_secret_version(path='kv/k8s/global/all/my-secret')
print(s['data']['data'])

Note

If you are using a ServiceAccount in your manifests, make sure that the automountServiceAccountToken parameter is not set to false.

Vault injector¶

Vault injector allow to inject an init container in the application Pod. The initialization process that is added to the Pod is responsible for fetching secrets from Vault. Secrets are written to files which can be read by the application.

Note

Vault injector is not deployed by default. You need to ask Caascad to deploy it in clusters if you need it.
If you are using a ServiceAccount in your manifests, make sure that the automountServiceAccountToken parameter is not set to false.

The activation of the injector is done with Pod annotations.

Minimal annotations¶

Example of the minimal set of annotations to use:

annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "caascad-secret-reader"
    vault.hashicorp.com/agent-inject-secret-foo: "kv/data/k8s/global/all/foo"

Note

on any cluster, the role to use is always caascad-secret-reader
as the KV store is in version 2 the secret path must start with kv/data/...
the content of the secret kv/k8s/global/all/foo will be written in a volume which is attached to the Pod in /vault/secrets/<unique_name> (where <unique_name> is derived from the following annotation suffix vault.hashicorp.com/agent-inject-secret-<unique_name>)
it is possible to specify multiple times the annotation vault.hashicorp.com/agent-inject-secret-<unique_name> to read multiple secrets

The format of the annotation to read a secret is:

vault.hashicorp.com/agent-inject-secret-<unique_name>: /path/to/secret

Where <unique_name> is the name of the file that will be created in /vault/secrets.

The generated file will have this format (json parsable):

{"data":{"key1":"secret1","key2":"secret2","key3":"secret3"},"metadata":{"created_time":"2021-06-12T09:38:53.854966686Z","custom_metadata":null}}

Templates¶

If the default file format doesn't suit the application it is possible to define a template to render the file by adding another annotation:

Examples¶

To generate a file compatible with posix shell interpreters:

vault.hashicorp.com/agent-inject-template-foo: |
  {{- with secret "kv/data/k8s/global/all/foo" -}}
  export KEY1={{ .Data.data.key1 }}
  export KEY2={{ .Data.data.key2 }}
  {{- end }}

The generated file will look like this:

export KEY1=secret1
export KEY2=secret2

Note

This example shows how to generate a file that can be used to expose environment variables to the application. To effectively use theses variables the container command should be altered:

apiVersion: v1
kind: Pod
metadata:
  name: demo-pod
annotations:
  <vault-injector annotations...>
spec:
  containers:
    command: ["/bin/bash", "-c", "source /vault/secrets/foo && exec /bin/app"]

To loop on all keys available in a secret, the following template can be used:

vault.hashicorp.com/agent-inject-template-foo: |
  {{- with secret "kv/data/k8s/global/all/foo" -}}
  {{- range $k, $v := .Data.data -}}
  {{ $k }}: {{ $v }}
  {{ end -}}
  {{- end -}}

The generated file will look like this:

key1: secret1
key2: secret2
key3: secret3

Export secret as yaml:

vault.hashicorp.com/agent-inject-template-foo: |
  {{- with secret "kv/data/k8s/global/all/foo" -}}
  {{- .Data.data | toYAML }}
  {{- end }}

Here is a link to the official documentation for further information on Vault templating language.

Sidecar¶

By default an init container as well as a sidecar Pod (one container in the Pod) will be injected. The sidecar role is to watch for secrets changes and updating them in the filesystem when they change.

This sidecar is often useless and can be deactivated with the annotation:

vault.hashicorp.com/agent-pre-populate-only: "true"

Application secrets¶

Kubernetes secrets vs Vault secrets¶

Secrets hierarchy¶

Native integration of applications with Vault¶

Vault injector¶

Minimal annotations¶

Templates¶

Examples¶

Sidecar¶

Links¶