Secrets Management in Kubernetes

Published in

AVM Consulting Blog

7 min readFeb 16, 2021

Kubernetes Secrets are secure objects which store sensitive data, such as passwords, OAuth tokens, and SSH keys, etc. with encryption in your clusters.

Using Secrets gives you more flexibility in a Pod Life cycle definition and control over how sensitive data is used. It reduces the risk of exposing the data to unauthorized users.

Secrets are namespaced objects.
Secrets can be mounted as data volumes or environment variables to be used by a container in a pod.
Secret data is stored in tmpfs in nodes
API server stores secrets as plain text in etcd
A per-secret size limit of 1MB

Creating a secret:

Create username.txt and password.txt files.

echo -n 'root' > ./username.txt
echo -n 'Mq2D#(8gf09' > ./password.txt

And

kubectl create secret generic db-cerds \
  --from-file=./username.txt \
  --from-file=./password.txt
secret "db-cerds" created

List secret:

kubectl get secret/db-cerdsNAME       TYPE      DATA      AGE
db-cerds   Opaque    2         26s

View secret:

kubectl describe secret/db-cerds
Name:         db-cerds
Namespace:    default
Labels:
Annotations:Type:  OpaqueData
====
password.txt:  11 bytes
username.txt:  4 bytes

Using YAML file:

The Secret contains two maps: data and string data. The data field is used to store arbitrary data, encoded using base64.

echo -n 'root' | base64
cm9vdA==echo -n 'Mq2D#(8gf09' | base64
TXEyRCMoOGdmMDk=

Write a Secret yaml file

--- 
apiVersion: v1
data: 
  password: TXEyRCMoOGdmMDk=
  username: cm9vdA==
kind: Secret
metadata: 
  name: database-creds
type: Opaque

Create the secret using kubectl create

kubectl create -f creds.yaml
secret "database-creds" createdkubectl get secret/database-creds
NAME             TYPE      DATA      AGE
database-creds   Opaque    2         1m

View secret:

kubectl get secret/database-creds -o yaml

--- 
apiVersion: v1
data: 
  password: TXEyRCMoOGdmMDk=
  username: cm9vdA==
kind: Secret
metadata: 
  creationTimestamp: 2019-02-25 06:22:37 +00:00
  name: database-creds
  namespace: default
  resourceVersion: "2657"
  selfLink: /api/v1/namespaces/default/secrets/database-creds
  uid: bf0cef90-38c5-11e9-8c95-42010a800068
type: Opaque

Decoding secret values:

echo -n "cm9vdA==" | base64 --decode
rootecho -n "TXEyRCMoOGdmMDk=" | base64 --decode
Mq2D#(8gf09

Usage of Secrets

A Secret can be used with your workloads in two ways:

specify environment variables that reference the Secret’s values
mount a volume containing the Secret.

Environment variables:

--- 
apiVersion: v1
kind: Pod
metadata: 
  name: php-mysql-app
spec: 
  containers: 
    - 
      env: 
        - 
          name: MYSQL_USER
          valueFrom: 
            secretKeyRef: 
              key: username
              name: database-creds
        - 
          name: MYSQL_PASSWORD
          valueFrom: 
            secretKeyRef: 
              key: password
              name: database-creds
      image: "php:latest"
      name: php-app

Secret as Volume:

--- 
apiVersion: v1
kind: Pod
metadata: 
  name: redis-pod
spec: 
  containers: 
    - 
      image: redis
      name: redis-pod
      volumeMounts: 
        - 
          mountPath: /etc/dbcreds
          name: dbcreds
          readOnly: true
  volumes: 
    - 
      name: dbcreds
      secret: 
        secretName: database-creds

Additional Info :

Secret creation syntax

kubectl create secret [TYPE] [NAME] [DATA]

TYPE can be one of the following:

generic: Create a Secret from a local file, directory, or literal value.

docker-registry: Creates a dockercfg Secret for use with a Docker registry. Used to authenticate against Docker registries.

tls: Create a TLS secret from the given public/private key pair. The public/private key pair must exist beforehand. The public key certificate must be.PEM encoded and match the given private key.

DATA can be one of the following:

— from-file

kubectl create secret generic credentials \
  --from-file=username=./username.txt \
  --from-file=password=./password.txt
--from-env-file

— from-env-file

cat credentials.txt
username=admin
password=Ex67Hn*9#(jwkubectl create secret generic credentials \
--from-env-file ./credentials.txt

— from-literal flags

kubectl create secret generic literal-token \
--from-literal user=admin \
--from-literal password="Ex67Hn*9#(jw"

Thanks

Questions /Answers

How to make the Kubernetes pods unable to decrypt the Kubernetes secrets without a key?

I am trying to making this as step by step solution.

Encrypt your data using your-key (your encryption-logic, probably, in a script).

./encrypt.sh --key your-key --data your-data

Create a secret of this encrypted data

kubectl create secret generic your-secret-name --from-literal=secretdata=your-encrypted-data

You could add decryption logic like this in your pod ( either as a sidecar or init container)

# decrypt.sh will decode base64 then your decryption logic using your-key
./decrypt.sh --key your-key --data /var/my-secrets

Also, you need to mount this secret as volume to your container.

spec:
      containers:
      - image: "image"
        name: app
        ...
        volumeMounts:
          - mountPath: "/var/my-secrets"
            name: my-secret
      volumes:
        - name: my-secret
          secret:
            secretName: your-secret-name

Refreshing PODs automatically when mounted secrets get updated, but it looks not happening

Kubernetes does not support this feature at the moment and there is a feature in the works (https://github.com/kubernetes/kubernetes/issues/22368).

You can use custom solution available to achieve the same and one of the popular ones include Reloader.

The doc you linked describes that the secret values inside the mounted volume will get updated when you update the Kubernetes Secret object.

The application running within the pod can get to re-read the key once it’s updated tho’ and a brand new pod isn’t created to change the key itself.

Also, note that there can be some delay between the actual update of the secret and getting those values reflected in the volume.

As a result, the overall delay from the instant, once the key is updated to the instant once new keys are projected to the Pod, will be as long as the kubelet sync period + cache propagation delay, where the cache propagation delay depends on the chosen cache type (it equals to watch propagation delay, TTL of cache, or zero correspondingly)

Kubernetes deployment mounts secret as a folder instead of a file

Secrets let you store and manage sensitive information (e.g. passwords, private keys) and ConfigMaps are used for non-sensitive configuration data. As you’ll see within the Secrets associated ConfigMaps documentation:

A Secret is an object that contains a low quantity of sensitive information like a countersign, a token, or a key

A ConfigMap allows you to decouple environment-specific configuration from your container images, so that your applications are easily portable.

Mounting Secret as a file

It is possible to create Secret and pass it as a file or multiple files to Pods.
I've created a simple example for you to illustrate how it works. Below you can see a sample Secret manifest file and Deployment that uses this Secret:
NOTE: I used subPath with Secrets and it works as expected.

---
apiVersion: v1
kind: Secret
metadata:
  name: my-secret
data:
  secret.file1: |
    c2VjcmV0RmlsZTEK
  secret.file2: |
    c2VjcmV0RmlsZTIK
---
apiVersion: apps/v1
kind: Deployment
metadata:
...
    spec:
      containers:
      - image: nginx
        name: nginx
        volumeMounts:
        - name: secrets-files
          mountPath: "/mnt/secret.file1"  # "secret.file1" file will be created in "/mnt" directory
          subPath: secret.file1
        - name: secrets-files
          mountPath: "/mnt/secret.file2"  # "secret.file2" file will be created in "/mnt" directory
          subPath: secret.file2
      volumes:
        - name: secrets-files
          secret:
            secretName: my-secret # name of the Secret

Note: Secret should be created before Deployment.

After creating Secret and Deployment, we can see how it works:

$ kubectl get secret,deploy,pod
NAME                         TYPE                                  DATA   AGE
secret/my-secret             Opaque                                2      76sNAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1           76sNAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-7c67965687-ph7b8   1/1     Running   0          76s$ kubectl exec nginx-7c67965687-ph7b8 -- ls /mnt
secret.file1
secret.file2
$ kubectl exec nginx-7c67965687-ph7b8 -- cat /mnt/secret.file1
secretFile1
$ kubectl exec nginx-7c67965687-ph7b8 -- cat /mnt/secret.file2
secretFile2

Projected Volume

I think a better way to achieve your goal is to use projected volume.

A projected volume maps many existing volume sources into an equivalent directory.

Within the Projected Volume documentation you’ll be able to notice elaborated explanations however in addition, I created associate degree example that may assist you to perceive however it works. Using projected volume I mounted secret.file1, secret.file2 from Secret and config.file1 from ConfigMap as files into the Pod

---
apiVersion: v1
kind: Secret
metadata:
  name: my-secret
data:
  secret.file1: |
    c2VjcmV0RmlsZTEK
  secret.file2: |
    c2VjcmV0RmlsZTIK
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-config
data:
  config.file1: |
    configFile1  
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - name: all-in-one
      mountPath: "/config-volume"
      readOnly: true
  volumes:
  - name: all-in-one
    projected:
      sources:
      - secret:
          name: my-secret
          items:
            - key: secret.file1
              path: secret-dir1/secret.file1
            - key: secret.file2
              path: secret-dir2/secret.file2
      - configMap:
          name: my-config
          items:
            - key: config.file1
              path: config-dir1/config.file1

We can check how it works:

$ kubectl exec nginx -- ls /config-volume
config-dir1
secret-dir1
secret-dir2    
$ kubectl exec nginx -- cat /config-volume/config-dir1/config.file1
configFile1
$ kubectl exec nginx -- cat /config-volume/secret-dir1/secret.file1
secretFile1
$ kubectl exec nginx -- cat /config-volume/secret-dir2/secret.file2
secretFile2

If this response doesn’t answer your question, please provide more details about your Secret and what exactly you want to achieve.

For More, Recommendations

https://kubernetes.io/docs/concepts/configuration/secret/

https://platform9.com/blog/kubernetes-secrets-management/

https://blogs.oracle.com/developers/5-best-practices-for-kubernetes-security

https://dev.to/martinpham/secure-your-kubernetes-application-with-https-ng7

👋 Join us today !!

️Follow us on LinkedIn, Twitter, Facebook, and Instagram

If this post was helpful, please click the clap 👏 button below a few times to show your support! ⬇