Today we will see how to request Kubernetes API from within a pod in the cluster. I assume you already have a kubernetes cluster deployed somewhere. On my side, I will spin a sandbox cluster on my PC with k3d.

nginx deployment

For the purpose of this blog post, I will create a nginx deployment in my sandbox cluster:

kubectl create deploy --image=nginx nginx

create a dedicated serviceAccount and RBAC

Most of the time, you will get a 403 Forbidden response if you try to request the KubeAPI from within the cluster. You will need to create a dedicated service account and role.

You can do it by creating this rbac.yaml file:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-sa
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app: my-role
  name: "my-role"
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app: my-role-binding
  name: "my-role-binding"
  namespace: default
subjects:
  - kind: ServiceAccount
    name: my-sa
    namespace: default
roleRef:
  kind: Role
  name: "my-role"
  apiGroup: rbac.authorization.k8s.io

We will create here 3 resources:

  • a serviceAccount my-sa
  • a Role my-role with get and list rights on pods
  • a RoleBinding my-role-binding to bind the serviceAccount to the Role.

Let’s create these resources:

kubectl apply -f rbac.yaml

Please note these resources will be created in the default namespace

Create a pod to interact with Kube API

Create this alpine-pod.yaml file:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: alpine
  name: alpine
spec:
  containers:
  - image: alpine
    name: alpine
    command:
      - sleep
      - "3600"
    resources: {}
  serviceAccountName: my-sa
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

I created this yaml file with this command:

kubectl run --image=alpine alpine --dry-run=client -o yaml > alpine-pod.yaml

I just added the serviceAccountName: my-sa to make this pod use this dedicated serviceAccount.

Play with the Kube API with curl

In each pod deployed in any k8s cluster, you will find in /var/run/secrets/kubernetes.io some useful needed infos to interact with the Kube API:

  • The Kube API server url
  • The internal certificate authority
  • The namespace where the pod is running
  • The pod’s serviceAccount token

It is time now to jump into the container.

kubectl exec -it alpine -- /bin/ash

There is no curl by default in alpine pod so install it with apk add curl and define these variables:

# Point to the internal API server hostname
APISERVER=https://kubernetes.default.svc
 
# Path to ServiceAccount token
SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
 
# Read this Pod's namespace
NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)
 
# Read the ServiceAccount bearer token
TOKEN=$(cat ${SERVICEACCOUNT}/token)
 
# Reference the internal certificate authority (CA)
CACERT=${SERVICEACCOUNT}/ca.crt

You can now fetch the list of pods running in the namespace with this curl command:

curl -s --location --cacert ${CACERT} "${APISERVER}/api/v1/namespaces/${NAMESPACE}/pods" -H 'Accept: application/json' -H "Au
thorization: Bearer ${TOKEN}"

The above command will return a json with the pods running in the namespace. In my case, it returns only a nginx pod :-)

Update RBAC to get list of deployments

If you try to get the list of deployments instead of the pods:

curl -s --location --cacert ${CACERT} "${APISERVER}/apis/apps/v1/namespaces/${NAMESPACE}/deployments" -H 'Accept: application

You will get a 403 Forbidden :-(

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "deployments.apps is forbidden: User \"system:serviceaccount:default:my-sa\" cannot list resource \"deployments\" in API group \"apps\" in the namespace \"default\"",
  "reason": "Forbidden",
  "details": {
    "group": "apps",
    "kind": "deployments"
  },
  "code": 403
}

Your serviceAccount doesn’t have enough rights to do that. :-(

To fix it, update your rbac.yaml file with a new set of rules for your role “my-role”:

(...)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app: my-role
  name: "my-role"
  namespace: default
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
(...)

Apply your new rbac.yaml file:

kubectl apply -f rbac.yaml

And you should be able to get the list of deployments:

curl -s --location --cacert ${CACERT} "${APISERVER}/apis/apps/v1/namespaces/${NAMESPACE}/deployments" -H 'Accept: application

Tips and trick

How to grab the curl URL ?

The URL for requesting pods is different from the one for deployements:

# pods
curl -s --location --cacert ${CACERT} "${APISERVER}/api/v1/namespaces/${NAMESPACE}/pods" -H 'Accept: application/json' -H "Au
# deployments
curl -s --location --cacert ${CACERT} "${APISERVER}/apis/apps/v1/namespaces/${NAMESPACE}/deployments" -H 'Accept: application

I guess you wonder how to easily grab the good url to make API calls with curl ?

It is easy with the verbose level 6 of kubectl:

# For pods
$ kubectl get -v6 pod
(...)
I1031 17:45:22.818153  143417 round_trippers.go:553] GET https://0.0.0.0:41249/api/v1/namespaces/default/pods?limit=500 200 OK
(...)
# For deployments
$ kubectl get -v6 deploy
(...)
I1031 17:49:58.189061  143503 round_trippers.go:553] GET https://0.0.0.0:41249/apis/apps/v1/namespaces/default/deployments?limit=500 200 OK in 15 milliseconds
(...)

You can see api/v1/namespaces/default/pods for pods and apis/apps/v1/namespaces/default/deployments for deployments.

curl command for rollout restart

Let’s see another case, a rollout restart of a deployment. As this API call is not a GET but a PATCH, you will need additional information from kubectl, so let’s use the -v8 flag:

$ kubectl -v8 rollout restart deploy nginx
(...)
I1031 18:12:08.359926  144285 request.go:1154] Request Body: {"spec":{"template":{"metadata":{"annotations":{"kubectl.kubernetes.io/restartedAt":"2022-10-31T18:12:08+01:00"}}}}}
I1031 18:12:08.360129  144285 round_trippers.go:463] PATCH https://0.0.0.0:41249/apis/apps/v1/namespaces/default/deployments/nginx?fieldManager=kubectl-rollout
(...)
I1031 18:12:08.360176  144285 round_trippers.go:469] Request Headers:
I1031 18:12:08.360237  144285 round_trippers.go:473]     Accept: application/json, */*
I1031 18:12:08.360285  144285 round_trippers.go:473]     User-Agent: kubectl/v1.25.1 (linux/amd64) kubernetes/e4d4e1a
I1031 18:12:08.360331  144285 round_trippers.go:473]     Content-Type: application/strategic-merge-patch+json
(...)

As you can see, there is a Request Body who will patch the kubectl.kubernetes.io/restartedAt annotation with the date of restart of the deployment. This patch will automatically trigger a rollout restart. There is also a mandatory Request Header to make the rollout restart work: Content-Type: application/strategic-merge-patch+json

It is a bit more tricky than a simple GET but here is the curl command who will perform a rollout restart of your deployment (with the help of the date command):

curl -s --location --cacert ${CACERT} --request PATCH "${APISERVER}/apis/apps/v1/namespaces/${NAMESPACE}/deployments/nginx" -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: application/strategic-merge-patch+json" --data '{
        "spec": {
          "template": {
              "metadata": {
                  "annotations": {
                      "kubectl.kubernetes.io/restartedAt": "'$(date +%Y-%m-%dT%T)'"
                  }
              }
          }
      }
   }'

This command will fail:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "deployments.apps \"nginx\" is forbidden: User \"system:serviceaccount:default:my-sa\" cannot patch resource \"deployments\" in API group \"apps\" in the namespace \"default\"",
  "reason": "Forbidden",
  "details": {
    "name": "nginx",
    "group": "apps",
    "kind": "deployments"
  },
  "code": 403
}

Because you need to add the patch verb to the rules of your role in the rbac.yaml file!

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app: my-role
  name: "my-role"
  namespace: default
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["patch", "get", "list"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]

Apply the rbac.yaml file and you should be able to perform the rollout restart with curl.

Use kubectl instead of curl

It is cool but boring to type these long curl commands. Another option is to create a pod containing the kubectl binary, You will be able to use kubectl commands with the context of the service account, without any additional configuration ;-)

I hope you liked this post, please enjoy!