DevOps Attacks Managed Kubernetes cluster, Part 1.

About six months (in 2019 - editor comment) ago, I got my hands on one public (multi-tenant) Kubernetes cluster. Public means that there are some namespaces; the namespaces have users isolated in their namespace. All of the users belong to different companies. The cluster was intended to serve as a CDN. Put simply - you are given cluster access with a user role; you come there to your namespace and deploy your front pages.


I was asked to poke a cluster on the subject — to test whether such a solution is suitable or not for one of my previous employers.


I came to this cluster. I was given minimum rights, limited namespace. I assumed the guys (cluster admins - editor comment) understood what security is. They read what Kubernetes' Role-based access control (RBAC) is — and they configured it in such a way that I could not run pods separately from deployments. I don't remember the problem that I tried to solve by running it without deployment, but I wanted to run just the pod. I decided to see what rights I have in the cluster, just for the case — what I can do, what I cannot, what they twisted there. Further, I'll tell you what they have misconfigured in RBAC.


Just two minutes later, I got admin rights to their cluster, discovered all the neighbouring namespaces, saw running production fronts of companies that had already bought the service and deployed there. I barely stopped coming to someone's front page and putting an obscene word on the main page.


I will explain how I did it and how to defend against it, with examples.I will talk about the underwater reefs and challenges present in incorrect (or default) Kubernetes cluster security settings. We see these issues when we work with providers who deploy Kubernetes clusters and work with clients who come to our company. Hell, we see those issues when we work as consultants for other companies' clients, meaning the tragedy's scope is enormous.


Here are the three topics I am going to cover today:

1. Kubernetes privilege escalation - user privileges vs pod privileges. Those are entirely different things.
2. Gathering intel on the cluster. You can collect all information about the cluster, even if you do not have admin rights there.
3. DoS-attacking the cluster. Even if you face a well-protected cluster and cannot gather any data from it, you can still attack and shut down the vital elements of any K8s cluster.

Another essential point is describing how I tested it all, the environment where this story happened.

A Kubernetes cluster is deployed with Kubespray, which is a set of Ansible roles. It is very convenient, and we use it all the time, as it allows deploying Kubernetes anywhere — from bare metal clusters to cloud environments. One size fits all.


I had Kubernetes v1.14.5 in the cluster, split into namespaces for different teams, with user access rights only for the team's namespace. There also is an admin account with full access rights to the cluster.

I promised that the first thing we will show is getting admin rights to the cluster. We need a specially prepared pod that will hack the Kubernetes cluster. All we need to do is apply it to the Kubernetes cluster.

kubectl apply -f pod.yaml

This pod will come to one of the masters of the Kubernetes cluster. And after that, the cluster will happily return a file called admin.conf. In K8s, this file stores all the administrator's certificates, and at the same time, the cluster API configuration. It's so easy to get admin access, I think, to 98% of Kubernetes clusters are vulnerable in such a way.

Again, this pod was made by one developer in your cluster who has access to deploy his projects in a tiny namespace; he is all squeezed by RBAC limits. He had no rights. Nevertheless, the certificate was returned.

Let's discuss the pod in detail. We can run it on any image - let's take debian: jessie as an example. Here is the thing:

tolerations:
- effect: NoSchedule
operator: Exists
nodeSelector:
node-role.kubernetes.io/master: ""

What is toleration? Master nodes in a Kubernetes cluster are usually labelled with a thing called taint. And the essence of this "taint" is that it says that pods cannot be assigned to master nodes. But no one bothers to prohibit any pod to be tolerant of "taint". The Toleration section just declares that if there is NoSchedule on some node, our pod is tolerant of such a taint — and there are no problems with taint labels.


Further, we declare that our pod is not just tolerant but wellbeing from hitting the master nodes precisely. Because the masters carry the most delicious thing that we need — all the certificates. Therefore, we declare nodeSelector — and we have a standard label on the masters, which allows you to select from all the nodes in the cluster, precisely those nodes that are masters.


With these two sections, the pod will become the master node, and he will be allowed to live there.


But just becoming the master is not enough for us. It won't give us anything. Because, further, we have the following two security controls applied:

hostNetwork: true
hostPID: true


We indicate that our pod, which we're running, will live in the kernel namespace, network namespace, and PID namespace. As soon as the pod is launched as the master one, it will see all the node's live interfaces, listen to all traffic, and see all processes' PIDs.

The rest is self-explanatory. Grab etcd daemon and read whatever you want.


The most interesting thing is the following Kubernetes feature, which is default one:
volumeMounts:
- mountPath: /host
name: host
volumes:
- hostPath:
path: /
type: Directory
name: host

In essence, we can say that we want to create a volume hostPath in the pod that we will run, even without having rights in this cluster. It means we want to take the path from the host on which we will start — and take it as volume. Then we call it name: host. We mount this hostPath inside the pod. In this example, to the / host directory.


That's our attack chain - we told the pod to become the master, get hostNetwork and hostPID there - and mount the entire root master inside this pod.


Please remember - in Debian, we have bash inside, and this bash is running under root. That means we just received root access for the master node while not having any rights in the Kubernetes cluster itself.


The task is to go into the /host/etc/kubernetes/pki directory; if I'm not mistaken, take all the master certificates of the cluster there and become the cluster administrator.


If you look at it this way, these are some of the most dangerous rights in pods — regardless of what rights the user has:

• Privileged
• Root user
• Host namespace
• Host path volume

If I have the rights to run a pod in some cluster namespace, then this pod has these rights by default. I can run privileged pods, which means I generally have all rights, practically root access to the master node.

My favourite is the Root user. You see, Kubernetes has this Run As Non-Root option. This is a type of protection control. Do you know what the "Moldovan virus" is? If you are suddenly a hacker and come to my Kubernetes cluster, then we, poor administrators, ask: "Please indicate in your pods with which you will hack my cluster, to run as non-root. Otherwise, it so happens that you start the process in your pod under the root, and it will be straightforward for you to hack me. Please protect us from yourself. "

Host path volume is the fastest way to hack a Kubernetes cluster.

In next section we'll study how to prevent pod rights abuse and privilege escalation.

Thoughts that should come to any regular Linux admin guy who encounters Kubernetes: "Yeah, I told you, Kubernetes doesn't work. There are vulnerabilities in it. And the whole K8S is bullshit." Look no further than in official documentation, particularly at the section called Pod Security Policy.


There is a yaml object (we can create it in the Kubernetes cluster), which controls the pod description's security aspects. Pod Security Policy controls rights to use any hostNetwork, hostPID, certain types of volumes.


Please note - all PSP installers are simply disabled by default. Pod Security Policy is enabled using the admission plugin.


Let's deploy the Pod Security Policy to the cluster; for instance, we have some service pods in the namespace, which only admins can access. In our example, in all other pods, they have limited rights. Because most likely, developers do not need to run privileged pods on your cluster.


We seem to be doing well now, and our Kubernetes cluster cannot be hacked in two minutes, yeah? There is a Part 2, covering intel on a Kubernetes cluster and a DoS attack on it. Stay tuned for updates!