Kubernetes and Docker Security - Best Practices

Kubernetes Attack Vectors:

1. Access to Machine / VM
2. Host Compromise
3. Access to etcd API
4. Kubernetes uses etcd to store all its data, configuration data, state and also the metadata. 
5. Access via Kubernetes API or Proxy
6. K8s API provides master access to the cluster
7. Intercept / Modify / Inject Control Plane Traffic
8. Man-In-The-Middle Attack | CSRF | SQLi | IDOR
9. Access via Kubelet API
10. Cluster Control with Administrative Privileges.
11. Escaped Container to Host through vulnerabilities or volume mount
12. Weak and Vulnerable Code in the containers image, dependency or its application

First line of defense is to have multiple layers of security

These include securing the 4 C’s of Kubernetes

Cloud (the service provider)

• Gain the benefits of scale, implying that make the most of what the cloud vendor has on offer in terms of security, availability and disaster recovery.
• Make sure to follow the security recommendations made by the cloud vendor, it may be Multi-factor Authentication, IAM policies or Role Based Access Control within the IAM policies.
• Limiting access to authentic users only with RBAC.
• Restricting console access as most users do not need to have access to the administrative console.
• Limiting the programmatic access control to users with RBAC
• Disabling all public access to the internal cloud infrastructure.
• Encryption of data either via the cloud providers built in encryption strategies or implementation of custom encryption strategy. (Datastores and Databases)
• Implementing network security policies to allow or deny ports,IP addresses etc.

Cluster (Kubernetes Cluster)

Principle of least privilege

“ Restrict access so that different components can access only the information and resources they need to operate correctly”

In the incidence of an attack, the attacker only gets to access the fragments of information and resources that are available to that particular component limiting the damage.

• Fortunately, layers of defense available in K8s that makes it easy to prevent an attack.
• Limiting the API access to Authenticated requests (all others are ignored)
• Implementation of Role Based Access Control (RBAC)
• Securing the API Server
• Offers REST API for controlling Kubernetes, and a user with full permissions on this API has equivalent of a SuperUser access to every machine in the cluster.
• Kubectl is a client for this API and by default API server will listen to an insecure port i.e. 8080 and any requests to this ports bypass any authentication and authorization checks.
• Leaving the port open is a risk as anyone who gains access to the master can own the entire cluster.
• However, users need not worry about this anymore as it is on the verge of being phased out in the future versions of Kubernetes.

Disabling anonymous Kubelet access:

• Kubelets are responsible for interacting with the container runtime to launch pods, generate reports on node,pod and metrics.
• Kubelets also operates an API through which other components prompts it to do tasks such as starting and stopping of pods.
• Supposing that an attacker gets to access the Kubelet API, they can potentially own the cluster.
• Requires a client certificate and also a key to get authenticated access
• Ensuring that requests are authorized by a policy or a setting (Allow X : Deny Y)
• Limit the permissions of the kubelet by including Node Restrictions, such that it restricts a kubelet from modifying pods other than what is bound to it.
• Disabling the read only port. This port might not give the attacker with access to the cluster but may reveal valuable information pertaining to the internals of a cluster such as running workloads which might pave way to exploit any other vulnerability

Running etcd safely

• Requires setting up the requirement of certificate file and key file to enable a HTTPS connection.
• Enabling the authentication of client certificates
• Setting up of a trusted Certificate Authority list file to specify the Certificate authority that has signed the client certificates.
• Disabling the authentication of self signed certificates.
• Setting up of a requirement where in nodes communicate with each other securely using peer client authentication certificate. By specifying a peer certificate file and a peer key file along with a list of peer trusted certification authority

The Dashboard

• Historically has been the exploit point of choice to gain control of a k8s cluster.
• Allowing limited, known and authenticated users is recommended.
• Usage of RBAC, will limit the privilege of users such that they get to administer only the resources that they are authorized to.
• Dashboard service account should have minimum access.
• Finally, Exposure of the dashboard to a public domain is strongly discouraged.

Kubernetes Authentication

Name spacing: in Kubernetes, it is a way to logically divide the cluster into smaller units of management.

• Client presents its credentials to API server
• API Server uses one of the configured plugin to establish Identity with an ID provider.
• ID Provider verifies the request information including username password and group membership.
• If the credentials are valid, API server moves to check permissions or else throws a 401 - Unauthorised User error.
• Identity provider and its behaviour depends on the authentication plugin.
• It could simply be a file with usernames and passwords that you provide to an external system like an Active Directory

Role Based Access Control (RBAC)

• Entity: Group, User or service account (representing an app that wants to carry out a certain operation and requires permission in order to do so.
• Resource: A pod , service, or a secret that the entity wants to access.
• Role: used to define rules for actions on resources.
• Role Binding: This attaches (binds) to a role to an entity stating that a set of actions are permitted for certain entity on the specified resources

Authorization in Kubernetes

• Client request is authenticated.
• If the authentication was successful, the credentials are taken as one of the inputs of the authorization module.
• The second input to the authorization module is a vector containing request path, resource and namespace.
• If the user or application is permitted to execute a certain action on a certain resource, the request is passed on further to the next component in the chain, the admission controller. If not, then the authorization module returns an HTTP 403 forbidden client error and with that the request fails.

Authorization Modes:

• Node authentication: special purpose authorizer that grants permissions to kubelets based on the pods they are scheduled to run.
• Attribute Based Access Control

Container (The Docker container)

• Container vulnerability scanning, OS dependency security using tools such as Docker Bench, Dockscan, OpenSCAP workbench etc.
• The following are some recommended security practices.

Avoid running as root:

• Docker’s default setting for user in an image is root.
• To prevent escalated attacks, run as an unprivileged user
• Linux User ID 0 - 99 belongs to system 
• User ID 100 - 499 dynamically allocated by system administrators and post install scripts
• User ID 499 - 65535 are unprivileged.
• The user could be specified at the time of building an image
• /> docker run -u 9459 <my_image>
• However it is recommended to include the USER docker file instruction. Best practice is to include the USER instruction after all the other instructions that requires the capabilities that come with root

##### docker file#####

FROM <base image>

RUN < commands>

………

USER <$num>

Capabilities:

• Capabilities are a collection of allowed processes.
• Capabilities could be adjusted through the command line with --cap-drop and --cap-add
• Best policy is to use --cap-drop-all and then add back the necessary capabilities with --cap-add.
• Example: 

/> docker run --cap-drop=stop <my_image_name|image id>

• To inspect multiple aspects of the image including capabilities use the following command

/> docker container inspect <container_name>

Resource Management

• Restrict a containers access to system resources such as memory and CPU.
• Without a resource limiter, supposedly the container uses all the available memory, if and when that anomaly occurs, the Linux Kernel will enter a state termed Kernel Panic because of an OOM(OUT_OF_MEMORY) exception

Image Safety

• Usage of an Image from a trustworthy source is advised
• Images such as Alpine, Ubuntu, BusyBox, etc. are recommended.
• Using an image from a reputed organization leaves the developer with little worries as they are supported by a vast community.
• Quoting Linus Trovalds “Given enough eyes, all bugs are shallow”  implying that a resolution to an issue or a bug in the code of the kernel will be annihilated with swift actions, all in appreciation to the developer community.
• Usage of a minimalist base image is advised as it would promote a higher level of transparency and the time required to resolve issues.
• Usage of a leaner image with just the bare minimum dependencies installed reduces the surface of attack.
• Enabling the requirement of signed Images ensures that arbitrary images are not used during the build process as the image origins as they possess the risk of providing a malicious gateway to potential attackers.
• By default, the content trust (requirement of signed images) is turned off and enabling it is highly recommended.
• The content trust could be enabled by using the command

/> export DOCKER_CONTENT_TRUST=1

Secret Management

• Usage of Access Key ID and Secret Access Keys in plain text is strongly discouraged.
• Also Setting up Environmental Variables and Storing Keys in a file is frowned upon as by running a simple docker inspect command could reveal the keys in plain text.
• What is recommended is to use an Identity Access Management (IAM) service which is usually provided by the cloud platform supporting the content or the automation tool.

Code (the application deployed on the container)

• Access over TLS only, HTTPS 
• Limiting the Port range communication 
• Certain ports on the network do not 
• Checks the security of 3rd party dependencies 
• Static code analysis 
• Consider the idea of sending the application through a vulnerability scan.
• Emphasizing on encrypting critical information.
• Consider the recommendations of your repository provider (GitHub) to optimize security with respect to dependencies used.

References:

Overview of Cloud Native Security - Kubernetes | Docker Security 101: Risks and 33 Best Practices | StackRox | Learn Enough Docker to be Useful - Towards Data Science | Top 20 Docker Security Tips - Towards Data Science |Docker security Part 1 | Opensource.com | 9 Kubernetes Security Best Practices Everyone Must Follow - Cloud Native Computing Foundation |A hacker's guide to Kubernetes security | TechBeacon |Top 5 Kubernetes Best Practices From Sandeep Dinesh (Google)