How to Use NamespaceHound to Apply PEACH to your K8s Clusters

Running multi-tenant clusters? Red-teaming your Kubernetes environments? Access "NamespaceHound", our tool to verify tenant separation and find lateral movement paths:

Use for free

Overview

There are two widely used multi-tenancy models in Kubernetes: multi-cluster and multi-namespace. In the multi-cluster model, each tenant operates in a separate cluster. In this case, higher-level cloud security boundaries and considerations apply – the cluster itself serves as a security boundary between tenants. We will focus on the multi-namespace model, where each tenant operates in its own K8s namespace. In this case, cluster architecture – both control plane and data plane – affects security boundaries. Thus, the hardening factors of these boundaries should be measured in the context of the cluster architecture.

Kubernetes architecture

Kubernetes was designed to manage distributed workloads. The workloads are typically represented by a pod, which, in turn, is comprised of one or more containers. This automatically imposes container security boundaries between the different workloads. In addition to containerization, Kubernetes offers Namespaces as a means of isolating groups of resources. The following architecture diagram illustrates the security boundaries in a production cluster and will help us reason about the multi-namespace tenancy model in the next sections:

The typical threat model for multi-tenant clusters assumes execution capabilities in any of the tenants. In this classic scenario an attacker will find themselves confined by a series of several security boundaries (see diagram below):

Container / pod execution boundaries
Namespace resource and RBAC boundaries
Worker node locality boundaries

Given the selection of the attacker goals (be it accessing other tenant’s data, abusing other tenant’s resources, or performing a Denial of Service), for each of the boundaries, the attacker can try and employ various TTPs to cross them (refer to the K8s Threat Matrix for the detailed list of TTPs).

The unique architecture of a Kubernetes cluster, along with the unique set of isolation controls, imposes specific framework changes that we discuss below.

Determine security boundaries in K8s cluster

In a multi-tenant cluster, the tenants are typically exposed via a data plane access means, such as load balancer service or application gateway. The interface functionality, however, is not different from other cloud and application interfaces. Employ similar techniques for exposed Kubernetes services.

Hardware Separation

In some cluster, namespace-level separation can be augmented with node affinity. The scheduling algorithm will follow the designed affinity model to only schedule the workloads belonging to a certain tenant to the assigned K8s worker node. Of course, this will impose a performance hit in terms of number of extra nodes needed to support the load.

Network Segmentation

Instances belonging to different tenants run on separate machines (virtual or physical) in the same network, which is segmented into single tenant VPNs. Use of Network Policies between namespaces is encouraged.

Containerization

Instances belonging to different tenants run in separate containers running on shared hardware (or nested within a shared virtual machine) via operating-system-level virtualization.

Identity Segmentation

Present by default. K8s service accounts are namespaced.

Data Segmentation

Control plane configuration is stored in ETCD and thus (even if encrypted) is shared among the tenants. Data plane resources are typically namespaced, however, certain types of resources are global and require per-resource review. K8s RBAC is granular enough to ensure service account access per resource name.

Measure Hardening Factors in K8s cluster

The strength of a security boundary is limited by the weakest link in its design or implementation. We can measure the strength of a namespace security boundary according to 5 hardening factors (P.E.A.C.H.):

Privilege Hardening
- Kubernetes RBAC (role-based access control) is a method of regulating access to cluster resources based on the roles of individual principals within the cluster. RBAC should be considered a strong security boundary only if the following guidelines are adhered to:
  - There are two types of roles in K8s RBAC: namespace-level Role and cluster-level ClusterRole. Tenant principals must be only assigned Roles that are not global admins.
  - It is important to follow the principle of least privileges when assigning roles to principals. Assign only necessary verb access to specific resources and never assign * to verbs or resources.
  - Be careful when assigning privileges that allow host access to tenant principals (for example, /pod:create). Unless a pod is sandboxed, host is a shared medium between the namespaces and can be used to access other namespace data and secrets.
  - Use Admission Controller to enforce out-of-line behaviors, such as running privileged pods, starting pods running as root, starting pods with the default service account, etc.

Encryption hardening
- Some of the Kubernetes resources are globally scoped (i.e. Webhooks) and thus may be shared across tenants. Sensitive data or workflows should not use global resources. If they do – employ additional authentication via additional RBAC dedicated ClusterRoles.
- PersistentVolume and PersistentVolumeClaim are widely used K8s objects for managing custom storage access in the cluster. Official Kubernetes documentation on multi-tenancy suggests configuring a separate StorageClass for each tenant and using it to strengthen isolation. If a StorageClass is shared, you should set a reclaim policy of Delete to ensure that a PersistentVolume cannot be reused across different namespaces.
- If using external secret storage make sure to isolate the storage access per tenant.

Authentication Hardening
- Consider the need of tenant communication with API server. If not needed – opt-out of automounting service account token into the pods – either on a pod (good) or kubelet (better) level via the usage of automountServiceAccountToken: false option.

Connectivity Hardening
- By default, any pod within the cluster is allowed to communicate with any other pod. Cross tenant and tenant egress connectivity should be blocked by Kubernetes Network Policies. It is advisable to use namespace labels for Network Policies as a more stable labeling mechanism over pod labels.

Summarize Findings and Testing

NamespaceHound is an open-source tool developed by Wiz for detecting the risk of potential namespace crossing violations in multi-tenant clusters. Given a cluster, NamespaceHound will run analysis and determine all the possible ways to cross the security boundaries between namespaces. It is the perfect tool to test the outcome of applying PEACH to a particular cluster.

Use for free

Usage

In addition to revealing the cross-namespace issues, the tool inspects the cluster configuration for anonymous access opportunities. If given a specific namespace (via the -n namespace parameter), it will focus on this namespace plus anonymous access to find all possible ways to reach or interfere with resources in another namespace.

For help, use –help option:

Here is an example output of running the tool:

To review the library of potential violations and learn more details about NamespaceHound, please refer to this Github repository.

Presented with 🤍 by