TYPO3 on Kubernetes under IT-Grundschutz, part 6: Network and tenant separation

01 — A namespace per tenant

Each tenant gets its own namespace. It is the unit that permissions, network rules, quotas and Pod Security attach to. Pod Security Admission enforces the restricted profile through a label:

apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest

Among other things, the profile forbids privileged containers, host namespaces and running as root. It requires a seccomp profile and dropping all capabilities.

A ResourceQuota caps how much CPU, memory and storage a tenant may use in total. A LimitRange sets defaults for individual containers. That way, a tenant under load or with a bug cannot crowd out the others.

02 — Default deny for ingress and egress

The first NetworkPolicy in the namespace denies everything, in both directions. Targeted allows per component follow: the ingress controller may reach the proxy, the proxy may reach the PHP runtime, the application may reach database and cache, and all may reach DNS.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: tenant-a
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: app-egress
  namespace: tenant-a
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: app
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - {port: 53, protocol: UDP}
        - {port: 53, protocol: TCP}
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/component: db
      ports:
        - {port: 3306, protocol: TCP}
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/component: cache
      ports:
        - {port: 6379, protocol: TCP}
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16
              - 169.254.0.0/16
      ports:
        - {port: 443, protocol: TCP}

Internet access is limited to port 443. Private networks and the range 169.254.0.0/16 are excluded. That range holds the cloud metadata service, through which a compromised pod could otherwise obtain the node's credentials. In dual-stack operation the same block is needed for IPv6, or everything stays open there. More on dual-stack is in the post on IPv6-only and dual-stack on Kubernetes.

Important: NetworkPolicies only take effect if the CNI plugin enforces them. A quick test with a pod that tries a forbidden connection is therefore part of the setup.

03 — Admission rules instead of hope

Pod Security covers the standard profile. Custom rules come in through a ValidatingAdmissionPolicy. It has been generally available since Kubernetes 1.30 and needs no external webhook. The rules are CEL expressions that the API server evaluates itself:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: tenant-pod-rules
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: [CREATE, UPDATE]
        resources: [pods, pods/ephemeralcontainers]
  variables:
    - name: images
      expression: "object.spec.containers.map(c, c.image) + (has(object.spec.initContainers) ? object.spec.initContainers.map(c, c.image) : []) + (has(object.spec.ephemeralContainers) ? object.spec.ephemeralContainers.map(c, c.image) : [])"
  validations:
    - expression: "!has(object.spec.hostNetwork) || !object.spec.hostNetwork"
      message: "hostNetwork is not allowed"
    - expression: "!has(object.spec.volumes) || object.spec.volumes.all(v, !has(v.hostPath))"
      message: "hostPath is not allowed"
    - expression: "variables.images.all(i, i.startsWith('registry.example.org/'))"
      message: "images only from our own registry"
    - expression: "has(object.spec.automountServiceAccountToken) && !object.spec.automountServiceAccountToken"
      message: "do not mount the service account token automatically"

The registry rule deliberately checks every container of a pod, including initContainers and debug containers attached later. Otherwise a foreign image would simply slip into the pod through one of those routes. That is why the policy also covers the pods/ephemeralcontainers subresource. The debug image for kubectl debug from part 2 then has to come from your own registry as well.

A ValidatingAdmissionPolicyBinding binds the rule to all tenant namespaces. A second rule checks that every tenant namespace carries the restricted label. That way, nobody can weaken Pod Security unnoticed.

The rule on the service account token has a simple reason. A TYPO3 application needs no access to the Kubernetes API. An automatically mounted token would only be an extra tool for an attacker after a compromise.

04 — Where isolation ends

Namespaces, NetworkPolicies and admission rules separate tenants at the Kubernetes level. All pods on a node still share the same kernel, though. A kernel vulnerability can cross these boundaries.

With a high protection need, there are two ways to push the boundary further down:

• Dedicated nodes for individual tenants or for applications with different protection needs. Taints, tolerations and node selectors make sure only matching pods land there.
• Sandboxing with gVisor or Kata Containers. The container gets its own kernel or an intermediate layer. That costs performance and does not suit every application.

How it maps to IT-Grundschutz

Planning the separation is part of APP.4.4.A1. Pod Security and NetworkPolicies implement the separation of pods (APP.4.4.A4) and networks (APP.4.4.A7). The rule against automatically mounted tokens belongs to the use of service accounts (APP.4.4.A9). For an increased protection need, dedicated nodes (APP.4.4.A14), separation at node and cluster level (APP.4.4.A15), micro-segmentation (APP.4.4.A18) and further isolation and encapsulation of containers (SYS.1.6.A26) come on top. From SYS.1.6, the separation of administration and access networks applies (SYS.1.6.A5), and from network planning NET.1.1.

Conclusion

Tenant separation in a cluster is a series of layers. The namespace organises, Pod Security limits, NetworkPolicies cut down the paths, and admission rules stop anyone from softening these layers by accident. With a high protection need, the honest question is whether a shared kernel is still enough.

The series continues with part 7: secrets without Git. Back to part 5: mTLS on every hop.