KubePrism

Kubernetes pods running in CNI mode can use the kubernetes.default.svc service endpoint to access the Kubernetes API server, however pods running in host networking mode can only use the external cluster endpoint to access the Kubernetes API server.

Because Kubernetes controlplane and CNI components run in host networking mode, they can only use the external cluster endpoint to access the Kubernetes API server. If the external cluster endpoint is unavailable (due to misconfiguration, network issues, etc), this will cause issues in the cluster: pods will not be scheduled, service IPs stop working, etc.

KubePrism solves this problem by enabling an in-cluster highly-available controlplane endpoint on every node in the cluster.

Video Walkthrough

To see a live demo of this writeup, see the video below:

Enabling KubePrism

As of Talos 1.6, KubePrism is enabled by default with port 7445.

To enable KubePrism, apply the following machine config patch either during the machine config generation, or to a running cluster (the patch should be applied to all nodes):

machine:
  features:
    kubePrism:
      enabled: true
      port: 7445

Note: the port specified should be available on every node in the cluster.

How it works

Talos spins up a TCP loadbalancer on every machine on the localhost on the specified port which automatically picks up one of the endpoints:

• the external cluster endpoint as specified in the machine configuration
• for controlplane machines: https://localhost:<api-server-local-port> (http://localhost:6443 in the default configuration)
• https://<controlplane-address>:<api-server-port> for every controlplane machine (based on the information from Cluster Discovery)

KubePrism automatically filters out unhealthy (or unreachable) endpoints, and prefers lower-latency endpoints over higher-latency endpoints.

Talos automatically reconfigures kubelet, kube-scheduler and kube-controller-manager to use the KubePrism endpoint. The kube-proxy manifest is also reconfigured to use the KubePrism endpoint by default, but when enabling KubePrism for a running cluster the manifest should be updated with talosctl upgrade-k8s command.

When using CNI components that require access to the Kubernetes API server, the KubePrism endpoint should be passed to the CNI configuration (e.g. Cilium, Calico CNIs).

Notes

As the list of endpoints for KubePrism includes the external cluster endpoint, KubePrism in the worst case scenario will behave the same as the external cluster endpoint. For controlplane nodes, the KubePrism should pick up the localhost endpoint of the kube-apiserver, minimizing the latency. Worker nodes might use direct address of the controlplane endpoint if the latency is lower than the latency of the external cluster endpoint.

KubePrism listen endpoint is bound to localhost address, so it can’t be used outside the cluster.