K8s Affinity and Tolerations: The Nightclub Bouncer Model

Kubernetes scheduling has two mechanisms that sound redundant until you understand they do opposite things. Affinity says “I want to go there.” Tolerations say “I can survive there.” You need both, and the reason you need both is the same reason a nightclub has both a guest list and a bouncer.

The analogy

Think of Kubernetes nodes as venues. Each node has characteristics — how much RAM, what kind of CPU, whether it has GPUs. Some nodes are VIP sections: reserved for specific workloads, off-limits to random pods.

Labels + Affinity = the guest list. Your pod says “I want to be in the VIP section” (affinity to nodes with label processing-unit: cicd). This is preference. The pod is requesting a specific type of node.

Taints + Tolerations = the bouncer. The VIP node says “nobody gets in unless they can handle my rules” (taint mai/cicd=true:NoSchedule). This is enforcement. The node is rejecting pods that don’t belong.

You need both:

The first row is the one that bites you. You carefully write a nodeAffinity rule, deploy, and the pod sits in Pending state. kubectl describe pod says “0/12 nodes are available: 3 node(s) had taints that the pod didn’t tolerate, 9 node(s) didn’t match Pod’s node affinity.” You matched the affinity but forgot the toleration. The guest list says yes, the bouncer says no.

The real config

Here’s what our CI runner pods use to land on dedicated build machines:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: mai.azure.com/processing-unit
                operator: In
                values: ["cicd"]
  tolerations:
    - key: "mai/cicd"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

Let’s unpack both sides.

The affinity side

requiredDuringSchedulingIgnoredDuringExecution

This mouthful means: “required when scheduling the pod, but if the node’s labels change after the pod is running, don’t evict it.” There’s also preferredDuringSchedulingIgnoredDuringExecution, which is a soft preference — “I’d like cicd nodes, but if none are available, put me somewhere else.”

We use required because CI builds need specific hardware. A Docker build on a node with 16GB RAM instead of 256GB will OOM and fail. It’s not a preference — it’s a requirement.

The match expression:

- key: mai.azure.com/processing-unit
  operator: In
  values: ["cicd"]

This selects nodes where the label mai.azure.com/processing-unit has the value cicd. The In operator checks membership — you could list multiple values if you wanted the pod to accept different node types.

Labels are key-value pairs on nodes. They’re metadata — they don’t enforce anything by themselves. A node labeled processing-unit: cicd doesn’t reject non-CI pods. It just has a label. Enforcement comes from taints.

The toleration side

tolerations:
  - key: "mai/cicd"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

This says: “I tolerate the taint mai/cicd=true with effect NoSchedule.” The taint is on the node:

kubectl describe node cicd-node-1
# Taints: mai/cicd=true:NoSchedule

NoSchedule means the scheduler won’t place pods on this node unless they explicitly tolerate this taint. Pods already running aren’t evicted (that would be NoExecute). It’s a scheduling-time gate, not a runtime enforcement.

Three taint effects:

We use NoSchedule because we want to reserve the nodes, not kill existing workloads.

Why taints exist

Without taints, any pod with the right resource requests could land on our CI nodes. A researcher submits a training job that requests 32 CPUs. The scheduler sees the cicd node has 32 CPUs available. Pod lands on the CI node. Now Docker builds are competing with a training job for CPU, and both are slow.

The taint prevents this. The cicd nodes say “I’m tainted with mai/cicd=true:NoSchedule.” Random pods don’t tolerate this taint, so the scheduler ignores these nodes for them. Only pods that explicitly add the toleration — our CI runner pods — can land there.

This is resource isolation without namespaces or resource quotas. Simpler, more direct. Taint the nodes you want to reserve, tolerate the taint from the pods that belong.

The three-layer architecture

Setting up dedicated CI nodes involves three layers, each with its own configuration:

Layer 1: Terraform

The node pool is defined in Terraform with the AKS module:

resource "azurerm_kubernetes_cluster_node_pool" "cicd" {
  name                = "cicd"
  vm_size             = "Standard_E32ds_v5"
  node_count          = 3
  enable_auto_scaling = false

  node_labels = {
    "mai.azure.com/processing-unit" = "cicd"
  }

  node_taints = [
    "mai/cicd=true:NoSchedule"
  ]
}

Three nodes, no autoscaling. Fixed capacity. We know exactly how many CI runners we need — three concurrent builds max. Autoscaling adds complexity (cold start latency, scale-down disruption) that’s not worth it for a predictable workload.

Layer 2: Runner config

The GitHub Actions runner controller (ARC) deploys runner pods using a values.yaml that injects the affinity and toleration:

template:
  spec:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: mai.azure.com/processing-unit
                  operator: In
                  values: ["cicd"]
    tolerations:
      - key: "mai/cicd"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"

This is the bridge between infrastructure and workloads. Terraform creates the nodes. ARC creates the runner pods. The affinity + toleration glues them together.

Layer 3: Workflow

GitHub Actions workflows reference the runner label:

jobs:
  build:
    runs-on: arc-runner-set-cicd

arc-runner-set-cicd is the runner set name in ARC. When GitHub dispatches a job with this label, ARC creates a runner pod using the template from Layer 2, which has the affinity and toleration from Layer 2, which targets the nodes from Layer 1.

Three layers. Three config files. One scheduling decision. Change any layer without updating the others and the pod either can’t schedule, schedules on the wrong node, or never gets created.

The node pool specs

The VM choice isn’t random:

E-series VMs give you 8 GB per vCPU. D-series (general purpose) gives 4 GB per vCPU. For Docker builds that pull and decompress 40GB images, the extra memory headroom prevents OOM kills during docker buildx operations.

256 GB sounds excessive until you realize a single Docker build can have three 40GB images in memory simultaneously — the base image being pulled, the new image being built, and the build cache from the previous run. Plus the runner process, plus whatever test workload runs in parallel. It adds up.

The mental model

Kubernetes scheduling is two independent systems that look like they do the same thing:

Affinity is the pod’s preference. “I want to be here.” It’s pull-based — the pod reaches toward specific nodes. Without it, the scheduler puts pods wherever there’s capacity.

Tolerations are the pod’s passport. “I’m allowed to be here.” It’s gate-based — the node blocks pods that don’t have the right toleration. Without it, tainted nodes are off-limits.

Labels and taints are on the node. Affinity and tolerations are on the pod. They’re configured in completely different YAML blocks, at different spec levels, by potentially different teams (infra vs platform vs application).

The nightclub model: affinity is the guest list (pod-side desire), taints are the rope (node-side enforcement), tolerations are the VIP pass (pod-side credential). You need to be on the list AND have the pass. The bouncer doesn’t care about the list. The list doesn’t know about the bouncer. They’re independent checks that both must pass.

Get one wrong and your pod sits in Pending, quietly waiting for a node that will never accept it.