K8s SecurityContext: Why fsGroup Alone Won’t Fix Permission Denied

The CI runner pod couldn’t create symlinks on its PVC mount. ln: failed to create symbolic link 'sgl_latest.sqsh': Permission denied. We had fsGroup set. The volume was mounted. The pod was running. It just couldn’t write.

This kicked off a four-hour debugging session that taught me more about Kubernetes security contexts than I ever wanted to know. The root cause is a subtle interaction between five fields that the docs explain individually but never explain together.

The five fields

Kubernetes has five security context fields that affect file permissions. They’re split across two levels:

The confusion starts because these fields are at different levels. runAsUser and runAsGroup are per-container. fsGroup and supplementalGroups are per-pod. They all affect the same permission check, but they’re configured in different YAML blocks.

How they interact

Here’s what fsGroup actually does. When Kubernetes mounts a volume into a pod with fsGroup: 12345, it does two things:

1. Recursively chowns all files on the volume to group 12345
2. Sets group permissions — ensures g+rwx on directories, g+r on files

This happens at pod startup, before any container runs. The volume’s files are now owned by group 12345 with group-writable permissions.

But here’s the critical part that the docs bury: the process running inside the container must actually be a member of that group. Setting fsGroup changes the files. It does not change the process.

If your container runs as runAsUser: 1000 and nothing else, the process has:

• UID: 1000
• Primary GID: 0 (root group, the default)
• Supplemental groups: none

The volume files are owned by group 12345. The process is not in group 12345. Permission denied.

The missing link

The fix is runAsGroup. This field sets the primary GID of the container process. When it matches fsGroup, the circle closes:

spec:
  securityContext:
    fsGroup: 346669362        # volume files owned by this group
  containers:
    - name: runner
      securityContext:
        runAsUser: 1000
        runAsGroup: 346669362  # process IS in this group

Now:

• Volume files: owned by group 346669362, permissions g+rwx
• Process: UID 1000, primary GID 346669362
• Permission check: process GID matches file group → access granted

The number 346669362 looks arbitrary and enormous. It’s not — it’s the GID assigned by Azure AD for the service identity. Enterprise environments love these galaxy-brained GIDs.

Why symlinks are special

Our specific failure was symlink creation, not just file writing. Creating a symlink requires write permission on the directory — not the target file, not the symlink itself, but the directory where the symlink will live.

# This needs write permission on /mnt/ci/images/
ln -s /mnt/ci/images/sgl_20260207.sqsh /mnt/ci/images/sgl_latest.sqsh

Directory write permission comes from the g+rwx that fsGroup sets. But only if the process is in the group. Reading a file from the volume might work with just the group-read bit (g+r). Writing a file needs group-write (g+w). Creating or deleting entries in a directory needs group-write on the directory.

Symlink creation is a directory modification — you’re adding a directory entry. If you can read files on the volume but can’t create symlinks, the problem is almost always that you have read access but not write access to the directory. Which means you’re not in the group that fsGroup set up for you.

The debugging checklist

When you hit Permission denied on a PVC mount in Kubernetes, walk through this:

1. Which container fails?

In a multi-container pod, each container has its own securityContext. One might work while another doesn’t. In our case, the dind (Docker-in-Docker) sidecar had privileged: true and could do anything. The runner container had restrictive settings and couldn’t create symlinks.

2. Check fsGroup on the pod spec.

kubectl get pod <name> -o jsonpath='{.spec.securityContext.fsGroup}'

If this is empty, volume files keep their original ownership. No group-based access.

3. Check what UID/GID the failing container runs as.

kubectl exec <pod> -c <container> -- id
# uid=1000 gid=0(root) groups=0(root)

If the groups list doesn’t include the fsGroup GID, that’s your problem. The process isn’t in the group that owns the volume.

4. Check if runAsGroup matches fsGroup.

kubectl get pod <name> -o jsonpath='{.spec.containers[?(@.name=="runner")].securityContext.runAsGroup}'

If this is empty or doesn’t match fsGroup, add it.

5. Verify directory permissions on the volume.

kubectl exec <pod> -c <container> -- ls -la /mnt/ci/
# drwxrwsrwx 2 root 346669362 4096 Feb 7 10:00 images

The s in rwsrwx means the setgid bit is set — new files created in this directory inherit the group. This is another thing fsGroup does. If you see drwxr-xr-x (no group write), fsGroup isn’t applied to this volume.

The matrix of confusion

Here’s every combination of the three most common fields and what happens:

The second row is the trap. You set fsGroup, you see the volume files owned by the right group, you assume it works. It doesn’t, because the process isn’t in the group. Kubernetes set up the room perfectly but didn’t give you the key.

The privileged escape hatch

Our Docker-in-Docker sidecar runs with privileged: true. This gives the container full host-level access — it bypasses all Linux capability checks, including file permission checks. It’s the nuclear option.

containers:
  - name: dind
    securityContext:
      privileged: true  # can do anything, permission checks don't apply

This is why the dind container never had permission issues. It’s not that the permissions were right — it’s that privileged means permissions don’t matter. The process operates as if it were running directly on the host with root access.

You should never use privileged: true as a fix for permission issues on application containers. It works, but it also means a container escape gives an attacker full host access. Fix the actual permission chain instead.

The minimal fix

For our runner, the fix was one line:

  containers:
    - name: runner
      securityContext:
        runAsUser: 1000
+       runAsGroup: 346669362

Combined with the existing pod-level fsGroup: 346669362, this closes the loop. Process is in group 346669362. Volume files are owned by group 346669362. Symlink creation requires directory write. Directory has g+rwx. Permission granted.

One YAML line. Four hours of debugging. The Kubernetes experience.

The real lesson

The docs for each of these five fields are individually correct and clear. fsGroup “defines a special supplemental group that applies to all containers in a pod.” runAsGroup “specifies the primary group ID for all processes in the container.”

What the docs don’t say — what you learn by watching Permission denied scroll past in CI logs at 11pm — is that these fields form a system. fsGroup without runAsGroup is a lock without a key. runAsUser without runAsGroup means your GID defaults to 0. privileged makes the question irrelevant but creates a security hole.

The fields are designed to be composed. Nobody tells you the composition rules. Now you know them.