Our CI builds Docker images and pushes them to four Azure Container Registries. One of them — maimanifold — was being migrated from global endpoints to regional endpoints, the same way maifalcon already works. The change looked trivial. One flag:
- az acr login --name maimanifold
+ az acr login --name maimanifold --endpoint westus2
It failed with:
The resource with name 'maimanifold' and type 'Microsoft.ContainerRegistry/registries'
could not be found in subscription 'GenAI Dev AME'
Understanding why that one flag change broke everything requires understanding a stack of Azure concepts that nobody explains well until you’re already in trouble. So here’s the glossary I wish I’d had, taught through the debugging story.
The cast of jargon
Subscription
An Azure subscription is a billing and access boundary. Think of it as a project that determines who pays for resources and who can see them. Our subscription is GenAI Dev AME — a specific Azure account that the infra team manages. Every Azure resource (VMs, storage, container registries) lives inside exactly one subscription.
The mental model: a subscription is like a credit card. Resources you create get charged to it, and you control who gets to use it.
Tenant
A tenant is the Azure Active Directory (now called Entra ID) organization. It’s the company-level identity boundary — the thing that says “these users and these subscriptions all belong to Microsoft.” All subscriptions live under a tenant.
If a subscription is a credit card, the tenant is the wallet. You can have multiple cards, but they all belong to one person (or in this case, one organization).
Service Principal
A service principal (SP) is a robot account. When CI runs in GitHub Actions, it can’t type a password into a browser. Instead, it authenticates using a service principal — a non-human identity with a client-id, tenant-id, and subscription-id. It’s the bot that logs in on your behalf.
The SP used by our build workflow is shared infrastructure: the DevEx team uses it in CI, and the infra team grants it permissions. This coordination is where things go wrong.
Azure Container Registry (ACR)
A Docker registry hosted in Azure. You push and pull container images to and from it. Like Docker Hub, but private and inside your Azure environment. We push to four of them: inf5acr, maifalcon, mangopdx, and maimanifold.
Regional Endpoints
By default, az acr login --name maimanifold talks to maimanifold.azurecr.io — a global endpoint that routes you to wherever Azure feels like. Regional endpoints let you talk to a specific datacenter: maimanifold.westus2.geo.azurecr.io. Closer server, lower latency, better reliability during regional outages.
This is what the PR was migrating to. maifalcon was already using regional endpoints without issues. maimanifold should have been the same. It was not.
The two planes
This is the concept that actually explains the bug, so it gets its own section.
Azure has two layers for interacting with any resource:
Control plane — managing resources. Creating, deleting, listing, configuring. When you ask “what ACRs exist in this subscription?”, that’s a control plane operation. It goes through ARM (Azure Resource Manager), which is the central management API for all of Azure.
Data plane — using resources. Pushing a Docker image, pulling a Docker image, reading a blob. The actual work you care about.
These are separate systems with separate permissions. You can have data plane access (push and pull images) without control plane access (can’t even see that the registry exists in the subscription). This is not a hypothetical edge case. It’s exactly what happened.
RBAC: the permission system
RBAC (Role-Based Access Control) determines who can do what. Azure has built-in roles for ACR:
| Role | Plane | What it does |
|---|---|---|
AcrPush |
Data | Push images |
AcrPull |
Data | Pull images |
Reader |
Control | See resources via ARM |
Contributor |
Both | Full read/write, no role assignment |
Our service principal had AcrPush and AcrPull. It could push and pull images all day. But it likely did not have Reader on the maimanifold ACR — meaning ARM didn’t know the SP was allowed to see it.
This never mattered before. Until we added --endpoint.
Why --endpoint breaks it
Here’s the key insight from the debugging session.
Without --endpoint, az acr login does roughly this:
- Try to find the ACR as an ARM resource (control plane lookup)
- If that fails, fall back to direct Docker credential exchange (data plane)
- Either way, you end up logged in
With --endpoint, the behavior changes:
- Try to find the ACR as an ARM resource (control plane lookup)
- Use the ARM response to resolve the regional endpoint URL
- If step 1 fails, hard-fail. No fallback.
The regional endpoint needs ARM to tell it “yes, maimanifold exists, and its westus2 endpoint is at maimanifold.westus2.geo.azurecr.io.” Without control plane visibility, that lookup returns “resource not found” — and az acr login gives up.
The basic login was always failing the ARM lookup too. It just didn’t care, because it had a fallback path. Adding --endpoint removed the safety net, and the underlying permissions gap became a hard failure.
The debugging path
The error showed up in the upload_image_manifold job in our image build workflow. The first instinct was to isolate it.
I created a test workflow (test_manifold_regional.yml) that ran only the login step with some diagnostic commands. One problem: workflow_dispatch (manual trigger) only works for workflow files that exist on the default branch. A new workflow file on a feature branch can’t be manually triggered. Workaround: use a push trigger on the branch instead.
The test workflow ran these checks:
| Check | Result | Meaning |
|---|---|---|
| DNS resolution of regional endpoint | Passed | The endpoint exists, networking is fine |
az acr login --name maimanifold (basic) |
Passed | Data plane auth works, fallback kicks in |
az acr login --name maimanifold --endpoint westus2 |
Failed | ARM lookup fails, no fallback |
az account list |
Only 1 subscription visible | SP has limited scope |
That narrowed it down. DNS is fine. Data plane works. Control plane doesn’t. The SP can use the registry but can’t see it through ARM.
Who owns what
This is the kind of bug that lives in the gap between two teams:
DevEx team (us) — owns the CI workflows. We build images and push them. We changed the workflow to use --endpoint. We own the “how” of the push.
Infrastructure team — owns the ACR resources, creates regional endpoints, manages RBAC on the registries. They own the “what” — the registry exists, it’s configured correctly, the right identities have the right roles.
The service principal — the shared identity that both teams touch. DevEx uses it in CI. Infra grants it permissions. When something goes wrong with it, both teams look at each other.
The infra team confirmed: the regional endpoints were set up correctly, and the subscription ID was right. The issue was that the SP didn’t have the control plane visibility it now needed. When --endpoint wasn’t in the picture, the gap in permissions was invisible.
The fix
Grant the service principal Reader (or equivalent control-plane role) on the maimanifold ACR resource, so the ARM lookup succeeds. Alternatively, scope the role at the subscription or resource group level so the SP can enumerate resources.
This is a one-line RBAC change on the infra side. The fact that it took a debugging session to get here is the whole point — the error message says “resource not found,” not “you don’t have permission to see this resource.” Azure’s error messages for permission issues are famously misleading. A 403 dressed up as a 404.
Takeaways
- Data plane and control plane are separate permission sets. You can push Docker images to a registry without being able to see it exists. This will work fine until it suddenly doesn’t.
- Fallback paths hide bugs.
az acr loginwithout--endpointwas silently failing the ARM lookup every single time. We never noticed because the fallback worked. Adding--endpointdidn’t introduce a new bug — it revealed one that was always there. - “Resource not found” often means “you can’t see it.” Azure frequently returns 404-style errors for permission problems. If a resource exists and you get “not found,” check RBAC before checking whether the resource actually exists.
- Test infrastructure changes in isolation. The test workflow with diagnostic commands was the fastest path to understanding. Running the full build and hoping for the best would have been slower and noisier.
- Shared identities need shared ownership. When a service principal spans two teams, neither team has full visibility. The team that uses it doesn’t control the permissions. The team that controls the permissions doesn’t see how it’s used. Coordination costs are real.