Python Packaging: PyPI Registries and Proxies Demystified

I sat in a meeting where my team discussed setting up a PyPI registry on Azure DevOps. They mentioned proxy registries, Nexus licensing costs, auth blockers, TLS issues. I nodded along.

I had no idea what half of it meant.

So I went and figured it out. Here’s the mental model I wish someone had given me before that meeting — and the actual architecture we ended up building.

The bakery and the convenience store

There are two kinds of package registries, and confusing them is where most of the confusion starts.

A hosted registry (private PyPI) stores packages your team builds. Internal libraries, proprietary code — stuff that doesn’t belong on public PyPI. You publish to it. You own it. If it’s not there, it doesn’t exist.

A proxy registry stores nothing of its own. It sits between your machines and pypi.org, forwarding requests and caching the results. First time someone pip install requests? The proxy fetches it from the internet and saves a copy. Second time? Served from cache. The internet never gets hit again.

Think of it this way — a hosted registry is a bakery. It only sells bread it bakes. A proxy registry is a convenience store. It stocks popular items from wholesale so you don’t have to drive there every time.

Most teams use both, behind a single URL. You pip install my-internal-lib and it finds it in the hosted registry. You pip install numpy and it finds it in the proxy cache. One URL, two sources, zero confusion for the developer.

Why proxy registries in every cluster

This one confused me too. Why not just one proxy?

Because production clusters usually can’t reach the internet. Network isolation is a security requirement. So without a proxy inside the cluster:

Pod → pip install numpy → ❌ Can't reach pypi.org

With a local proxy:

Pod → pip install numpy → Proxy (inside cluster) → ✅ Cached copy
                              ↓ (first time only)
                            pypi.org

There are other reasons — local caches are faster, they survive pypi.org outages, and they give you a chokepoint for vulnerability scanning. But the network isolation thing is the big one.

Azure DevOps Artifacts — the free option you probably already have

Azure Artifacts is a package management service built into Azure DevOps. If your team is already on Azure DevOps, you have it. It supports PyPI feeds out of the box.

You create a “feed” (Azure’s word for registry), point pip at it, and you’re done. It can host your internal packages and proxy public PyPI — all behind one URL. Free up to 2 GB.

Setting one up is roughly:

1. Azure DevOps → Artifacts → Create Feed
2. Enable “upstream sources” (this is the proxy part)
3. Configure pip.conf with the feed URL + auth
4. Publish internal packages with twine upload

That’s it. No servers to manage. No licenses to buy.

So what’s Nexus and why does it cost money

Sonatype Nexus is a self-hosted package server. You install it on your own infrastructure, and it manages packages for Python, Java, Docker, npm — everything in one place.

Nexus OSS (open source) is free and genuinely capable. It does hosted registries, proxy registries, grouped registries. For a lot of teams, it’s enough.

Nexus Pro is where the money comes in — $50K to $120K+ per year. You get high availability, vulnerability scanning, enterprise SSO, and Sonatype support. JFrog Artifactory is the main competitor and it’s even more expensive.

Here’s the comparison that matters:

If you’re already on Azure DevOps, Azure Artifacts is the obvious first thing to try. Nexus makes sense when you need self-hosted control or multi-format support beyond what Azure Artifacts offers. Nexus Pro makes sense when you need enterprise features and have the budget.

The actual architecture — it’s not either/or

Here’s what confused me the most: I thought we were choosing between Azure Artifacts and Nexus. Turns out we’re using both. At different layers.

┌──────────────────────────────────────────┐
│  Azure DevOps Artifacts (PyPI feed)      │
│  ┌──────────────────────────────┐        │
│  │  Source of truth for         │        │
│  │  internal packages           │        │
│  │  + corporate auth (Azure AD) │        │
│  │  + TLS built-in              │        │
│  └──────────────────────────────┘        │
└──────────────┬───────────────────────────┘
               │
      ┌────────┼────────┐
      ▼        ▼        ▼
  Cluster 1  Cluster 2  Cluster 3
  ┌──────┐   ┌──────┐   ┌──────┐
  │Nexus │   │Nexus │   │Nexus │    ← one per cluster
  │(OSS) │   │(OSS) │   │(OSS) │       local cache + proxy
  └──┬───┘   └──┬───┘   └──┬───┘
     │          │          │
     ▼          ▼          ▼
   Pods       Pods       Pods       ← pip install just works

Azure DevOps is the central registry — where developers publish internal packages. One place, corporate auth, managed by Microsoft.

Nexus per cluster is the local cache — where pods consume packages. Fast, local, no internet dependency.

Why this design:

• Developers publish once to Azure DevOps. Don’t care how many clusters there are.
• Each cluster is self-sufficient. If Azure DevOps goes down, Nexus serves from cache.
• Auth is simplified. The hard auth problem (PATs, managed identity) only lives between Nexus → Azure DevOps. Pods → Nexus is cluster-internal, no auth headaches.
• Network isolation respected. Pods never reach the internet directly.

What Nexus actually does in each cluster

Inside each Nexus instance, you set up three repos:

pypi-group (the single URL pods use)
    │
    ├── pypi-hosted        ← packages published directly to this Nexus
    │                         (rare, mostly for cluster-specific stuff)
    │
    ├── pypi-azure-proxy   ← proxies to Azure DevOps feed
    │                         (internal packages, needs auth)
    │
    └── pypi-public-proxy  ← proxies to pypi.org
                              (numpy, requests, etc.)

The group repo is the magic. Pods point at one URL — the group. Nexus checks each member in order: hosted first, then Azure proxy, then public proxy. The developer doesn’t know or care which source served the package.

The flow when a pod does pip install:

pip install my-internal-lib
    → Nexus group → checks hosted → nope
                  → checks azure-proxy → proxies to Azure DevOps → found!
                  → caches locally → returns to pod

pip install numpy
    → Nexus group → checks hosted → nope
                  → checks azure-proxy → nope
                  → checks public-proxy → proxies to pypi.org → found!
                  → caches locally → returns to pod

pip install numpy (second time)
    → Nexus group → checks public-proxy → cached! → returns immediately

The connection between Nexus and Azure DevOps

This is the piece that requires actual configuration. Nexus needs to talk to Azure DevOps, and Azure DevOps needs to trust Nexus.

In the Nexus UI, when you create the pypi-azure-proxy repo, the Remote storage field points at your Azure DevOps feed:

https://pkgs.dev.azure.com/your-org/_packaging/pypi-internal/pypi/

But Azure DevOps feeds aren’t public. Nexus needs credentials:

TLS is already handled — Azure DevOps is HTTPS by default. No cert wrangling on this side.

The real blockers — TLS and auth

Setting up the registry is the easy part. Getting TLS and authentication right is where teams spend days debugging.

TLS is the encryption that puts the “S” in HTTPS. When pip talks to your registry, TLS encrypts the traffic and verifies identity. The problem? Your internal registry probably uses a certificate signed by your company’s internal CA — and pip doesn’t trust it by default.

pip install my-package
  ERROR: SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

The fix is telling pip where your company’s CA certificate is:

export PIP_CERT=/path/to/company-ca-bundle.crt

That’s it. But finding which certificate file, and making sure it’s available in every cluster and CI runner — that’s where the pain is.

Auth is how pip proves you’re allowed to download packages. A few options:

• PAT (Personal Access Token) — a revocable token in your pip config. Good for developers.
• Managed Identity — Azure auto-authenticates your pod. No credentials to manage. Best for CI/CD and AKS.
• Basic auth — username and password. Simple but risky.

The nastiest auth problem is the chicken-and-egg: you need the artifacts-keyring package to authenticate to your private registry, but you can’t download it without authenticating. Pre-install it in your Docker base images.

Where we are now

Here’s the actual status of our setup:

The Nexus deployment itself — Helm chart, Kustomize overlay, ArgoCD application, node scheduling, ingress — is all done. The registry configuration and Azure DevOps wiring are also done. What’s left is the networking/DNS side and developer-facing docs.

Setting up PyPI repos in Nexus

I described the four repos earlier — pypi-hosted, pypi-azure-proxy, pypi-public-proxy, and pypi-group. Here’s how to actually create them.

Via REST API (what we did)

Nexus has a full REST API. Once you have the URL and admin credentials, creating repos is just curl calls. Here’s the full set:

NEXUS_URL="http://nexus-falcon-phx-ca-staging:8081"
NEXUS_AUTH="admin:admin123"
ADO_TOKEN=$(az account get-access-token \
  --resource 499b84ac-1321-427f-aa17-267ca6975798 \
  --query accessToken -o tsv)

# 1. pypi-hosted (cluster-specific internal packages)
curl -X POST "${NEXUS_URL}/service/rest/v1/repositories/pypi/hosted" \
  -u "${NEXUS_AUTH}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "pypi-hosted",
    "online": true,
    "storage": {
      "blobStoreName": "default",
      "strictContentTypeValidation": true,
      "writePolicy": "ALLOW"
    }
  }'

# 2. pypi-azure-proxy (proxies to ADO feed)
curl -X POST "${NEXUS_URL}/service/rest/v1/repositories/pypi/proxy" \
  -u "${NEXUS_AUTH}" \
  -H "Content-Type: application/json" \
  -d "{
    \"name\": \"pypi-azure-proxy\",
    \"online\": true,
    \"storage\": {
      \"blobStoreName\": \"default\",
      \"strictContentTypeValidation\": true
    },
    \"proxy\": {
      \"remoteUrl\": \"https://pkgs.dev.azure.com/msasg/Falcon/_packaging/Falcon_PublicPackages/pypi/\",
      \"contentMaxAge\": 1440,
      \"metadataMaxAge\": 1440
    },
    \"httpClient\": {
      \"blocked\": false,
      \"autoBlock\": true,
      \"authentication\": {
        \"type\": \"username\",
        \"username\": \"any\",
        \"password\": \"${ADO_TOKEN}\"
      }
    },
    \"negativeCache\": {
      \"enabled\": false,
      \"timeToLive\": 1440
    }
  }"

# 3. pypi-public-proxy (proxies to pypi.org as fallback)
curl -X POST "${NEXUS_URL}/service/rest/v1/repositories/pypi/proxy" \
  -u "${NEXUS_AUTH}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "pypi-public-proxy",
    "online": true,
    "storage": {
      "blobStoreName": "default",
      "strictContentTypeValidation": true
    },
    "proxy": {
      "remoteUrl": "https://pypi.org/",
      "contentMaxAge": 1440,
      "metadataMaxAge": 1440
    },
    "httpClient": {
      "blocked": false,
      "autoBlock": true
    },
    "negativeCache": {
      "enabled": true,
      "timeToLive": 1440
    }
  }'

# 4. pypi-group (the single URL pods use — combines all 3)
curl -X POST "${NEXUS_URL}/service/rest/v1/repositories/pypi/group" \
  -u "${NEXUS_AUTH}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "pypi-group",
    "online": true,
    "storage": {
      "blobStoreName": "default",
      "strictContentTypeValidation": true
    },
    "group": {
      "memberNames": [
        "pypi-hosted",
        "pypi-azure-proxy",
        "pypi-public-proxy"
      ]
    }
  }'

The 499b84ac-1321-427f-aa17-267ca6975798 is the Azure DevOps resource ID — same for everyone, hardcoded by Microsoft. The username in the auth block can be anything (any, user, foo) — ADO only checks the token.

Via Nexus UI (if you prefer clicking)

1. Login as admin → Settings (gear icon) → Repositories → Create repository
2. Choose pypi (hosted) → name: pypi-hosted, default blob store → Create
3. Choose pypi (proxy) → name: pypi-azure-proxy, Remote URL: https://pkgs.dev.azure.com/msasg/Falcon/_packaging/Falcon_PublicPackages/pypi/, Authentication: Username any, Password: your ADO token → Create
4. Choose pypi (proxy) → name: pypi-public-proxy, Remote URL: https://pypi.org/ → Create
5. Choose pypi (group) → name: pypi-group, add all 3 as members (hosted first, then azure-proxy, then public-proxy) → Create

The order in the group matters — Nexus checks members top to bottom. Hosted first means your internal packages always win over public ones with the same name.

Testing the whole pipeline

Once the repos are created, you want to prove the whole chain works before telling anyone it’s ready. Here’s the sequence.

Step 1: Get an ADO token

TOKEN=$(az account get-access-token \
  --resource 499b84ac-1321-427f-aa17-267ca6975798 \
  --query accessToken -o tsv)

This gives you a short-lived OAuth token (~1 hour). Good enough for testing — production should use a PAT or managed identity.

Step 2: Test ADO feed directly

Before testing through Nexus, make sure the ADO feed itself works:

uv venv /tmp/test-ado && source /tmp/test-ado/bin/activate
uv pip install -v \
  --index-url "https://any:${TOKEN}@pkgs.dev.azure.com/msasg/Falcon/_packaging/Falcon_PublicPackages/pypi/simple/" \
  requests 2>&1 | grep "GET request for"

Look for pkgs.dev.azure.com in the download URLs — not files.pythonhosted.org. If packages come from the ADO URL, the upstream proxy is working. That means the Falcon_PublicPackages feed is pulling from pypi.org on your behalf.

Step 3: Test Nexus group (the URL pods will use)

uv venv /tmp/test-nexus && source /tmp/test-nexus/bin/activate
uv pip install -v \
  --index-url "http://nexus-falcon-phx-ca-staging:8081/repository/pypi-group/simple/" \
  requests 2>&1 | grep "GET request for"

All downloads should come from nexus-falcon-phx-ca-staging:8081. No internet needed — Nexus proxied through ADO, which proxied through pypi.org. Your pod only talked to Nexus.

Verification cheat sheet

A few things that tripped me up:

• uv requires an active venv — unlike pip with --break-system-packages, uv refuses to install into system Python. Create a throwaway venv.
• The az token expires in ~1 hour — if your tests start failing after a coffee break, get a new token.
• Add -v to see what’s happening — verbose output shows the actual download URLs, which is how you prove traffic is flowing through the right proxy.

What “Simple” actually means — the PyPI protocol

Before the debugging story, you need to understand the protocol that makes all of this work. It’s called the Simple Repository API (PEP 503), and it really is simple. Two endpoints. That’s it.

Endpoint 1: Package index

GET /simple/

Returns an HTML page listing every package name as a link:

<a href="/simple/requests/">requests</a>
<a href="/simple/flask/">flask</a>
<a href="/simple/numpy/">numpy</a>

Endpoint 2: Package versions

GET /simple/requests/

Returns an HTML page listing every version of that package as a download link:

<a href="https://files.pythonhosted.org/packages/.../requests-2.32.5-py3-none-any.whl#sha256=...">requests-2.32.5-py3-none-any.whl</a>
<a href="https://files.pythonhosted.org/packages/.../requests-2.32.5.tar.gz#sha256=...">requests-2.32.5.tar.gz</a>

Each link has an href pointing to the actual .whl or .tar.gz file, plus a #sha256=... fragment for integrity verification. That’s the whole protocol. pip and uv hit endpoint 1 to discover packages, endpoint 2 to find versions, and then download the file from whatever URL the href points at.

Here’s the part that matters for our debugging story: different registries put different URLs in those hrefs.

pypi.org points downloads at its CDN:

<a href="https://files.pythonhosted.org/packages/.../requests-2.32.5-py3-none-any.whl#sha256=...">
  requests-2.32.5-py3-none-any.whl</a>

Azure DevOps points downloads back at itself:

<a href="https://pkgs.dev.azure.com/msasg/.../_packaging/.../pypi/download/requests/2.32.5/requests-2.32.5-py3-none-any.whl#sha256=...">
  requests-2.32.5-py3-none-any.whl</a>

Both are valid. Both use absolute URLs. Both work perfectly with pip and uv directly. The difference only shows up when a proxy tries to follow those download URLs on your behalf — which is exactly what Nexus does.

When Nexus returns 404 and it’s not what you think

This is the part of the blog post I wish I’d had before we started. The fix was a one-line change. Finding it took hours.

The symptom

pip install requests --index-url http://nexus:8081/repository/pypi-azure-proxy/simple/
# → 404 Not Found

Same package, same Nexus, through pypi-public-proxy:

pip install requests --index-url http://nexus:8081/repository/pypi-public-proxy/simple/
# → ✅ Works fine

Through pypi-group it also worked — but only because pypi-public-proxy was the fallback. The azure proxy was silently failing and the group was routing around it. Every package was coming from pypi.org, none from ADO. Our whole “proxy to Azure DevOps for internal packages” story was broken.

Step 1: It’s always auth, right?

First instinct: the token expired. It’s always the token.

# Refresh the token
ADO_TOKEN=$(az account get-access-token \
  --resource 499b84ac-1321-427f-aa17-267ca6975798 \
  --query accessToken -o tsv)

# Update the proxy config via Nexus REST API
curl -X PUT "${NEXUS_URL}/service/rest/v1/repositories/pypi/proxy/pypi-azure-proxy" \
  -u "${NEXUS_AUTH}" \
  -H "Content-Type: application/json" \
  -d "{...new token...}"

# Invalidate caches for good measure
curl -X POST "${NEXUS_URL}/service/rest/v1/repositories/pypi-azure-proxy/invalidate-cache" \
  -u "${NEXUS_AUTH}"

Still 404. Not auth.

Step 2: Can the pod even reach ADO?

Maybe it’s a network issue. We exec’d into the Nexus pod to test:

kubectl exec -it nexus-repository-manager-6b67997f75-2ltnp -n nexus -- /bin/bash

curl -s -u "any:${ADO_TOKEN}" \
  "https://pkgs.dev.azure.com/msasg/Falcon/_packaging/Falcon_PublicPackages/pypi/simple/requests/" \
  | head -5

HTTP 200. Valid HTML. Package links everywhere. The pod can reach ADO with proper auth. Network is not the problem.

Step 3: Read the logs

This is where I should have started. The Nexus application logs told a completely different story than “auth failure” or “network error”:

Failed to resolve package Unable to find reference for itsdangerous-2.2.0-py3-none-any.whl in package itsdangerous

Not a 401. Not a connection timeout. A format parsing error. Nexus was connecting to ADO, getting the Simple index HTML, but then failing to resolve the download URLs for individual packages.

This was the moment the debugging shifted from “what’s broken” to “what does Nexus expect and what is it getting?”

Step 4: Compare the Simple index formats

I pulled the Simple index HTML from both ADO and pypi.org for the same package. Both were valid HTML. Both had absolute URLs in the hrefs. Both conformed to PEP 503.

So what’s Nexus actually doing when it processes these links?

Step 5: Follow the download chain

The download URLs from ADO have a twist. When you GET a package download URL from ADO:

GET /pypi/download/requests/2.32.5/requests-2.32.5-py3-none-any.whl
  → 303 redirect to Azure Blob Storage (vsblob.vsassets.io) with SAS token
  → 200 (actual file)

The download URL requires the same auth as the Simple index. Without auth, you get a 401. But Nexus was getting past auth to list packages — it was failing on the download step. Why?

Step 6: The eureka moment

I checked the remoteUrl configuration for both proxy repos:

// pypi-public-proxy (WORKS)
"remoteUrl": "https://pypi.org/"

// pypi-azure-proxy (BROKEN)
"remoteUrl": "https://pkgs.dev.azure.com/msasg/Falcon/_packaging/Falcon_PublicPackages/pypi/simple/"

See it?

Nexus PyPI proxy repositories automatically append /simple/ to whatever remoteUrl you give them. That’s how the protocol works — the remoteUrl is the base, and Nexus knows to add /simple/ to reach the index.

With pypi.org/ as the base, Nexus hits pypi.org/simple/ — correct.

With .../pypi/simple/ as the base, Nexus hits .../pypi/simple/simple/ — double /simple/.

But wait — the listing worked. We could see packages. So what was actually breaking?

The listing might have worked by accident (ADO may be lenient about the double path on the index endpoint), but the download URL resolution was wrong. When Nexus reads the Simple index, it gets absolute download URLs. It then needs to resolve those URLs relative to its understanding of the remote server’s base URL. With the wrong base URL, the path math for downloads was broken, and Nexus couldn’t map the download hrefs back to something it knew how to fetch.

The error message now makes perfect sense: “Unable to find reference for itsdangerous-2.2.0-py3-none-any.whl” — Nexus found the file listed in the index but couldn’t resolve the download path.

The fix

- "remoteUrl": "https://pkgs.dev.azure.com/msasg/Falcon/_packaging/Falcon_PublicPackages/pypi/simple/"
+ "remoteUrl": "https://pkgs.dev.azure.com/msasg/Falcon/_packaging/Falcon_PublicPackages/pypi/"

Six characters removed. /simple/ deleted from the end.

After updating the proxy config and invalidating the cache:

pip install flask --index-url http://nexus:8081/repository/pypi-azure-proxy/simple/

Flask plus all 7 dependencies — Werkzeug, Jinja2, itsdangerous, click, blinker, MarkupSafe, importlib-metadata — installed perfectly. Through ADO. Through Nexus. Zero errors.

Why this was so hard to find

1. The listing worked. The Simple index returned valid HTML with real packages. You’d never guess the remoteUrl was wrong from the listing behavior alone.
2. The error message was misleading. “Unable to find reference” sounds like the package doesn’t exist in the feed. It doesn’t sound like a URL resolution problem.
3. pypi.org worked fine. The public proxy masked the problem because pypi.org/ was already the correct format — no trailing /simple/. Easy to assume the ADO config was correct too.
4. Auth was a red herring. With ADO, auth issues are so common that “token expired” is always the first guess. We burned time refreshing tokens when auth was never the problem.
5. Negative caching made it worse. With negativeCache enabled, Nexus cached the 404 results. Even after fixing config, you had to explicitly invalidate the cache — otherwise Nexus would keep serving stale 404s for packages it had already failed to resolve.

The lesson

Always check what URL the tool actually expects. Nexus PyPI proxy appends /simple/ itself. The remoteUrl is the base, not the index endpoint. pypi.org’s remoteUrl is https://pypi.org/, not https://pypi.org/simple/. Same rule applies to Azure DevOps: https://pkgs.dev.azure.com/.../pypi/, not https://pkgs.dev.azure.com/.../pypi/simple/.

This is a general principle that shows up everywhere in infrastructure work: the tool has opinions about URL structure, and you need to match them. Duplicating what the tool already adds is a silent, confusing failure mode.

And disable negative caching on proxy repos that you’re still debugging. A cached 404 is the most frustrating kind of wrong.

Gotcha: ADO feed permission wall

We originally wanted to create a dedicated mai-pypi feed. Clean separation, our own space, our own rules. So I tried the REST API:

# Tried Falcon org, MAI UX project, MAI_DPS project, org-scope...
# All returned the same thing:
# 403 — CreateFeed permission denied

We don’t have CreateFeed permissions. Tried multiple orgs and projects — same wall everywhere. This is one of those enterprise permission things where the right answer is probably a ServiceNow ticket and a two-week wait.

So we pivoted. The Falcon_PublicPackages feed already existed, already had 141 PyPI packages, and already had pypi.org configured as an upstream source. It was doing exactly what we needed a new feed to do.

Sometimes the thing you need already exists. We pointed our Nexus proxy at Falcon_PublicPackages and moved on.

The mental model

Here’s how I think about the whole thing now:

Developer laptop                        Cluster (AKS)
┌──────────────┐                        ┌─────────────────────────┐
│  twine upload│                        │  Pod                    │
│  (publish)   │                        │  pip install (consume)  │
└──────┬───────┘                        └────────────┬────────────┘
       │                                             │
       ▼                                             ▼
┌──────────────────────┐                ┌─────────────────────────┐
│  Azure DevOps Feed   │◄───── proxy ───│  Nexus (per cluster)    │
│  (source of truth)   │                │  ┌───────┐ ┌─────────┐ │
│  + corporate auth    │                │  │hosted │ │proxy→ADO│ │
│  + TLS built-in      │                │  │       │ │proxy→pub│ │
└──────────────────────┘                │  └───────┘ └─────────┘ │
                                        │  = pypi-group URL      │
                                        └─────────────────────────┘

Developers publish to Azure DevOps (one place, corporate auth). Pods consume from Nexus (local, fast, cached). Nexus proxies to Azure DevOps for internal packages and pypi.org for public ones. One group URL, all packages, no internet dependency.

That’s the whole picture. Everything else is implementation details — and I’ve written about those too.