Python Packaging: How BlobFuse Works

Your app calls open("/mnt/packages/numpy-1.26.whl"). The file is sitting in Azure Blob Storage, thousands of miles away. Your app has no idea. It just sees a file. Here’s how that works, starting from the bottom.

What happens when you open a file

When a program calls open() or read(), it doesn’t talk to a hard drive. It makes a syscall — a request to the Linux kernel. The kernel then figures out where that file actually lives.

The thing that does this figuring is the VFS — Virtual File System. VFS is an abstraction layer inside the kernel. It takes every filesystem operation (open, read, stat, readdir, etc.) and dispatches it to the right driver. ext4? NFS? tmpfs? Doesn’t matter. The app speaks one language — POSIX file operations — and VFS translates.

App: open("/mnt/data/file.txt")
        ↓
    Kernel syscall
        ↓
    VFS: "What's mounted at /mnt/data?"
        ↓
    ext4 driver / NFS driver / ???

This is the key insight you need for everything that follows: apps don’t know what filesystem they’re talking to. They talk to VFS, and VFS handles the rest.

FUSE: the rule-breaker

Normally, filesystem drivers live in kernel space. Writing a kernel module is painful — one bug and the whole machine panics. You need deep kernel knowledge, you can’t use most libraries, and debugging is brutal.

FUSE (Filesystem in Userspace) breaks this model. It’s a kernel module whose entire job is to route filesystem calls back to a userspace program.

Here’s what that looks like:

App: read("/mnt/fuse-mount/file.txt")
        ↓
    Kernel VFS: "This is a FUSE mount"
        ↓
    FUSE kernel module: forward to userspace
        ↓
    Your userspace daemon: handles the read however it wants
        ↓
    Response flows back through FUSE → VFS → app

The app still thinks it’s reading a normal file. VFS still dispatches normally. The only difference is that instead of an ext4 driver in kernel space handling the operation, a regular program in userspace handles it.

This is what makes projects like sshfs, rclone mount, and BlobFuse possible. You can write a “filesystem” in Python if you want. It’s just a program that responds to file operation requests.

The FUSE kernel module is the adapter — it speaks kernel on one side and userspace on the other. The /dev/fuse device is the communication channel between them.

BlobFuse: blob storage as a directory

BlobFuse2 is a FUSE-based daemon that makes Azure Blob Storage containers look like local directories. It registers a mount point (say, /mnt/packages), and from that point forward, any filesystem operation on that path gets routed through FUSE to the BlobFuse process.

The BlobFuse daemon translates filesystem operations into Azure Blob REST API calls:

App: read("/mnt/packages/numpy-1.26.whl")
        ↓
Kernel VFS: "FUSE mount, route to userspace"
        ↓
BlobFuse2 daemon:
    GET https://account.blob.core.windows.net/container/numpy-1.26.whl
        ↓
App sees: a normal file. No idea it came from the cloud.

stat() becomes a HEAD request. readdir() becomes a List Blobs call. read() becomes a GET with a byte range. Every POSIX operation maps to a REST call (or a cache hit — more on that next).

Two caching modes

Fetching from blob storage on every read() would be brutal. BlobFuse has two caching strategies, and which one you pick matters a lot.

File cache — downloads the entire blob to local disk on open(). Every subsequent read() hits local disk. Simple and effective when you’re going to read the whole file anyway. This is the mode you want for package installs — pip opens a .whl, reads the whole thing, and it’s cached. Second install? Zero network traffic.

Block cache (streaming) — downloads in chunks as the app reads. Only fetches the parts you actually touch. Good for huge files where you might only need a slice — think ML model checkpoints, large datasets, one-time sequential reads.

For the package install use case, file cache is the obvious pick. Packages are small-to-medium sized, read entirely, and accessed repeatedly across CI runs.

The package install pattern

Here’s the actual workflow. You have a blob container full of Python wheels — pre-built packages uploaded once. Mount it:

blobfuse2 mount /mnt/packages \
    --config-file=config.yaml \
    --tmp-path=/tmp/blobfuse-cache

Now pip can find them:

pip install --find-links /mnt/packages/ torch numpy transformers

pip doesn’t know these files are in Azure. It does a readdir() on /mnt/packages/, gets a list of .whl files, picks the right ones, opens and reads them. Standard filesystem operations. BlobFuse handles the translation invisibly.

Why this is useful:

• No custom download logic. No az storage blob download scripts, no pre-stage steps. Just mount and go.
• Cache across runs. File cache means the first CI run downloads from blob, but subsequent runs on the same node hit local cache. Docker builds with --mount=type=bind to the blobfuse mount get the same benefit.
• Offline-ish installs. Once cached, you’re not hitting the network. Useful when you want repeatable, fast installs without depending on PyPI availability.

In containers (Docker, Kubernetes pods), you need two things:

docker run --cap-add=SYS_ADMIN --device=/dev/fuse ...

SYS_ADMIN capability lets the container create FUSE mounts. /dev/fuse is the device node that the FUSE kernel module uses to communicate with userspace. Without both, the mount call fails.

Trade-offs

BlobFuse is a translation layer, not magic. The translation is lossy.

Not fully POSIX compliant. Blob storage is a flat key-value store with / as a convention. rename() isn’t atomic — under the hood it’s a copy + delete. If your program depends on atomic renames (some package managers do for safety), things get weird.

First access is network-bound. The cache is empty on first mount. That initial pip install is as slow as downloading from blob storage, because that’s literally what’s happening. The win is on the second, third, hundredth run.

Concurrent writes from multiple clients = dragons. Two BlobFuse mounts on different machines writing to the same blob? No distributed locking, no conflict resolution. Blob storage has last-writer-wins semantics. For the read-heavy package install use case this doesn’t matter — but don’t try to use it as a shared writable filesystem.

SYS_ADMIN in containers. Granting SYS_ADMIN is a meaningful security surface expansion. It’s not root, but it’s close. In production Kubernetes, you’d typically use a CSI driver (like the Azure Blob CSI driver) instead, which runs the FUSE mount in a privileged sidecar so your application container stays unprivileged.

The adapter plug

BlobFuse is a 转接头 — a universal adapter plug. Your app speaks “filesystem” (open, read, stat). Azure Blob Storage speaks “REST API” (GET, PUT, LIST). BlobFuse sits between them and translates, and neither side needs to know about the other.

The reason this is even possible — the reason you can jam a cloud object store into the POSIX filesystem interface — is that Linux was designed with VFS as an abstraction layer from the start. Apps don’t talk to storage. They talk to VFS. And FUSE lets anyone write a new backend for VFS, in userspace, without touching the kernel.

That’s the whole trick. Everything else is plumbing.