Key Takeaways
Most AI pipelines maintain two copies of the same data: one in a file system, one in object storage, purely to bridge incompatible interfaces.
The sync job between them isn’t a pipeline step; it’s a symptom of architectural failure that worsens as datasets grow (Lumarian’s sync went from minutes to 30 hours at 40 TB).
AI agents work natively in files and directories, flat object stores force them to reconstruct structure, inflating latency and token cost.
Convergence stores data once and exposes it through both file and object interfaces, so a record written via the file API is instantly readable via the object API, no copy, no staging.
The Great Divide: How File vs. Object Storage Creates An AI Bottleneck
Let me tell you about Lumarian, a fictional mid-sized financial services company running fraud detection. Every week, their model retrains on data generated at the edge, coming from an NFS share. It’s structured, permissioned, incrementally updated by a dozen upstream services. But their GPU cluster? It reads from Amazon S3.
So, every Sunday night, an automated job kicks off. It copies 4TB from NFS to S3. When it finishes, training begins. By Wednesday, they ship a new model.
This worked great. Until the dataset hit 40TB.
Now, the sync takes 30 hours. Training bleeds into the next cycle. Someone suggests adding incremental sync logic. Someone else builds a change detection daemon. Three engineers now maintain infrastructure that exists for one reason: to move data from one storage system to another.
Lumarian isn’t unusual. They’re every company trying to do AI at scale.
Here’s what’s actually happening: Most organizations are maintaining two copies of the same dataset. One in a file system for the humans and applications that need directories, permissions, and the ability to actually edit things. Another in object storage so distributed compute can process it efficiently at scale.
Nobody designed it this way. It just… happened. Why? Because file and object storage are fundamentally incompatible paradigms that evolved for different worlds, and modern AI workloads need both simultaneously.
At small scale, this duplication is annoying. At AI scale, it’s insane. Storage footprints double. Pipelines become Rube Goldberg machines. GPU clusters worth thousands per hour sit idle waiting for data to be copied. And governance? Good luck keeping permissions synchronized across two completely different systems.
As the CTO for CTERA, I’ve spent two years diving deep into this problem and realized that the solution isn’t a better sync job. It’s rethinking the entire architecture.
File Storage vs Object Storage: Two Worlds That Can't Talk To Each Other
To understand why unifying file and object storage is so hard, you need to understand how differently these systems think about data.
Object storage is brilliantly simple. Take S3: everything is a key-value pair in a flat namespace. That “folder” you see? It’s just a naming convention using slashes. There’s no actual directory structure to maintain, which means no central metadata bottleneck when you write millions of objects.
This simplicity enables everything else. Objects can be scattered across availability zones without caring about their “neighbors.” The system can serve any object from any node because there’s no session state to track. Want to read 10,000 objects in parallel? Go for it. Each request is independent.
AWS says S3 is designed to exceed 99.999999999% durability. That’s 11 nines. You can only promise that when objects are truly independent with no cross-object state to keep consistent when a data center goes dark.
But here’s what object storage can’t do: You can’t atomically rename anything (it’s copy-then-delete). You can’t edit a single byte without rewriting the entire object. There’s no file locking. No real directories. No append operations. These aren’t bugs. They’re the price of that beautiful scalability.
File storage is the opposite bet. It assumes you want structure, relationships, and mutability. Linux implements this through the Virtual File System (VFS) layer with three key abstractions: dentries (directory entries – the name-to-file mappings cached in memory), inodes (the actual file metadata), and file descriptors (open file state).
This is why navigating a project directory is instant; the entire active tree is probably in the dentry cache. It’s why you can atomically rename files, which is critical for databases and package managers. It’s why multiple processes can safely edit the same dataset with proper locking.
NFS extends this over the network, adding client-side caching to make remote files feel slightly more local. The complexity is hidden, but it’s there: cache coherence, lease management, storage quotas, consistency guarantees.
The weakness? That beautiful directory tree becomes a bottleneck at scale. Every file operation potentially touches shared mutable state. Scaling horizontally means distributed locks, metadata replication, or accepting coordination latency.
Why AI Workloads Break Traditional Storage
Back to Lumarian. Their problem isn’t technical incompetence. It’s that their fraud detection pipeline straddles two incompatible worlds.
Upstream, they need file semantics: services incrementally updating records, atomic renames for consistency, permissions that map to their organizational structure. Downstream, they need object semantics: hundreds of GPUs reading training data in parallel at 100 Gbps.
No storage system that natively does both. So, they built a bridge. That bridge is now their biggest operational burden.
The industry has tried everything to fix this, but every attempt results in a compromise:
File gateways that speak NFS but store in S3? They work, but implementing POSIX semantics over REST APIs is like playing Beethoven on a kazoo. Atomic rename becomes a multi-step dance. Directory listings require prefix scans. Performance suffers.
Hierarchical namespaces bolted onto object stores? Azure Blob Storage has this as an option. It helps, but you’re still missing real file semantics. No true locking and no atomic operations, just a better illusion.
FUSE mounts that let you mount S3 as a file system? Similar to file gateways, but every I/O operation now has to traverse userspace, kernel space, and then an HTTP API. Under load, that overhead is brutal.
File-on-Object systems that use object storage for file chunks? These can perform very well because they treat objects as a block storage layer rather than mapping objects to files directly. However, the tradeoff can be significant: objects are not stored in a native object format, so you cannot read or write them directly through object APIs. All access must go through the file interface, which means you lose the extreme parallelism that makes object storage valuable for AI workloads.
Coding Agents Love Files
Just when we thought the problem couldn’t get more interesting, AI agents showed up with a fascinating requirement: they’re eerily good at working with file systems.
Think about what a coding agent actually does: it navigates directories, reads files, makes edits, runs tests, and commits changes. These map perfectly to POSIX primitives. Give agents good file system primitives, and they thrive. Make them work with flat object stores, and they struggle.
When Lumarian deployed an AI coding assistant for their feature engineering library on NFS, it worked beautifully. The agent navigated the repo, proposed edits, and ran tests, exactly like a junior developer.
When they tried the same with their S3 training data, it was painful. The agent had to reconstruct structure from key prefixes, manage state externally, issue ListObjects calls to approximate ls, and spawn many sub-agents to combat latency. AI token cost skyrocketed. The file system interface was built for what agents do. Object storage wasn’t.
This isn’t a coincidence. AI agents are stateful. They build context over time, persist intermediate results, and coordinate with other agents. A file system naturally provides all of this: directories organize work, paths encode relationships, and the namespace itself becomes shared memory, an ideal surface for collaboration.
Object storage is flat. An agent has to reconstruct every relationship, manage its own state, approximate every file system operation. It’s possible, but why make it hard?
For AI Agents, Files Are All You Need
LlamaIndex coined a deliberately provocative phrase: “Files are all you need.” They’re describing something important. File systems are becoming the primary interface for agent context, memory, skills, and tool use. Git is an interesting option: It’s a sort-of file system optimized for history, collaboration, branching, and auditability. Exactly what agents need. Each commit is an immutable snapshot. Branches are reasoning paths (i.e., an agent can explore multiple approaches in parallel). The collaboration model already works; agents create pull requests and humans review them.
But Git wasn’t built for massive binaries, semantic relationships between documents, real-time coordination, or the search and retrieval agents need. That’s why the future isn’t “Git or filesystem.” Instead, it’s a combination of object storage for durability, file system semantics for interaction, and Git-like versioning for traceability and rollbacks.
The Converged Architecture: One Dataset, Two Views
Here’s the breakthrough: stop trying to bridge two systems. Build one unified file and object storage system that natively speaks both languages. This is what we have built with CTERA Fusion Direct.
In a converged architecture, file and object interfaces are just two views of the same data. No copying. No translation layer in the critical path. Data is stored once in a format optimized for object-scale durability and throughput but exposed through genuine file system semantics where needed.
For Lumarian, convergence changes everything. The Sunday sync job disappears. Their NFS share and S3 bucket become two views of the same dataset. When a service writes a record through the file interface, the GPU cluster reads it through the object interface immediately. No sync. No staging. No 30-hour wait.
Those three engineers maintaining the sync daemon? They’re working on the models now.
This isn’t easy to build. But it’s the only approach that doesn’t just move the complexity around. It eliminates it.
What Unified Storage Means For Your AI Strategy
If your data pipeline has a step whose only job is copying between storage systems, that’s not a pipeline step. It’s a symptom of architectural failure. At AI scale, that symptom will eat you alive.
If you’re evaluating storage, stop asking, “file or object?” Instead, start asking: “Can this system serve both interfaces natively over the same data?” The answer determines whether you’ll be maintaining one system or two.
If you’re deploying AI agents, design their workspaces over file system semantics, even if the underlying storage is object-native. Agents think in files and directories. Don’t fight it.
The Future: Unifying Files And Objects
For decades, file and object storage lived in different worlds. Files for human-scale work. Objects for cloud-scale storage. That separation made sense when those worlds didn’t overlap.
AI shattered that boundary.
Training pipelines need object throughput. AI agents and enterprise applications need file system semantics. The same data must serve both, at the same time, without compromise.
The companies that figure this out will have a structural advantage: their storage amplifies their AI instead of constraining it. Their pipelines will be simpler because data doesn’t move unnecessarily. Their agents will be effective because they have the interfaces they need.
Everyone else will still be running sync jobs on Sunday night, wondering why their GPU utilization is so low.
The question isn’t whether to converge file and object storage. It’s whether you’ll do it before or after it becomes a crisis.
Lumarian learned the hard way. You don’t have to.
Share Post
Frequently Asked Questions
- What is the difference between file storage and object storage?
File storage organizes data in a hierarchical directory tree with POSIX features (e.g., atomic rename, byte-level edits, and file locking), ideal for applications and humans. Object storage uses a flat key-value namespace with no real directories or locking, trading those features for near-infinite parallel scalability and extreme durability. Both serve different needs, which is why AI workloads often require both.
- Why can't I just use object storage for AI training and file storage for everything else?
You can, but it forces you to maintain two copies of the same dataset and a sync job to keep them aligned. As data grows, that sync becomes a bottleneck. In one example, it stretched to 30 hours, idling expensive GPUs and consuming engineering time that delivers no business value.
- Why do AI agents work better with file systems than object storage?
AI agents are stateful: they navigate directories, read and edit files, persist intermediate results, and coordinate through a shared namespace. These actions map directly to POSIX primitives. On flat object stores, agents must reconstruct structure from key prefixes and manage state externally, which raises latency and token cost.
- What is unified file and object storage?
Converged storage stores a dataset once and exposes it simultaneously through both file and object interfaces. There’s no copying or translation layer in the critical path; a record written through the file interface is immediately readable through the object interface, and vice versa.
- Does unifying file and object storage require copying data between systems?
No. That’s the point of convergence. Instead of two systems and a sync job, file and object are two views of the same underlying data, which eliminates the duplication, the storage overhead, and the staging delay entirely.
