AI Storage: Ask What Runs on Your Nodes Before Asking About Throughput

In AI storage conversations, throughput usually gets the first slide. I have learned to ask a different first question: what do I have to install on the nodes?

The answer tells me something a bandwidth figure cannot. It starts to reveal how the platform will be patched, how failures will be investigated, and how many teams will have to coordinate every change to a GPU worker.

NetApp’s Novus announcement at INSIGHT 2026 provides an interesting example. The scale claims deserve examination, but the architectural choice that caught my attention was the use of standard Linux pNFS clients. For a Kubernetes platform team, the software required on every worker can shape the operating model for years.

This article expands on my LinkedIn post. It examines the published architecture and proposes an evaluation approach; it does not present measurements from a Novus deployment I have operated. Sources were checked on October 22, 2026.

Start by Separating the Announcement from the Acceptance Criteria

NetApp announced Novus on September 29, describing an architecture designed for up to 100 TB/s of aggregate throughput and a zettabyte-scale filesystem. The announcement positions it for large AI infrastructure providers and describes an initial combination of Novus Data Director metadata software on qualified Supermicro infrastructure with ONTAP data services on AFF A90 systems. (NetApp announcement )

Those are architecture-level claims. They are not evidence that a particular installation will deliver that throughput to a particular training job.

The same announcement says GPU utilization can fall below 30% when data delivery cannot keep up. I would treat that as NetApp’s description of a potential problem, not a universal baseline for AI infrastructure. Low utilization is a symptom that needs investigation; storage is one possible cause. (NetApp’s description of the utilization problem )

Before discussing a purchase, I would translate the headline into questions about the proposed configuration: how many clients, which network, what dataset, what cache state, and which mix of reads, writes, and metadata operations? I would also ask which capabilities are supported in the release being offered and which remain roadmap commitments.

The useful number is the one that helps us complete our workload within its service objective.

Why the Client Stack Belongs in the Architecture Review

A storage client is part of the compute platform’s dependency graph. When it requires additional software on every worker, that software becomes part of image building, security review, compatibility testing, and incident response.

An out-of-tree filesystem module can introduce another qualification cycle around kernel changes. A privileged installer can complicate a minimal node image. A separate agent can introduce its own configuration, telemetry, and upgrade sequence. The actual cost depends on the implementation and the support model, so I would examine those details rather than assume every parallel filesystem has the same constraints.

NetApp states that Novus supports standard in-kernel NFSv4.2/pNFS Flex Files clients, without a proprietary filesystem client on the GPU servers. That is a meaningful design choice to evaluate. (Novus client model )

My interpretation is that it can remove a layer of vendor-specific integration from the node lifecycle. The platform team may have fewer components to package and fewer moving parts to coordinate when replacing a worker image.

That benefit is more tangible than a promise of operational independence. Supported kernel versions, distribution backports, mount behaviour, and server interoperability still need qualification. A standard protocol reduces one source of coupling; it does not make every client configuration interchangeable.

What pNFS Flex Files Changes in the Data Path

Parallel NFS separates metadata access from data access. A client obtains layout information from a metadata server and uses that information to communicate with the relevant data servers. The metadata service therefore does not have to relay every byte of a file through the same path.

Flex Files is a standardized pNFS layout type, specified in RFC 8435. It describes how clients access storage devices using existing protocols and includes mechanisms such as client-side mirroring. Those are protocol capabilities, not a claim that every product implements or enables every possible configuration. (IETF RFC 8435 )

The illustration above is a simplified view of that separation. It is not an installation diagram, and it does not imply that ordinary pNFS reads transfer data directly into GPU memory.

From an operator’s perspective, the separation creates a useful diagnostic question: is a stalled workload waiting for a metadata operation, for data transfer, or somewhere else in the application pipeline?

It also changes what must be verified on the network. Reaching the initial mount endpoint is not sufficient evidence that all required data-server paths are reachable and correctly authorized. I would include those paths in the connectivity and isolation tests.

Standard NFS Does Not Mean an Empty Kubernetes Node

There are several layers to distinguish before concluding that a standard client makes deployment straightforward.

The Linux kernel documentation identifies an in-tree Flex Files implementation. That is different from asserting that every distribution image includes the required configuration. Kubernetes also documents that NFS volumes may require the mount.nfs helper. (Linux pNFS implementation , Kubernetes Persistent Volumes )

A CSI node plugin may still run as a DaemonSet. Its presence is not equivalent to installing a proprietary filesystem kernel module: CSI coordinates storage operations with Kubernetes, while the filesystem client handles access to the mounted storage. The selected driver can still bring privileges and dependencies that belong in the review. (Kubernetes CSI volume model )

This makes immutable operating systems more interesting candidates, but compatibility must be established for the exact image and integration. I would not infer support for Talos, or any other minimal distribution, solely from the statement that pNFS exists in Linux.

The question becomes precise: can we build, patch, replace, and recover the intended node image through our normal process, with a documented support boundary?

Keep GPUDirect Storage Separate from the Baseline

NetApp also mentions NVIDIA GPUDirect Storage in its Novus client description. That should be evaluated as a distinct I/O configuration. (Novus product overview )

NVIDIA documents a cuFile software path and, for relevant configurations, the nvidia-fs kernel module. Its requirements vary with the transport, filesystem, GPU stack, and software versions. NVIDIA also distinguishes direct operation from compatibility mode. (GPUDirect Storage overview , NVIDIA filesystem driver requirements )

Consequently, “no proprietary Novus filesystem client” should not be rewritten as “no additional software is ever needed for GPU storage access.”

I would establish a standard NFS baseline first. If GPUDirect Storage is part of the proposal, I would then test it separately, record the additional dependencies, and verify the actual path being used. A successful application run is not enough to prove that an accelerated path is active.

That separation lets the team judge whether any performance improvement justifies the additional lifecycle responsibilities.

Metadata Deserves Its Own Capacity Plan

A large sequential-read test answers a narrow question. An AI workload may also discover datasets, open many files, inspect attributes, create temporary objects, and write checkpoints. Those operations do not all place pressure on the same part of a filesystem.

NetApp describes Novus Data Director as an independently scalable metadata tier, separate from the ONTAP data systems. In the INSIGHT presentation, Arindam Banerjee further described the metadata service as separate protocol, state, and catalog planes. This is the vendor’s architectural description, rather than an independently verified scaling result. (NetApp architecture explanation , INSIGHT keynote transcript )

The evaluation question is whether the design isolates the bottlenecks encountered by the intended workload. I would ask which metrics expose pressure in each subsystem, how capacity is added, and what happens while a subsystem is recovering.

Consider a deliberately simple sizing example: writing a 10 TB checkpoint in 60 seconds requires about 167 GB/s of payload throughput before allowing for overhead, contention, or protection mechanisms. That calculation does not describe Novus performance. It describes a workload requirement.

Now add many workers creating files at once and a second tenant reading its training dataset. The checkpoint’s completion time may depend on both data transfer and metadata latency. A good average bandwidth result can conceal a long tail of workers waiting to finish.

I would therefore measure metadata latency distributions, checkpoint completion, and application progress together. The goal is to identify which resource limits useful work, then test whether adding capacity in the corresponding plane actually helps.

A Proof of Concept Built Around the Platform’s Lifecycle

A useful PoC should produce a decision and a repeatable operating procedure. I would agree on the workload, the supported configuration, and the acceptance thresholds before starting. The following tests would form its core.

1. Measure Useful GPU Work

Record training step time, samples or tokens processed per second, time waiting for input data, checkpoint overhead, and total job completion time. Capture GPU utilization alongside CPU, network, client, and server metrics.

Utilization alone can mislead. A device can be busy while a job progresses inefficiently, and a low reading can originate in preprocessing, synchronization, or scheduling. Correlated measurements are what make a storage diagnosis credible.

Use both cold and warm runs, document the cache state, and keep application settings consistent. Include enough data to avoid accidentally benchmarking only a convenient cache.

2. Upgrade a Worker During the Test

Run a controlled node replacement or OS upgrade in the test environment using the platform team’s normal procedure. Record the drain behaviour, remount time, application restart, and any manual intervention required.

Use a workload with an explicit recovery strategy. A distributed training job may need to restart from a checkpoint after losing a worker; seamless continuation should not be assumed.

The result should explain what changes on the node, what must remain compatible, and how long useful work takes to resume. Repeat with the agreed rollback procedure where supported.

3. Exercise Metadata and Checkpoints Together

Use a file-count and size distribution representative of the real dataset. Test directory traversal, concurrent opens, creates, renames, and deletes, including both shared-directory contention and operations spread across directories.

Then overlap those operations with checkpoint writes and sustained reads. Track tail latency as well as throughput. A second tenant is useful here: it helps reveal whether one workload’s checkpoint burst delays another workload’s training.

4. Remove a Data Server Under Load

In an isolated PoC, use a vendor-supported failure procedure while a workload is reading or writing. Separately test a metadata-service failure or restart if that scenario is supported.

Record client stalls, retries, errors, application recovery, and the return to steady-state performance. Validate completed data with checksums and test whether the most recent acknowledged checkpoint can actually be restored.

A mounted filesystem and a running process are not sufficient evidence of successful recovery. The acceptance criterion is continued or correctly recovered application work without undetected corruption.

5. Practise the Support Handoff

Take one incident through the proposed escalation process. Establish who examines the Linux client, who owns the CSI integration, who investigates the network, and who coordinates with the storage and GPU vendors.

Standard Linux tooling can make initial evidence collection more familiar. It does not assign responsibility for a kernel regression or an interoperability problem. I would want a named support path, an agreed diagnostic bundle, and clarity about who owns the incident until resolution.

Turn the Results into Acceptance Criteria

I would summarize the outcome in terms that a platform owner and an application team can both use:

The thresholds should come from the service requirements. A research cluster that tolerates a restart and a shared platform with strict availability objectives will not make the same trade-offs.

Cost should follow the same logic. I would compare the infrastructure and operating effort needed to complete the workload, including protection settings, spare capacity, and support. The largest bandwidth figure is not necessarily the lowest cost per completed job.

The Architecture Is What Remains After the Benchmark

Novus is interesting to me because of the choices behind its announcement: a standard client interface, separation of metadata from data, and a design intended to scale those responsibilities independently.

Those choices deserve a serious evaluation. They do not exempt the product from proving its behaviour under the workload, failure modes, and maintenance procedures that matter to the buyer.

My first storage question therefore remains the same: what runs on the nodes, and who owns its lifecycle?

Throughput tells me how fast a tested configuration moved data. The architecture and support model tell me what the team will have to operate for the next five years.