Deep prefetching APIs now trigger the hydration of data from HDD lakes into flash caches minutes before the GPU requires the next training batch. This sophisticated orchestration forms the bedrock of Meta’s Project Prometheus, a gigawatt-scale supercomputing initiative centered in New Albany, Ohio, designed to facilitate the training of the Muse family of foundational AI models. As the industry pushes toward the era of one million GPUs, the traditional bottlenecks of storage latency and network congestion have become existential threats to progress. Meta’s response is a complete reimagining of the data center, moving away from fragmented hardware configurations toward a unified, tiered architecture that treats storage, networking, and compute as a single, fluid organism. By eliminating GPU idle time, the architectural framework ensures that every cycle of the massive Blackwell-based clusters is dedicated to active learning rather than waiting for data packets. This transition marks a departure from standard network file systems, favoring a multi-tiered cached object storage system that can handle the exabyte-scale requirements of frontier AI research without staggering under the weight of its own metadata.
Revolutionary Networking and Interconnectivity
The BAG Fabric and Global Scaling
To connect these massive GPU clusters across the New Albany site, Meta implemented a multi-layer hub-and-spoke Backend Aggregation (BAG) network. This Ethernet-based fabric spans distances from 2 km to over 100 km, allowing disparate clusters to function as a single, cohesive unit despite being physically separated by massive campus layouts. Utilizing Arista 7800-series chassis and Broadcom Jericho3 silicon, the network delivers multiple petabits per second of bandwidth, which is a requirement for modern distributed training. This high-throughput environment is essential for synchronizing data across thousands of nodes without the lag that typically plagues large-scale distributed systems. The BAG fabric effectively acts as a nervous system, ensuring that the gradient updates and model weights can move between pods with minimal resistance, thereby maintaining the synchronous nature of the training process across the entire million-GPU fleet.
The engineering of the BAG fabric also addresses the physical limitations of fiber optics and signal degradation over long distances. By employing advanced forward error correction and specialized optical transceivers, the network maintains a consistent latency profile even when data must travel across the full 100 km span of the aggregation layer. This capability is critical because it allows the Prometheus project to scale beyond the power limits of a single building, tapping into multiple utility substations across the Ohio region. The hub-and-spoke topology ensures that even if one segment of the campus experiences a localized failure, the rest of the aggregation fabric remains resilient, rerouting traffic through redundant spines to maintain the integrity of the training job. This level of interconnectivity transforms the site from a collection of data centers into a singular, unified supercomputer of unprecedented proportions.
Strategic Geographical Expansion: Managing Power and Connectivity
While New Albany serves as a primary hub, Meta is also developing a second supercluster named Hyperion in Richmond Parish, Louisiana. This decentralized approach indicates a shift toward a global, interconnected AI research infrastructure that prioritizes access to stable, high-capacity power grids. By distributing the physical load while maintaining high-speed connectivity, Meta can manage the immense power and cooling requirements of million-GPU clusters more effectively than within a single, localized data center. The Hyperion site is designed to mirror the Prometheus architecture, ensuring that researchers can move workloads between regions without needing to refactor their dataloading or checkpointing code. This regional diversification also serves as a hedge against localized grid instability, allowing the training of massive models like Muse to continue even during peak demand periods on the local utility infrastructure.
The strategic choice of these locations involves deep collaboration with energy providers to secure the gigawatt-scale power necessary for these operations. Each site is built with sustainability in mind, utilizing advanced liquid cooling and heat recovery systems that integrate with the surrounding environment. The connectivity between New Albany and Richmond Parish is facilitated by dedicated long-haul fiber routes that extend the principles of the BAG fabric across hundreds of miles. This wide-area networking strategy allows Meta to treat its global footprint as a tiered compute resource, where the largest jobs are pinned to high-density hubs like Prometheus while smaller experimental runs are distributed to outlying clusters. The goal is to create a seamless fabric where geographical distance is no longer a limiting factor for the speed of AI innovation, provided the underlying storage and networking layers are sufficiently optimized to hide the latency.
Transforming the Data Storage Paradigm
Shifting from File Systems to BLOB Storage
A pivotal change in the Prometheus architecture is the transition from traditional Network File Systems (NFS) to a Binary Large Object (BLOB) storage scheme. In previous training runs for models like Llama, Meta relied on hierarchical file systems, which created massive metadata bottlenecks as the number of files reached the billions. By adopting a unified object storage model, Meta eliminates the overhead of complex directory structures and the associated POSIX locking mechanisms that often slow down data access at scale. This allows for a flatter, more scalable schema that enables direct-data streaming, ensuring that the GPUs are constantly fed with training information. The shift represents a fundamental realization that AI training data is inherently read-heavy and does not require the complex write-consistency models provided by traditional filesystems.
To manage this data effectively, Meta engineered a custom four-layer storage hierarchy designed to minimize the distance between the data and the processor. The first two layers utilize local RAM and NVMe flash on individual GPU hosts for immediate, low-latency data ingestion. The third layer acts as a regional flash-backed cache, serving as a high-speed buffer that minimizes data retrieval times from the deeper global repositories. Finally, the fourth layer serves as the “source of truth,” using durable, HDD-backed storage powered by the Tectonic fabric. This tiered approach ensures that data is always positioned as close to the compute resource as possible, using predictive algorithms to move batches up the hierarchy just before they are needed. By decoupling the durable storage from the high-speed access layers, Prometheus can scale capacity and performance independently, meeting the demands of exabyte-scale datasets.
Software Innovations and Direct Access
The software backbone of Prometheus relies on Tectonic, a multi-tenant block storage fabric, and ZippyDB for metadata management. A critical innovation here is the Fat Client SDK, which removes the need for a standard data proxy or intermediary server. Instead of a middleman handling requests, the SDK contacts an API server to receive a ReadPlan, which contains the exact mapping of data blocks to storage nodes. The client then streams data directly from the storage servers to the GPU host over the high-speed backend network. This direct-access model drastically reduces latency and removes potential bottlenecks in the data plane, allowing for much faster training cycles. It essentially turns every GPU host into a sophisticated storage client capable of orchestrating its own data flow based on the specific needs of the current training epoch.
Meta also integrated its Owl system to enable peer-to-peer data distribution among GPU hosts, further reducing the load on the central storage infrastructure. When one host retrieves a chunk of training data, it can share that information with other hosts in the cluster using their spare memory as a distributed cache. This creates an ephemeral distribution tree that replicates hot data across the cluster without straining the central storage servers. This decentralized sharing mechanism ensures that common data points, such as those used in repetitive training loops, are always available at the local level. The combination of Owl and the Fat Client SDK means that the storage fabric can support the simultaneous ingestion demands of one million GPUs without experiencing the “thundering herd” problem that typically crashes traditional storage architectures during massive training starts.
Operational Efficiency and Specialized Hardware
Optimizing Dataloading and Performance
The operational impact of these storage and networking innovations is most visible in the reduction of data ingestion times. By using advanced prefetching strategies where data is moved from long-term storage to local caches minutes before it is needed, Meta has seen ingestion times for large jobs drop from 89 hours to just three hours. This efficiency ensures that the gigawatt-scale power consumption of the facility is translated into actual computation rather than wasted on idle wait times. The system uses a specialized API to pre-warm the regional flash caches, ensuring that by the time the training script reaches the next epoch, the required data is already residing in the lowest-latency tiers. This proactive management of the data lifecycle is what allows Prometheus to maintain such high utilization rates across its massive GPU footprint.
To support this infrastructure, Meta utilizes custom, OCP-aligned hardware like the Yosemite V3.5 chassis and the Grand Canyon storage platform. The Yosemite system provides a modular, multi-node compute environment that can be tailored for either high-density compute or high-capacity storage depending on the specific expansion card used. The Grand Canyon system, for example, is a dense HDD platform capable of holding up to 216 drives per head node, handling the massive bulk storage requirements of the L4 tier with extreme efficiency. By co-designing the hardware and software, Meta maintains total control over the data pipeline, allowing them to optimize every component for the specific demands of AI at the million-GPU scale. This vertical integration means that hardware failures can be identified and bypassed at the software level with minimal impact on the overall training job, a necessity when dealing with hundreds of thousands of individual components.
Fault Tolerance and Checkpointing
In a cluster of one million GPUs, hardware failure is an everyday reality rather than a rare event, requiring a robust strategy for maintaining progress. To combat this, Prometheus employs a distributed checkpointing architecture that asynchronously writes model states to the multi-tier storage without halting the training process. By using delta checkpointing—saving only the changes made since the last save—Meta can protect training progress with minimal bandwidth overhead. This strategy allows the cluster to maintain a goodput rate of over 85%, ensuring that the vast majority of the time spent is productive training rather than recovery. The model states are initially staged in local NVMe flash and then gradually pushed to the durable Tectonic layer, providing multiple points of recovery should a massive network or power failure occur.
The project established a new baseline for how massive-scale AI training should be conducted, moving the industry away from traditional HPC storage models. Engineers solved the latency issue by prioritizing a tiered caching strategy that effectively hid the slower response times of high-capacity HDD storage. Organizations seeking to replicate these gains prioritized the transition toward decentralized object storage while investing in peer-to-peer data replication protocols. The industry moved toward a hardware-software co-design model where storage was no longer a peripheral service but a core component of the compute fabric. Stakeholders recognized that as model sizes continued to swell, the efficiency of data movement became the ultimate differentiator in AI leadership. Future efforts focused on further automating the prefetching logic through predictive analytics, ensuring that the next gigawatt-scale clusters maintained near-perfect utility. By standardizing on OCP-aligned hardware, the broader ecosystem began to adopt these sophisticated checkpointing and dataloading strategies, effectively lowering the barrier for training the next generation of multi-modal agents.
