Meta wants to bring together one million GPUs in Prometheus to train its future AI models.

Meta details Prometheus, its Ohio supercomputer designed to bring together one million GPUs, already being used to train Muse Wonder with over 85% useful compute.

In Ohio, Meta is transforming a former data center site into an infrastructure designed to operate nearly one million graphics processing units as a single system. Named Prometheus, this gigawatt-scale supercomputer has already been used to train the Muse models, including Muse Wonder, and currently brings together several hundred thousand GPUs distributed across different buildings. In a technical publication dated October 19, 2026, the group details less the theoretical power of the facility than the issues encountered in making it operational: hardware failures, inter-processor communications, electrical variations, and cooling. The challenge is to keep training runs active despite the proliferation of components likely to interrupt them. Hundreds of thousands of GPUs distributed across several buildings

Located in New Albany, Prometheus occupies a site that, just two years prior, still hosted conventional infrastructures intended notably for Meta's web services. Its transformation was accompanied by an overhaul of the buildings, IT equipment, and communication networks. The first phase is complete and was used to train the Muse models, while planned expansions are expected to bring the facility to approximately one million GPUs.

At this scale, computing power no longer depends solely on the number of available processors. The machines must exchange large amounts of data during training, sometimes with very strict synchronization constraints. A slowed processor, a failing connection, or an unavailable server can then affect the entire workload. Meta therefore designed Prometheus as a distributed system at the scale of multiple buildings, rather than as a simple addition of data centers.

The infrastructure currently combines two generations of NVIDIA hardware: the GB200 Blackwell and the GB300 Blackwell Ultra. A third, based on Vera Rubin, is to be added later. The first assembly, named Catalina, brings together 72 GB200 GPUs in two interconnected racks; Clemente, based on GB300, gathers the 72 GPUs into a single rack. This evolution increases the available memory from 13.4 to 20.7 terabytes per set of 72 GPUs, while increasing compute density. However, it comes with considerable electrical requirements, with approximately 120 kilowatts for the Catalina system and 135 kilowatts for a Clemente rack. A network that must function as a single machine

Distributed training requires GPUs to communicate rapidly, even when located in different buildings. To achieve this, Meta uses an Ethernet network separate from the one intended for traditional IT services. At the building level, a single network infrastructure can connect up to 20,736 GPUs, while an aggregation network handles exchanges between facilities separated by two to over a hundred kilometers.

These distances complicate the synchronization of calculations. Communications within a group of machines are particularly sensitive to latency, whereas other exchanges can be distributed among more distant sets. Meta explains that it adapted the organization of training runs and communication mechanisms to this hierarchy, with the goal of reducing the periods during which GPUs wait for data instead of computing.

The group also evolved the architecture of its internal networks to limit congestion caused by large simultaneous flows. The facility mobilizes tens of thousands of switches, millions of optical components, and a very large-scale fiber infrastructure. Its complexity explains why a local outage can have consequences extending far beyond the server concerned. Eight spare GPUs per set of 72

One of the most concrete decisions concerns failure management. NVIDIA NVL72 systems allow 72 GPUs to operate within the same fast communication domain, but this grouping also increases the number of components that a single failure can disrupt simultaneously. Compared to older NVL8 configurations, the failure domain is nine times larger.

Meta chose to mobilize only 64 GPUs out of the 72 available for each compute group, retaining the other eight as replacement units. When a processor fails, a spare GPU can take its place, and training restarts from its last checkpoint without waiting for a physical repair. The group also maintains approximately 2% additional racks for incidents that would affect an entire unit.

For a training run using 32,768 active GPUs, this organization thus requires 523 NVL72 racks, or 37,656 installed GPUs. The total capacity exceeds that actually assigned to computation by approximately 13%, primarily due to the choice of using only 64 GPUs on each group of 72. This redundancy represents a hardware cost, but reduces the risk of a failure persistently halting training.

Meta adds two software mechanisms to this organization. When a failure remains limited to a replica of the model, the Parallelism-Aware Fault Tolerance technique allows it to be isolated while the others continue their calculations. If the incident affects a larger part of the system, training restarts from a distributed checkpoint. These saves are performed asynchronously to limit the loss of work without excessively slowing execution. A 99% reduction in interruptions

The early phases of Prometheus demonstrated the difficulties of this architecture. In mid-2025, Meta recorded up to nearly 1,000 daily interruptions for 8,000 GPUs, according to its feedback. After a year of hardware and software corrections, the company states it has reduced this rate by 99%, enough to run multiple training tasks each mobilizing tens of thousands of processors.

The work notably focused on NVLink connections, the reliability of HBM and LPDDR memories, switches, as well as network incidents. Meta and NVIDIA jointly analyze the root causes of interruptions using telemetry data to distinguish transient faults from components that need repair or replacement. Diagnostic procedures are also applied before hardware is used in production. Firmware updates can now be deployed to over 100,000 GPUs in a