Reve reveals how a small team built an infrastructure capable of mobilizing 10,000 GPUs

Reve publishes the technical behind-the-scenes of its image models, featuring a 10,000-GPU training infrastructure, multi-region clusters, and automated failure recovery.

Training artificial intelligence models across thousands of GPUs requires far more than access to computing power. Engineers must coordinate machines, continuously feed them data, synchronize distributed training, and prevent hardware failures from wasting hours of computation. Reve, the company developing AI image generation and editing models, has published a detailed account of the infrastructure built to address those challenges.

In a 19-page technical report, the research team describes a platform built on Google Cloud, Kubernetes, and Ray, supported by internal tools for resource management, distributed execution, and automated recovery. The infrastructure has been validated at the 10,000-GPU scale, while remaining manageable for a relatively small engineering team.

The report covers the training infrastructure behind Reve 2.0, Reve 2.1, and their predecessors. It does not disclose the complete architecture or training datasets of the generative models themselves. Instead, it explains how the underlying computing platform was designed and operated. Thousands of H100 and H200 GPUs on Google Cloud

Reve uses Google Kubernetes Engine to manage its computing infrastructure and Ray to coordinate distributed training.

Its primary training hardware consists of A3 Mega and A3 Ultra machines, each equipped with eight NVIDIA GPUs.

A3 Mega nodes contain eight 80 GB H100 GPUs and offer a maximum network bandwidth of 1,800 Gbps. A3 Ultra nodes use eight 141 GB H200 GPUs, with networking rated at up to 3,600 Gbps.

High-speed networking matters because large-scale model training requires processors to exchange gradients and model parameters. Performance depends not only on individual GPU capacity but also on how efficiently those accelerators communicate.

Rather than relying exclusively on spot instances, Reve secures longer-term reservations of computing capacity on Google Cloud. This helps provide predictable access to collocated machines for uninterrupted training runs.

The platform does not mix different GPU types within an individual standard training job. Different accelerator groups can participate in a broader experiment through its cross-region training architecture. A single command to launch distributed training

Reve developed an internal command-line tool called GKR, short for Google Kubernetes Engine Runner, to simplify researchers' interactions with the infrastructure.

Through GKR, researchers can launch experiments, select clusters, request GPUs and memory, inspect logs, manage resources, and access development environments.

A configuration can request, for example, two groups of eight H200 GPUs, allocating 16 accelerators alongside the necessary CPU and memory resources.

GKR translates those specifications into Kubernetes resources, applies the relevant hardware constraints, and prepares the execution environment.

It also performs preflight checks before expensive jobs begin, examining hardware faults, driver errors, memory issues, GPU communication, and access to required resources.

Detecting a faulty node before the training run begins can prevent large amounts of computing time from being wasted.

This abstraction is central to Reve's engineering approach. Researchers can focus on models and experiments without needing to manage every operational detail of the underlying clusters. Why Reve moved some workloads away from Ray

One of the most revealing sections of the report concerns architectural decisions made after the team encountered scalability problems.

Multimodal training requires more than GPUs executing the main optimization loop. Additional services handle data preparation, encoding, inference, and evaluation.

Reve implements these auxiliary services as persistent workers within Ray clusters, with separately allocated computing resources.

However, its initial approach to data loading created significant overhead.

Each training process required several parallel data loaders. Representing all those loaders as Ray actors eventually created tens of thousands of actors, placing excessive pressure on Ray's Global Control Service.

The engineers replaced some of those components with Python multiprocessing backends operating independently of Ray.

They also migrated communication between training processes and auxiliary services to ZeroMQ, replacing earlier Ray RPC-based interactions after observing a substantial performance improvement.

Another optimization involves bounded queues dedicated to individual training ranks. This prevents one particularly busy process from monopolizing the auxiliary services needed by others.

The report illustrates how architectural choices that work well at smaller scales can become bottlenecks when thousands of processors operate simultaneously. Training across multiple cloud regions

GPU availability is not always sufficient in a single data center or cloud region.

Reve therefore developed an approach that allows a training experiment to span multiple Google Cloud regions, despite the higher communication latency and lower bandwidth associated with cross-region networking.

Its implementation is based on a customized form of Distributed Low-Communication training, or DiLoCo.

Instead of continuously synchronizing every processor, each region operates an independent local learner. Each learner maintains its own model replicas and optimizer state, performing multiple training steps locally.

A central synchronization service periodically collects model-weight snapshots from the learners, combines their updates using an outer optimizer, and distributes the revised shared model.

Reve supports two synchronization strategies.

In barrier mode, local learners wait for the merged weights before proceeding. In decoupled mode, they continue training while synchronization takes place, requiring additional rules to reconcile local and global updates.

This design reduces the need for constant cross-region communication but introduces complexity in coordinating model states.

The report describes