Two new Google TPU systems for long contexts and confidential inference

Two chips power the TPU 8i Max for long-context workloads. Hardware protections on the TPU 8tx help keep sensitive data isolated during inference.

The latest additions to Google’s eighth generation of AI accelerators build on two chips introduced in April. Designed for low-latency inference, the TPU 8i serves as the basis for the 8i Max. The 8tx uses the same silicon as the TPU 8t, which was originally designed for large-scale training. The two configurations announced by Google Cloud on November 9 bring the generation to four systems. Two chips in one package for long-context workloads

An AI assistant working through a codebase or analyzing hours of video needs to retain some of the work it has already done on that input. As the context grows, so does the amount of data the system has to store and move. Memory bandwidth can then limit how fast the chips can work.

The TPU 8i Max packages two TPU 8i chips together. Compared with the 8i, Google says it doubles compute capacity, on-chip SRAM, and bandwidth to high-bandwidth memory (HBM) within the same rack footprint. Those figures describe the hardware. How much context a system can handle also depends on the model and how it’s configured. Protecting data while it’s in use

When a government agency runs a model on sensitive records, encryption at rest and in transit covers two stages of the process. The data also needs protection while the model is using it to generate a response. According to Google Cloud’s technical announcement, the 8tx pairs the TPU 8t silicon with fifth-generation AMD EPYC processors and uses hardware-level protections to isolate sensitive data and model weights during processing.

The system also uses the same low-latency network as the TPU 8i. Alongside inference, it’s designed for smaller-scale training. A team could fine-tune a model on private data and then run it on the same system without moving that data to a separate cluster. Less waiting for models and agents

Alongside the hardware announcement, Google released TPU Sync, an open-source library for moving data in the key-value cache, or KV cache. The cache stores intermediate results so a model can continue generating a response without repeating earlier calculations. TPU Sync moves cached data between accelerator and server memory. It can also hand off the cache from the machine that processes a long prompt to the one that generates the response.

In internal tests on seventh-generation Ironwood TPUs, Google reported a drop of more than 90% in median KV cache transfer time for Qwen3 235B, from several seconds to about 230 milliseconds. On the same model, time to first token fell by an average of 60% in tests with few simultaneous requests.

On the software side, TorchTPU runs PyTorch code on TPUs and supports selected models from the Hugging Face Hub through Transformers. A separate service, Agent Compute Engine, already powers Gemini agents. Google said on October 22 that a customer preview would follow in the coming months. When an agent is waiting for an external tool or a human response, the service can save the state of its virtual machine, pause it, and resume the session from where it left off.