Google opens TorchTPU: PyTorch can now natively leverage Google's AI accelerators

Google open-sources TorchTPU to run PyTorch on its TPUs, with support for vLLM, SGLang, Hugging Face, and TorchTitan.

AI developers now have another way to use Google's accelerators without rewriting their models for a specialized software environment. At PyTorch Conference 2026, Google Cloud open-sourced TorchTPU, an infrastructure stack that enables native PyTorch execution on its Tensor Processing Units (TPUs). Developed with Meta's PyTorch team, it connects familiar framework APIs with compilation, distributed training, and inference tools. The October 20 announcement also includes integrations with Hugging Face, vLLM, SGLang, Ray, and TorchTitan. Beyond compatibility, Google is seeking to make its accelerators more accessible to developers already working in PyTorch, allowing them to choose among CPUs, GPUs, and TPUs according to their workloads. Reducing Dependence on a Single Hardware Architecture

PyTorch has become one of the primary development environments for machine learning. Researchers and engineers use its Python interface to define models, organize training, and control computation. But running the same software efficiently on different types of accelerators still requires substantial engineering, particularly when compilation tools and specialized libraries differ.

Google's TPUs have supported AI training and inference for years through ecosystems including JAX, TensorFlow, and PyTorch/XLA. TorchTPU is therefore not the first way to execute PyTorch on TPUs. The new development is native integration: it introduces a PyTorch backend designed to work through familiar APIs with fewer hardware-specific adaptations.

Developers can use eager mode for interactive execution and debugging, then switch to `torch.compile` to optimize computations for production. Distributed primitives also make it possible to spread workloads across multiple accelerators. The objective is to bring TPU development closer to the experience already familiar to engineers using CPUs and GPUs.

Meta participated in developing the backend. For companies operating large collections of AI models, the broader goal is to avoid tying every workload to a single processor vendor or software stack. Portability still depends on supported features and model architectures, but the approach reduces the need to maintain separate implementations of the same project. Hugging Face Brings Native TPU Support to Transformers and Diffusers

The collaboration with Hugging Face is one of the most practical parts of the release. In Transformers v5, compatible models can be assigned to Google's accelerators simply by passing `devicemap="tpu"`. This preserves a familiar developer interface without requiring every project to be manually rewritten for TPU execution.

The Diffusers library also supports native TPU execution. It can use eager mode, PyTorch compilation, and tensor parallelism to distribute generative models across multiple accelerators. This is particularly relevant to diffusion-based workflows used for image generation and other multimodal applications.

Google has established automated nightly tests to check Transformers compatibility on TPU infrastructure. That ongoing testing matters in an ecosystem where model architectures and software libraries change frequently. It does not, however, mean that every checkpoint hosted on Hugging Face will immediately run without additional work. Native PyTorch Backends for vLLM and SGLang

Inference, the process of running trained models to produce outputs, is another major focus. Google worked with Inferact on vLLM-TorchTPU, a new backend that brings the vLLM inference engine to Google's accelerators. The integration retains much of vLLM's existing architecture, including schedulers, API servers, and request-management mechanisms.

The backend supports continuous batching, quantization, and multi-chip parallelism. Specialized Pallas kernels optimize selected operations on TPU hardware. For teams already running vLLM, the main benefit is the ability to preserve model definitions and much of the software stack while changing the underlying compute infrastructure.

Google followed a similar approach with RadixArk to develop SGLang-TorchTPU. SGLang is particularly useful for structured and agentic workloads where multiple requests may share extensive context. Its RadixAttention mechanism reuses cached information so repeated prompt prefixes do not have to be processed from scratch.

Previous TPU implementations of these serving engines relied on JAX-based backends. The new versions run on native PyTorch infrastructure, which Google says simplifies debugging and customization. Their release does not establish immediate feature parity or identical performance with every GPU deployment. llm-d Coordinates Requests Across GPU and TPU Fleets

Running a model on an accelerator is only part of operating a large-scale service. Systems must also determine where to send requests, preserve context already stored in memory, and distribute work across machines.

Alongside TorchTPU, Google is releasing llm-d 1.0, a distributed inference project developed with Red Hat, IBM, and a community of more than 500 contributors across 25 organizations. Built around Kubernetes, llm-d connects inference engines such as vLLM and SGLang to routing infrastructure that considers server load and cached context.

The system also separates prefill, which processes input tokens, from decode, which generates the output. These stages have different computational requirements and can benefit from being allocated to different resources. Its router can direct a request to a server that already holds the relevant conversation context, reducing redundant computation.

On Google Kubernetes Engine, llm-d enables services running on GPUs or TPUs to be accessed through a common endpoint. Google cites Zenteiq, which reports increasing aggregate synthetic-data generation from 1.47 billion to 5.97 billion tokens per day per 1,000 TPU v6e accelerators, an improvement of more than fourfold. That figure describes a particular production environment and should not be