Google Revisits Five Generations of TPU: Why AI ASICs Still Have Room to Run

Google’s TPU paper is really a system-design retrospective

Google and UC Berkeley researchers Norman Jouppi, Sridhar Lakshmanamurthy, Cliff Young and David Patterson have published a retrospective covering five generations of Google training supercomputers—from TPU v2 to TPU7x, known as Ironwood.

The headline is impressive: over eight years, the authors report roughly 10× growth in per-chip HBM capacity and bandwidth, about 100× growth in peak per-chip performance when normalised to the paper's low-precision comparison, and approximately 3,600× growth in total pod performance. Ironwood scales to 9,216 chips, with 192 GiB of HBM and about 7.38 TB/s of HBM bandwidth per chip.

But this is not simply a story about a faster ASIC. The paper is an argument that an AI accelerator becomes durable when the chip architecture, memory, interconnect, compiler, cooling, power delivery and failure recovery are designed as one computer.

The comparison needs care. FP8 and BF16 are different numerical formats, so the paper's approximately 100× node-performance headline should not be read as a universal application speed-up. On a like-for-like BF16 basis, the stated peak increase is closer to 50×. Real training throughput still depends on model shape, utilisation, communication and software.

Why the original ASIC did not become obsolete

The classic objection to an ASIC is rigidity. A custom chip takes years to design; AI models can change in months. When TPU v2 was conceived, Transformers had not yet become the dominant workload. By the paper's 2026 internal mix, Transformer variants account for 74% of Google training use, while RNNs have almost disappeared.

Yet the TPU v2 block diagram remains recognisable in Ironwood. The durable parts were not a hard-coded implementation of one neural network. They were general primitives that continued to match how machine learning computes:

1. Systolic arrays for dense matrix multiplication. 2. Narrow floating-point formats that preserve range, including BF16 and FP8. 3. HBM for high-capacity, high-bandwidth model data. 4. A custom inter-chip interconnect for collective communication. 5. Compiler-managed DMA and scratchpad SRAM instead of a conventional CPU-like cache hierarchy. 6. Vector units for activation, normalisation, quantisation and other non-matrix operations.

This is the strongest evidence behind the claim that AI ASICs have room to grow. A successful ASIC does not need to predict every future model. It needs to accelerate computational structures that remain stable underneath model innovation, while leaving programmability in the compiler, vector path, numerical formats and interconnect.

The architecture grew by scaling components, not repeatedly starting over

TPU v2 introduced two large TensorCores. Each used scalar control, a wide VLIW instruction, vector lanes, compiler-managed vector memory and a 128×128 matrix multiplication unit. Ironwood still uses two TensorCores, but the resources inside them are wider and more numerous.

The matrix path grew from two 128×128 BF16 systolic arrays in TPU v2 to four 256×256 BF16 arrays in Ironwood. FP8 operation can use four 512×512 arrays. The vector register structure expanded, and the VLIW instruction became more than 50% wider to control the additional parallel hardware.

From TPU v4 onward, the XLA compiler can combine the two physical TensorCores into a logical “Megacore”. This is a useful example of hardware/software co-design: the silicon retains a practical two-core organisation, while the compiler can expose a larger effective core to the model.

Architectural stability also protects software investment. XLA remained the core compiler layer as the preferred programming environment evolved from TensorFlow toward JAX and lower-level Pallas kernels. Models and compiler optimisations do not have to restart from zero with every chip generation.

HBM is not an accessory; it is part of the compute architecture

TPU v1 was constrained by conventional DRAM bandwidth, so TPU v2 adopted HBM. Ironwood now pairs each chip with eight HBM3E stacks for 192 GiB of capacity and approximately 7.38 TB/s bandwidth.

This explains why AI-accelerator demand propagates far beyond the compute die. The complete package depends on HBM wafers, advanced logic, interposers or other high-density integration, substrates, assembly, test and thermal materials. More compute is valuable only if memory and interconnect can keep it occupied.

For buyers, “AI chip demand” therefore includes power stages, controllers, magnetics, capacitors, optical components, connectors, clocking and monitoring devices. These surrounding components may be orderable in the merchant market even when the TPU ASIC itself is not.

Optical circuit switching turned failure into a topology problem

As pods grew, requiring every node and host to remain healthy became impractical. TPU v4 introduced optical circuit switches (OCS) beneath the inter-chip network. The electrical network inside a local 4×4×4 cube remains fixed, while optical links can connect cubes into a larger 3D torus.

This produces three benefits:

• Failed resources can be bypassed and healthy cubes reconnected.
• A scheduler can assemble a requested slice from available cubes instead of

finding one perfectly contiguous block.

• New racks can be installed, tested and placed into service incrementally.

The lesson is broader than Google's specific topology. At thousands of nodes, availability is an architectural property. A theoretically faster chip can deliver less useful training if the system frequently stops, waits for a large contiguous allocation or loses time recovering from failures.

Google reports over 90% goodput for very large multi-pod training. Goodput is not the same as peak FLOPS: it is the useful training progress retained after communication, faults, recovery and other overheads.

Ironwood adds hardware checks for silent errors

Large, long-running training jobs are vulnerable to silent data corruption. A fault that crashes a job is visible; an intermittent arithmetic error may instead degrade convergence or model quality.

Ironwood addresses this with functional built-in self-test in the matrix engines and hardware replay in the vector path. The replay mechanism samples operations and uses otherwise idle instruction slots to repeat work without changing architectural state. A mismatch can identify a defective unit, which the optical network can then remove from production.

This is an important ASIC opportunity. Reliability features can be designed around the actual workload and datapath. At the same time, it raises validation requirements: production test, ageing screens, telemetry and fleet-level fault analysis become part of accelerator architecture rather than factory-only activities.

Power efficiency is becoming a capacity metric

The paper reports nearly 30× improvement in pod peak performance per TDP watt from TPU v2 to Ironwood. This matters because grid connection and facility power increasingly limit AI data-centre expansion. A more efficient accelerator does not only reduce the electricity bill; it can enable more training throughput inside a fixed megawatt envelope.

The authors also propose compute carbon intensity (CCI), measured as grams of CO₂-equivalent per floating-point operation. Unlike performance per watt, CCI can include both operational electricity emissions and embodied emissions from manufacturing.

One unit conversion in circulated summaries requires correction. Using the paper's example values, `3.14 × 10²³ FLOPs × 265 × 10⁻¹⁸ gCO₂e/FLOP` gives approximately `8.3 × 10⁷ grams`, or about 83 metric tonnes of CO₂e—not 83 million tonnes. The estimate itself remains only a rough multiplication of workload FLOPs and an average CCI; it should not be treated as a complete audit of a specific model run.

Does TPU prove that ASICs will replace GPUs?

No. TPU proves that ASICs can remain programmable and relevant when their specialisation is chosen at the right level and supported by a mature software stack. It does not prove that every company should build a custom accelerator.

Google benefits from workload visibility, enormous deployment volume, control of XLA/JAX, data-centre ownership and the ability to co-design models and hardware. A smaller company may gain more by buying GPUs or a merchant accelerator than by absorbing years of silicon and compiler development.

The likely future is heterogeneous: GPUs for broad programmability, custom ASICs for stable high-scale workloads, CPUs for orchestration, and FPGAs or network ASICs around the data path.

What this means for component sourcing

TPUs are Google-controlled infrastructure, not ordinary catalogue parts. A buyer should not expect to source an Ironwood chip independently and build a compatible board. The commercial opportunity lies mainly in the surrounding AI-infrastructure BOM and in other merchant devices using similar principles.

RFQ checklist for AI infrastructure components

1. Identify the exact platform, board or rack qualification—not only “AI server use”. 2. Provide the complete manufacturer part number, package, revision and approved-vendor restrictions. 3. Confirm voltage, current, telemetry, transient and thermal requirements for every power component. 4. For HBM-adjacent packaging and substrates, verify generation, assembly flow, traceability and qualification ownership. 5. For optics, state data rate, reach, connector, wavelength, temperature and firmware/management requirements. 6. Freeze cooling interfaces, coolant specification, materials compatibility and leak-detection expectations. 7. Confirm lifecycle, PCN/EOL process, lot consistency and failure-analysis support before volume release. 8. Treat nearby accelerator, memory or network parts as engineering changes; similar headline bandwidth is not proof of compatibility.

Current stock, allocation, date code, lot condition and delivery must be confirmed before purchase. In AI infrastructure, the exact qualified component and platform revision matter more than a generic “data-centre grade” label.

The larger lesson: ASICs win when the system is the product

The TPU retrospective makes a strong case for AI ASICs, but not for isolated custom silicon. Google's advantage came from keeping a stable set of useful compute primitives while scaling HBM, numerical formats, interconnect, compiler capability, cooling, power efficiency and resilience around them.

That is why TPU v2 survived the transition from pre-Transformer workloads to a Transformer-dominated 2026 mix. The ASIC was specialised enough to be efficient and general enough at the matrix/vector level to evolve.

For the semiconductor industry, the opportunity is substantial: custom AI compute will expand alongside GPUs, and every accelerator pod pulls a much larger ecosystem of memory, packaging, power, optics, networking and thermal components with it. The durable competitive unit is no longer a chip alone—it is the complete, programmable and serviceable supercomputer.

Primary references

• Jouppi et al., “Google’s Training Supercomputers from TPU v2 to Ironwood”
• Google Cloud TPU7x (Ironwood) documentation
• Google Cloud: Ironwood TPU