Google Cloud published a benchmark on September 5, 2026, showing why LLM infrastructure decisions cannot be based on model size alone. The study compares Gemma 3 12B and Gemma 3 27B on Google Cloud TPU v6e, with substantially different results for classification and text generation.
One TPU setup, two very different workloads
The tests ran on Google Kubernetes Engine Autopilot with a single-host TPU v6e node pool configured in a 2x2-chip topology. Serving used vLLM through the vllm-project/tpu-inference plugin. Both workloads were measured at 16, 32, 64 and 128 concurrent users.
Classification used about 4,000 input tokens and roughly 10 output tokens for an e-commerce compliance case. Generation used a much shorter prompt of 500 input tokens but produced about 1,000 output tokens for an analytical policy brief about the future of AI in the labor market.
That difference changes what the infrastructure must do. The classification test represents a long input followed by a short decision, while the generation test requires the system to produce a much longer response for each request. Comparing the two therefore reveals how prompt length, output length and concurrency influence capacity.
Google Cloud kept key serving settings consistent, including max-model-len=128000, max-num-batched-tokens=8192 and max-num-seqs=512. The figures describe performance within that defined operating frame; they are not a universal ranking of the two models.
Generation widens the scaling gap
At 128 concurrent users in the generation test, Gemma 3 12B reached a normalized throughput multiplier of 8.19x, using Gemma 3 12B at 16 users as the reference. Gemma 3 27B reached 4.12x under the same comparison. The larger model therefore delivered about half the smaller model’s relative scaling result in this workload.
Classification produced a much narrower difference at the same concurrency. Gemma 3 12B reached 6.37x, while Gemma 3 27B reached 6.04x. The contrast does not show that Gemma 3 12B is always faster. It shows that the workload can alter the capacity picture enough to change the apparent gap between model sizes.
Throughput is only part of the deployment decision
Latency became a clearer constraint as concurrency increased. For Gemma 3 12B classification, normalized end-to-end latency rose from 0.99x at 32 users to 1.79x at 64 users, then reached 2.90x at 128 users. Google Cloud warns that hardware saturation can cause steep latency increases and silent request abandonment.
For teams planning an AI service, the practical compromise is between serving more work and preserving a predictable response time. A higher normalized throughput result at a crowded concurrency level may not translate into a better user experience if latency rises sharply or requests are abandoned.
The benchmark applies specifically to the two Gemma models, the TPU v6e configuration, the vLLM serving setup and the workload shapes tested. Different prompts, models, hardware topologies or traffic patterns would require separate measurements.
Google Cloud’s results make a useful operational point: classification and generation should be evaluated separately when sizing an LLM service. Throughput can indicate how a system scales under load, but the latency curve determines whether that capacity remains usable for people making requests.