Joules per Token: Cutting AI Energy Consumption Without New Hardware

Most conversations about AI's energy footprint focus on building greener datacenters or waiting for more efficient chips. Both take years. A third lever is available today: the accelerators already installed are running far below their achievable efficiency.During text generation, LLM inference is memory-bound: the dominant energy cost is moving model weights out of HBM, and that cost is paid once per forward pass regardless of how many requests share it. Energy per token is therefore governed largely by how well the serving engine is configured, not by the hardware itself.This session presents measured results from optimizing production inference on deployed NVIDIA accelerators in Saudi Arabia. We show how throughput tuning translates directly into joules saved per token, where the effect saturates, and how power capping trades a small amount of speed for a disproportionate energy gain. We also address the metric discipline the field currently lacks: energy per token is easy to quote and easy to game.