Groq’s ultrafast LPU accelerator smashes AI LLM benchmarks
When it comes to AI in some cases fast is best and a new rival GPU manufacturer has just one upped Nvidia.
Key takeaways
- Groq, led by ex-Google engineer and CEO Jonathan Ross, claims to have created the first ever Language Processing Unit (LPU) which it says can deliver the fastest speeds for AI applications.
- Ross, who previously designed Google's Tensor Processing Unit (TPU), launched Groq in 2016 to create a chip capable of executing deep learning inference tasks more efficiently than existing CPUs and GPUs.
- Matt Shumer posted the test on X and says more than 3/4 of the time was spent searching not generating.
Cite or link to this article
Griffin, M. (2024) 'Groq’s ultrafast LPU accelerator smashes AI LLM benchmarks', 311 Institute, 20 August. Available at: https://www.311institute.com/ai-accelerator-startup-groq-smashes-gpu-ai-benchmarks-with-new-chip/ (Accessed: 25 October 2026).
Groq, led by ex-Google engineer and CEO Jonathan Ross, claims to have created the first ever Language Processing Unit (LPU) which it says can deliver the fastest speeds for AI applications. It’s a bold claim, but one that the latest demos more than back up, suggesting it could well become an absolute game-changer for AI.
Ross, who previously designed Google's Tensor Processing Unit (TPU), launched Groq in 2016 to create a chip capable of executing deep learning inference tasks more efficiently than existing CPUs and GPUs.
The company's Tensor Stream Processor (TSP) is likened to an assembly line, processing data tasks in a sequential, organized manner. In contrast, a GPU is akin to a static workstation, where workers come and go to apply processing steps. The TSP's efficiency became evident with the rise of Generative AI, leading Ross to rebrand the TSP as the Language Processing Unit (LPU) to increase its recognizability.
See it in action.
Unlike GPUs, LPUs utilize a streamlined approach, eliminating the need for complex scheduling hardware, ensuring consistent latency and throughput. LPUs are also energy efficient, reducing the overhead of managing multiple threads and avoiding underutilization of cores. Groq's scalable chip design allows multiple TSPs to be linked without traditional bottlenecks, simplifying hardware requirements for large-scale AI models.
The first public demo of Groq was a lightning-fast AI answers engine that generated answers with hundreds of words in less that a second. Matt Shumer posted the test on X and says more than 3/4 of the time was spent searching not generating.
If you want to try Groq for yourself, to get an idea of just how fast it can be for AI, go to this chat page. Use the drop down on the left to switch between the different available models.
What this means for your organisation
Delay may not sit in the chip.
In the first public demo, Matt Shumer said more than 3/4 of the time went on searching, not generating. A fast chip only helps if the rest of the chain keeps up. Look at the whole service, not the model alone.
Steady response times aid planning.
The source says LPUs drop complex scheduling hardware, which gives consistent latency and throughput. They also cut the overhead of managing multiple threads. Predictable speed makes it easier to promise service levels to customers and staff.
Treat LPUs as one to explore.
311 data lists Language Processing Units as Explore, at readiness level 8, with wide use by 2032. Graphic Processor Units stay at Act now. Run small tests on inference work and keep current GPU plans in place.
Three questions for your next leadership meeting
- Where does delay sit in our AI services today, the chip or the search?
- Which of our workloads would gain most from fast, steady responses?
- What must we test before moving any inference work off GPUs?
The trends and technologies behind this story
From the 311 Starburst, the 311 Strategic Intelligence Unit’s map of the technologies and trends shaping the next 50 years.
Analysis by the 311 Strategic Intelligence Unit, from Matthew Griffin’s article and 311 data. First published on Fanatical Futurist.
FAQ
Why does this matter?
When it comes to AI in some cases fast is best and a new rival GPU manufacturer has just one upped Nvidia.
About the author
Matthew Griffin Founder, President and Futurist, 311 Institute
Matthew Griffin is a multi-award winning Futurist and expert in Disruption and Innovation, Geopolitics, Leadership, and Technology, who NASA have described as a "walking encyclopaedia of the future" and a "futurist Polymath."
Read full bio
Matthew Griffin is a multi-award winning Futurist and expert in Disruption and Innovation, Geopolitics, Leadership, and Technology, who NASA have described as a "walking encyclopaedia of the future" and a "futurist Polymath." 17-time best selling author of the "Codex of the Future" series, Matthew is the Founder, President and Futurist of the 311 Institute, a global Futures and Deep Futures advisory firm working with royal households, world leaders, G7, G20, and G77 governments, NGOs, and multi-national mid and mega cap firms to help them explore, shape, and lead the next 50 years of business and society.
An award-winning YouTube creator with over a million followers, with an unrivalled global reach and impact, Matthew is a highly sought-after international keynote speaker, lecturer, and mentor who collaborates with global leaders through the United Nations Alliance of Civilizations (UNAOC) and United Nations General Assembly (UNGA) to shape pivotal initiatives such as the UN’s AI for Humanity program, the United Nations Conference of the Parties (UN COP), and the World Economic Forum in Davos.
As the former Global Head of Cloud, National Security, and Enterprise Sales for companies including Atos, Dell-EMC, and IBM, Matthew has a proven track record of building multi-billion dollar business units and turning failing divisions into market leaders. His ability to identify, analyse, and communicate the implications of hundreds of emerging technologies and trends is unparalleled, and his insights are trusted by many of the world’s most respected organisations, including ABB, Accenture, Adidas, AON, ARM, BCG, Centrica, Citi, Coca-Cola, Dentons, Deloitte, Dow Jones, EY, Google, KPMG, Lego, Legal & General, LinkedIn, Microsoft, PepsiCo, Qualcomm, RWE, Samsung, Siemens AG and Siemens Energy, T-Mobile, UBS, VISA, Walmart, Workday, Worldpay and many others.
Regularly featured in the global media including the AP, BBC, Bloomberg, CNBC, Discovery, Forbes, Khaleej Times, Telegraph, TIME, ViacomCBS, WIRED, and the WSJ, Matthews mission is to help organisations create a fair and sustainable future whose benefits are shared by everyone irrespective of their ability, background, or circumstances.
What future do you need to see?
Choose one to get started on computing and the future of your organisation.
Sources and further reading
- page groq.com