Transformer-Squared: Self-Adaptive LLMs
The key idea
Posts by Tag
- LLMs 37
- efficient-inference 17
- efficient-training 11
- training-dynamics 10
- quantisation 10
- transformers 8
- fine-tuning 5
- computer-vision 4
- diffusion 4
- long-context 4
- sparsity 4
- reasoning 4
- GNNs 3
- scaling-laws 3
- inference 3
- mixture-of-experts 3
- mup 3
- unit-scaling 2
- number-formats 2
- materials 2
- position-embeddings 2
- speculative-decoding 2
- retrieval-augmented-generation 2
- sparse-attention 2
- not-transformers 2
- self-correction 2
- image-generation 2
- chip-design 1
- fp8 1
- DFT 1
- active-learning 1
- multi-modality 1
- automated-theorem-proving 1
- synthetic-data 1
- distributed-training 1
- local-updates 1
- optimization 1
- learning-rate-schedules 1
- RNNs 1
- state-space-models 1
- hallucinations 1
- reinforcement-learning 1
- optimisation 1
- hiring 1
- batch-size 1
- byte-level 1
- language-models 1
- generative-models 1
- embedding-models 1
- training 1
- synthetic data 1
- VLMs 1
- reinforcement learning 1
- memory 1
LLMs
Titans: Learning to Memorize at Test Time
The key idea
Evolving Deeper LLM Thinking
The key idea
DeepSeek-V3 & DeepSeek-R1 Technical Reports
With their V3 and R1 models, DeepSeek sets a new state-of-the-art in open-weight models and trades benchmark to benchmark with the best models from Anthropic...
Llama 3.2 Vision — A Deep Dive
Vision-Language Models (VLMs) allow LLMs to “see”, but how do they work? In this post, we’ll walk through the model changes needed to turn an LLM into a VLM ...
Phi-4 Technical Report
The key idea
The Super Weight in Large Language Models
The key idea
How Does Critical Batch Size Scale in Pre-training?
The key idea
Generative Hierarchical Materials Search
The key idea
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability
The key idea
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
The key idea
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
The key idea
Scalable MatMul-free Language Modeling
The key idea
xLSTM: Extended Long Short-Term Memory
The key idea
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
The key idea
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
The key idea
A transformer walk-through, with Gemma
Transformer-based LLMs seem mysterious, but they don’t need to. In this post, we’ll walk through a modern transformer LLM, Google’s Gemma, providing bare-bon...
G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering
The key idea
FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design
The key idea
Massive Activations in Large Language Models
The key idea
Simplifying Transformer Blocks
The key idea
ChipNeMo: Domain-Adapted LLMs for Chip Design
The key idea
efficient-inference
DeepSeek-V3 & DeepSeek-R1 Technical Reports
With their V3 and R1 models, DeepSeek sets a new state-of-the-art in open-weight models and trades benchmark to benchmark with the best models from Anthropic...
Scaling Laws for Precision
The key idea
Scaling FP8 training to trillion-token LLMs
The key idea
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
The key idea
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
The key idea
Scalable MatMul-free Language Modeling
The key idea
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
The key idea
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
The key idea
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
The key idea
efficient-training
Transformer-Squared: Self-Adaptive LLMs
The key idea
Memory Layers at Scale
The key idea
Scaling Laws for Precision
The key idea
How Does Critical Batch Size Scale in Pre-training?
The key idea
Mixture of a Million Experts
The key idea
training-dynamics
Scaling Laws for Precision
The key idea
How Does Critical Batch Size Scale in Pre-training?
The key idea
Scale-preserving nonlinearities for u-μP
My colleagues and I always get excited when, every once in a while, deep learning research throws up a fun little maths problem. Our recent work on u-μP does...
Sparse maximal update parameterization: A holistic approach to sparse training dynamics
The key idea
The Road Less Scheduled
The key idea
Massive Activations in Large Language Models
The key idea
Simplifying Transformer Blocks
The key idea
Almost-scaled dot-product attention
TL;DR: Scaled dot product attention isn’t properly scaled, and that’s a good thing!
quantisation
DeepSeek-V3 & DeepSeek-R1 Technical Reports
With their V3 and R1 models, DeepSeek sets a new state-of-the-art in open-weight models and trades benchmark to benchmark with the best models from Anthropic...
The Super Weight in Large Language Models
The key idea
Scaling Laws for Precision
The key idea
Scaling FP8 training to trillion-token LLMs
The key idea
Scalable MatMul-free Language Modeling
The key idea
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
The key idea
transformers
Llama 3.2 Vision — A Deep Dive
Vision-Language Models (VLMs) allow LLMs to “see”, but how do they work? In this post, we’ll walk through the model changes needed to turn an LLM into a VLM ...
Memory Layers at Scale
The key idea
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
A transformer walk-through, with Gemma
Transformer-based LLMs seem mysterious, but they don’t need to. In this post, we’ll walk through a modern transformer LLM, Google’s Gemma, providing bare-bon...
Massive Activations in Large Language Models
The key idea
Simplifying Transformer Blocks
The key idea
fine-tuning
Transformer-Squared: Self-Adaptive LLMs
The key idea
computer-vision
diffusion
Generative Hierarchical Materials Search
The key idea
long-context
Titans: Learning to Memorize at Test Time
The key idea
sparsity
Memory Layers at Scale
The key idea
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
Mixture of a Million Experts
The key idea
reasoning
Evolving Deeper LLM Thinking
The key idea
DeepSeek-V3 & DeepSeek-R1 Technical Reports
With their V3 and R1 models, DeepSeek sets a new state-of-the-art in open-weight models and trades benchmark to benchmark with the best models from Anthropic...
Phi-4 Technical Report
The key idea
GNNs
Generative Hierarchical Materials Search
The key idea
G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering
The key idea
Scaling Deep Learning for Materials Discovery
The key idea
scaling-laws
How Does Critical Batch Size Scale in Pre-training?
The key idea
inference
Transformer-Squared: Self-Adaptive LLMs
The key idea
mixture-of-experts
Mixture of a Million Experts
The key idea
DiPaCo: Distributed Path Composition
The key idea
mup
Scale-preserving nonlinearities for u-μP
My colleagues and I always get excited when, every once in a while, deep learning research throws up a fun little maths problem. Our recent work on u-μP does...
unit-scaling
Scale-preserving nonlinearities for u-μP
My colleagues and I always get excited when, every once in a while, deep learning research throws up a fun little maths problem. Our recent work on u-μP does...
Almost-scaled dot-product attention
TL;DR: Scaled dot product attention isn’t properly scaled, and that’s a good thing!
number-formats
Scaling Laws for Precision
The key idea
FP8-LM: Training FP8 Large Language Models
The key idea
materials
Generative Hierarchical Materials Search
The key idea
Scaling Deep Learning for Materials Discovery
The key idea
position-embeddings
speculative-decoding
retrieval-augmented-generation
sparse-attention
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
not-transformers
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
The key idea
xLSTM: Extended Long Short-Term Memory
The key idea
self-correction
image-generation
chip-design
ChipNeMo: Domain-Adapted LLMs for Chip Design
The key idea
fp8
FP8-LM: Training FP8 Large Language Models
The key idea
DFT
Scaling Deep Learning for Materials Discovery
The key idea
active-learning
multi-modality
automated-theorem-proving
Back to Top ↑synthetic-data
Back to Top ↑distributed-training
DiPaCo: Distributed Path Composition
The key idea
local-updates
DiPaCo: Distributed Path Composition
The key idea
optimization
The Road Less Scheduled
The key idea
learning-rate-schedules
The Road Less Scheduled
The key idea
RNNs
xLSTM: Extended Long Short-Term Memory
The key idea
state-space-models
hallucinations
reinforcement-learning
Back to Top ↑optimisation
SOAP: Improving and Stabilizing Shampoo using Adam
The key idea
hiring
Graphcore Research is hiring!
We are pleased to have announce we have open positions for Research Scientists and Engineers to join our team.
batch-size
How Does Critical Batch Size Scale in Pre-training?
The key idea
byte-level
Back to Top ↑language-models
Back to Top ↑generative-models
Back to Top ↑embedding-models
Back to Top ↑training
Phi-4 Technical Report
The key idea
synthetic data
Phi-4 Technical Report
The key idea
VLMs
Llama 3.2 Vision — A Deep Dive
Vision-Language Models (VLMs) allow LLMs to “see”, but how do they work? In this post, we’ll walk through the model changes needed to turn an LLM into a VLM ...
reinforcement learning
DeepSeek-V3 & DeepSeek-R1 Technical Reports
With their V3 and R1 models, DeepSeek sets a new state-of-the-art in open-weight models and trades benchmark to benchmark with the best models from Anthropic...
memory
Titans: Learning to Memorize at Test Time
The key idea