M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models
The key idea Language models applied to reasoning have recently been shown to benefit from longer chain-of-thought sequences, which require the model to proc...
Posts by Tag
- LLMs 44
- efficient-inference 21
- quantisation 12
- efficient-training 11
- training-dynamics 10
- transformers 10
- scaling-laws 6
- fine-tuning 6
- reasoning 6
- diffusion 5
- GNNs 4
- computer-vision 4
- long-context 4
- sparsity 4
- inference 3
- speculative-decoding 3
- mixture-of-experts 3
- mup 3
- unit-scaling 2
- number-formats 2
- materials 2
- position-embeddings 2
- retrieval-augmented-generation 2
- sparse-attention 2
- not-transformers 2
- self-correction 2
- reinforcement-learning 2
- image-generation 2
- generative-models 2
- chip-design 1
- fp8 1
- DFT 1
- active-learning 1
- multi-modality 1
- automated-theorem-proving 1
- synthetic-data 1
- distributed-training 1
- local-updates 1
- optimization 1
- learning-rate-schedules 1
- RNNs 1
- state-space-models 1
- hallucinations 1
- optimisation 1
- hiring 1
- batch-size 1
- byte-level 1
- language-models 1
- embedding-models 1
- training 1
- synthetic data 1
- VLMs 1
- reinforcement learning 1
- memory 1
- flow-matching 1
- drug-design 1
- normalisation 1
- activation-functions 1
- reward-modeling 1
- inference-time-compute 1
- test-time-compute 1
- mamba 1
LLMs
Overtrained Language Models Are Harder to Fine-Tune
The key idea
Distillation Scaling Laws
The key idea
Transformer-Squared: Self-Adaptive LLMs
The key idea
Titans: Learning to Memorize at Test Time
The key idea
Evolving Deeper LLM Thinking
The key idea
DeepSeek-V3 & DeepSeek-R1 Technical Reports
With their V3 and R1 models, DeepSeek sets a new state-of-the-art in open-weight models and trades benchmark to benchmark with the best models from Anthropic...
Llama 3.2 Vision — A Deep Dive
Vision-Language Models (VLMs) allow LLMs to “see”, but how do they work? In this post, we’ll walk through the model changes needed to turn an LLM into a VLM ...
Phi-4 Technical Report
The key idea
The Super Weight in Large Language Models
The key idea
How Does Critical Batch Size Scale in Pre-training?
The key idea
Generative Hierarchical Materials Search
The key idea
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability
The key idea
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
The key idea
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
The key idea
Scalable MatMul-free Language Modeling
The key idea
xLSTM: Extended Long Short-Term Memory
The key idea
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
The key idea
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
The key idea
A transformer walk-through, with Gemma
Transformer-based LLMs seem mysterious, but they don’t need to. In this post, we’ll walk through a modern transformer LLM, Google’s Gemma, providing bare-bon...
G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering
The key idea
FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design
The key idea
Massive Activations in Large Language Models
The key idea
Simplifying Transformer Blocks
The key idea
ChipNeMo: Domain-Adapted LLMs for Chip Design
The key idea
efficient-inference
Matryoshka Quantization
The key idea
Distillation Scaling Laws
The key idea
DeepSeek-V3 & DeepSeek-R1 Technical Reports
With their V3 and R1 models, DeepSeek sets a new state-of-the-art in open-weight models and trades benchmark to benchmark with the best models from Anthropic...
Scaling Laws for Precision
The key idea
Scaling FP8 training to trillion-token LLMs
The key idea
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
The key idea
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
The key idea
Scalable MatMul-free Language Modeling
The key idea
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
The key idea
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
The key idea
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
The key idea
quantisation
Matryoshka Quantization
The key idea
DeepSeek-V3 & DeepSeek-R1 Technical Reports
With their V3 and R1 models, DeepSeek sets a new state-of-the-art in open-weight models and trades benchmark to benchmark with the best models from Anthropic...
The Super Weight in Large Language Models
The key idea
Scaling Laws for Precision
The key idea
Scaling FP8 training to trillion-token LLMs
The key idea
Scalable MatMul-free Language Modeling
The key idea
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
The key idea
efficient-training
Transformer-Squared: Self-Adaptive LLMs
The key idea
Memory Layers at Scale
The key idea
Scaling Laws for Precision
The key idea
How Does Critical Batch Size Scale in Pre-training?
The key idea
Mixture of a Million Experts
The key idea
training-dynamics
Scaling Laws for Precision
The key idea
How Does Critical Batch Size Scale in Pre-training?
The key idea
Scale-preserving nonlinearities for u-μP
My colleagues and I always get excited when, every once in a while, deep learning research throws up a fun little maths problem. Our recent work on u-μP does...
Sparse maximal update parameterization: A holistic approach to sparse training dynamics
The key idea
The Road Less Scheduled
The key idea
Massive Activations in Large Language Models
The key idea
Simplifying Transformer Blocks
The key idea
Almost-scaled dot-product attention
TL;DR: Scaled dot product attention isn’t properly scaled, and that’s a good thing!
transformers
Transformers without Normalisation
The key idea
Llama 3.2 Vision — A Deep Dive
Vision-Language Models (VLMs) allow LLMs to “see”, but how do they work? In this post, we’ll walk through the model changes needed to turn an LLM into a VLM ...
Memory Layers at Scale
The key idea
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
A transformer walk-through, with Gemma
Transformer-based LLMs seem mysterious, but they don’t need to. In this post, we’ll walk through a modern transformer LLM, Google’s Gemma, providing bare-bon...
Massive Activations in Large Language Models
The key idea
Simplifying Transformer Blocks
The key idea
scaling-laws
Distillation Scaling Laws
The key idea
How Does Critical Batch Size Scale in Pre-training?
The key idea
fine-tuning
Overtrained Language Models Are Harder to Fine-Tune
The key idea
Transformer-Squared: Self-Adaptive LLMs
The key idea
reasoning
M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models
The key idea Language models applied to reasoning have recently been shown to benefit from longer chain-of-thought sequences, which require the model to proc...
Evolving Deeper LLM Thinking
The key idea
DeepSeek-V3 & DeepSeek-R1 Technical Reports
With their V3 and R1 models, DeepSeek sets a new state-of-the-art in open-weight models and trades benchmark to benchmark with the best models from Anthropic...
Phi-4 Technical Report
The key idea
diffusion
Motion Prompting: Controlling Video Generation with Motion Trajectories
The key idea The central concept of Motion Prompting is to gain fine-grained control over video generation by conditioning a video diffusion model on spatio-...
Generative Hierarchical Materials Search
The key idea
GNNs
Generative Hierarchical Materials Search
The key idea
G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering
The key idea
Scaling Deep Learning for Materials Discovery
The key idea
computer-vision
long-context
Titans: Learning to Memorize at Test Time
The key idea
sparsity
Memory Layers at Scale
The key idea
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
Mixture of a Million Experts
The key idea
inference
Transformer-Squared: Self-Adaptive LLMs
The key idea
speculative-decoding
mixture-of-experts
Mixture of a Million Experts
The key idea
DiPaCo: Distributed Path Composition
The key idea
mup
Scale-preserving nonlinearities for u-μP
My colleagues and I always get excited when, every once in a while, deep learning research throws up a fun little maths problem. Our recent work on u-μP does...
unit-scaling
Scale-preserving nonlinearities for u-μP
My colleagues and I always get excited when, every once in a while, deep learning research throws up a fun little maths problem. Our recent work on u-μP does...
Almost-scaled dot-product attention
TL;DR: Scaled dot product attention isn’t properly scaled, and that’s a good thing!
number-formats
Scaling Laws for Precision
The key idea
FP8-LM: Training FP8 Large Language Models
The key idea
materials
Generative Hierarchical Materials Search
The key idea
Scaling Deep Learning for Materials Discovery
The key idea
position-embeddings
retrieval-augmented-generation
sparse-attention
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
not-transformers
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
The key idea
xLSTM: Extended Long Short-Term Memory
The key idea
self-correction
reinforcement-learning
image-generation
generative-models
Motion Prompting: Controlling Video Generation with Motion Trajectories
The key idea The central concept of Motion Prompting is to gain fine-grained control over video generation by conditioning a video diffusion model on spatio-...
chip-design
ChipNeMo: Domain-Adapted LLMs for Chip Design
The key idea
fp8
FP8-LM: Training FP8 Large Language Models
The key idea
DFT
Scaling Deep Learning for Materials Discovery
The key idea
active-learning
multi-modality
automated-theorem-proving
Back to Top ↑synthetic-data
Back to Top ↑distributed-training
DiPaCo: Distributed Path Composition
The key idea
local-updates
DiPaCo: Distributed Path Composition
The key idea
optimization
The Road Less Scheduled
The key idea
learning-rate-schedules
The Road Less Scheduled
The key idea
RNNs
xLSTM: Extended Long Short-Term Memory
The key idea
state-space-models
hallucinations
optimisation
SOAP: Improving and Stabilizing Shampoo using Adam
The key idea
hiring
Graphcore Research is hiring!
We are pleased to have announce we have open positions for Research Scientists and Engineers to join our team.
batch-size
How Does Critical Batch Size Scale in Pre-training?
The key idea
byte-level
Back to Top ↑language-models
Back to Top ↑embedding-models
Back to Top ↑training
Phi-4 Technical Report
The key idea
synthetic data
Phi-4 Technical Report
The key idea
VLMs
Llama 3.2 Vision — A Deep Dive
Vision-Language Models (VLMs) allow LLMs to “see”, but how do they work? In this post, we’ll walk through the model changes needed to turn an LLM into a VLM ...
reinforcement learning
DeepSeek-V3 & DeepSeek-R1 Technical Reports
With their V3 and R1 models, DeepSeek sets a new state-of-the-art in open-weight models and trades benchmark to benchmark with the best models from Anthropic...
memory
Titans: Learning to Memorize at Test Time
The key idea
flow-matching
Back to Top ↑drug-design
Back to Top ↑normalisation
Transformers without Normalisation
The key idea
activation-functions
Transformers without Normalisation
The key idea
reward-modeling
Back to Top ↑inference-time-compute
Back to Top ↑test-time-compute
Back to Top ↑mamba
M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models
The key idea Language models applied to reasoning have recently been shown to benefit from longer chain-of-thought sequences, which require the model to proc...