Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learning
The key idea The authors present an agentic approach for RAG where, in each step, an LLM-based agent is given the choice to either (1) retrieve more informat...
Posts by Tag
- LLMs 52
- efficient-inference 27
- quantisation 13
- training-dynamics 12
- efficient-training 12
- transformers 10
- fine-tuning 10
- reasoning 9
- scaling-laws 7
- diffusion 7
- reinforcement-learning 6
- GNNs 4
- computer-vision 4
- long-context 4
- mixture-of-experts 4
- sparsity 4
- number-formats 3
- inference 3
- speculative-decoding 3
- retrieval-augmented-generation 3
- mup 3
- image-generation 3
- unit-scaling 2
- materials 2
- active-learning 2
- position-embeddings 2
- sparse-attention 2
- not-transformers 2
- self-correction 2
- optimisation 2
- generative-models 2
- test-time-compute 2
- dataset 2
- chip-design 1
- fp8 1
- DFT 1
- multi-modality 1
- automated-theorem-proving 1
- synthetic-data 1
- distributed-training 1
- local-updates 1
- optimization 1
- learning-rate-schedules 1
- RNNs 1
- state-space-models 1
- hallucinations 1
- hiring 1
- batch-size 1
- byte-level 1
- language-models 1
- embedding-models 1
- training 1
- synthetic data 1
- VLMs 1
- reinforcement learning 1
- memory 1
- flow-matching 1
- drug-design 1
- normalisation 1
- activation-functions 1
- reward-modeling 1
- inference-time-compute 1
- mamba 1
- AGI 1
- Evolutionary Algorithms 1
- RAG 1
- video-generation 1
- distillation 1
- molecule-generation 1
- ligand 1
LLMs
DataRater: Meta-Learned Dataset Curation
The key idea
Subliminal Learning: Language models transmit behavioral traits via hidden signals in data
The key idea
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
The key idea
ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
The key idea
Parallel Scaling Laws for Language Models
The key idea
M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models
The key idea Language models applied to reasoning have recently been shown to benefit from longer chain-of-thought sequences, which require the model to proc...
Overtrained Language Models Are Harder to Fine-Tune
The key idea
Distillation Scaling Laws
The key idea
Transformer-Squared: Self-Adaptive LLMs
The key idea
Titans: Learning to Memorize at Test Time
The key idea
Evolving Deeper LLM Thinking
The key idea
DeepSeek-V3 & DeepSeek-R1 Technical Reports
With their V3 and R1 models, DeepSeek sets a new state-of-the-art in open-weight models and trades benchmark to benchmark with the best models from Anthropic...
Llama 3.2 Vision — A Deep Dive
Vision-Language Models (VLMs) allow LLMs to “see”, but how do they work? In this post, we’ll walk through the model changes needed to turn an LLM into a VLM ...
Phi-4 Technical Report
The key idea
The Super Weight in Large Language Models
The key idea
How Does Critical Batch Size Scale in Pre-training?
The key idea
Generative Hierarchical Materials Search
The key idea
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability
The key idea
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
The key idea
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
The key idea
Scalable MatMul-free Language Modeling
The key idea
xLSTM: Extended Long Short-Term Memory
The key idea
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
The key idea
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
The key idea
A transformer walk-through, with Gemma
Transformer-based LLMs seem mysterious, but they don’t need to. In this post, we’ll walk through a modern transformer LLM, Google’s Gemma, providing bare-bon...
G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering
The key idea
FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design
The key idea
Massive Activations in Large Language Models
The key idea
Simplifying Transformer Blocks
The key idea
ChipNeMo: Domain-Adapted LLMs for Chip Design
The key idea
efficient-inference
Subliminal Learning: Language models transmit behavioral traits via hidden signals in data
The key idea
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
The key idea
Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation
Real-time interactive video generation requires (1) low latency (ideally frame generation requires a single model evaluation) and (2) the model can only use ...
Optimal Formats and the Cube Root of the PDF
Your boss emails you a point in 128-billion-dimensional space. “Llama 3.1 8B,” the message reads. “A not-so-large language model in bfloat16. But it’s too bi...
Parallel Scaling Laws for Language Models
The key idea
Matryoshka Quantization
The key idea
Distillation Scaling Laws
The key idea
DeepSeek-V3 & DeepSeek-R1 Technical Reports
With their V3 and R1 models, DeepSeek sets a new state-of-the-art in open-weight models and trades benchmark to benchmark with the best models from Anthropic...
Scaling Laws for Precision
The key idea
Scaling FP8 training to trillion-token LLMs
The key idea
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
The key idea
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
The key idea
Scalable MatMul-free Language Modeling
The key idea
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
The key idea
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
The key idea
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
The key idea
quantisation
Optimal Formats and the Cube Root of the PDF
Your boss emails you a point in 128-billion-dimensional space. “Llama 3.1 8B,” the message reads. “A not-so-large language model in bfloat16. But it’s too bi...
Matryoshka Quantization
The key idea
DeepSeek-V3 & DeepSeek-R1 Technical Reports
With their V3 and R1 models, DeepSeek sets a new state-of-the-art in open-weight models and trades benchmark to benchmark with the best models from Anthropic...
The Super Weight in Large Language Models
The key idea
Scaling Laws for Precision
The key idea
Scaling FP8 training to trillion-token LLMs
The key idea
Scalable MatMul-free Language Modeling
The key idea
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
The key idea
training-dynamics
ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
The key idea
Scaling Laws for Precision
The key idea
How Does Critical Batch Size Scale in Pre-training?
The key idea
Scale-preserving nonlinearities for u-μP
My colleagues and I always get excited when, every once in a while, deep learning research throws up a fun little maths problem. Our recent work on u-μP does...
Sparse maximal update parameterization: A holistic approach to sparse training dynamics
The key idea
The Road Less Scheduled
The key idea
Massive Activations in Large Language Models
The key idea
Simplifying Transformer Blocks
The key idea
Almost-scaled dot-product attention
TL;DR: Scaled dot product attention isn’t properly scaled, and that’s a good thing!
efficient-training
ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization
This is a Graphcore co-authored paper.
Transformer-Squared: Self-Adaptive LLMs
The key idea
Memory Layers at Scale
The key idea
Scaling Laws for Precision
The key idea
How Does Critical Batch Size Scale in Pre-training?
The key idea
Mixture of a Million Experts
The key idea
transformers
Transformers without Normalisation
The key idea
Llama 3.2 Vision — A Deep Dive
Vision-Language Models (VLMs) allow LLMs to “see”, but how do they work? In this post, we’ll walk through the model changes needed to turn an LLM into a VLM ...
Memory Layers at Scale
The key idea
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
A transformer walk-through, with Gemma
Transformer-based LLMs seem mysterious, but they don’t need to. In this post, we’ll walk through a modern transformer LLM, Google’s Gemma, providing bare-bon...
Massive Activations in Large Language Models
The key idea
Simplifying Transformer Blocks
The key idea
fine-tuning
ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization
This is a Graphcore co-authored paper.
ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
The key idea
Parallel Scaling Laws for Language Models
The key idea
Overtrained Language Models Are Harder to Fine-Tune
The key idea
Transformer-Squared: Self-Adaptive LLMs
The key idea
reasoning
ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
The key idea
M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models
The key idea Language models applied to reasoning have recently been shown to benefit from longer chain-of-thought sequences, which require the model to proc...
Evolving Deeper LLM Thinking
The key idea
DeepSeek-V3 & DeepSeek-R1 Technical Reports
With their V3 and R1 models, DeepSeek sets a new state-of-the-art in open-weight models and trades benchmark to benchmark with the best models from Anthropic...
Phi-4 Technical Report
The key idea
scaling-laws
Parallel Scaling Laws for Language Models
The key idea
Distillation Scaling Laws
The key idea
How Does Critical Batch Size Scale in Pre-training?
The key idea
diffusion
Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation
Real-time interactive video generation requires (1) low latency (ideally frame generation requires a single model evaluation) and (2) the model can only use ...
Motion Prompting: Controlling Video Generation with Motion Trajectories
The key idea The central concept of Motion Prompting is to gain fine-grained control over video generation by conditioning a video diffusion model on spatio-...
Generative Hierarchical Materials Search
The key idea
reinforcement-learning
Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learning
The key idea The authors present an agentic approach for RAG where, in each step, an LLM-based agent is given the choice to either (1) retrieve more informat...
GNNs
Generative Hierarchical Materials Search
The key idea
G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering
The key idea
Scaling Deep Learning for Materials Discovery
The key idea
computer-vision
long-context
Titans: Learning to Memorize at Test Time
The key idea
mixture-of-experts
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
The key idea
Mixture of a Million Experts
The key idea
DiPaCo: Distributed Path Composition
The key idea
sparsity
Memory Layers at Scale
The key idea
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
Mixture of a Million Experts
The key idea
number-formats
Optimal Formats and the Cube Root of the PDF
Your boss emails you a point in 128-billion-dimensional space. “Llama 3.1 8B,” the message reads. “A not-so-large language model in bfloat16. But it’s too bi...
Scaling Laws for Precision
The key idea
FP8-LM: Training FP8 Large Language Models
The key idea
inference
Transformer-Squared: Self-Adaptive LLMs
The key idea
speculative-decoding
retrieval-augmented-generation
Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learning
The key idea The authors present an agentic approach for RAG where, in each step, an LLM-based agent is given the choice to either (1) retrieve more informat...
mup
Scale-preserving nonlinearities for u-μP
My colleagues and I always get excited when, every once in a while, deep learning research throws up a fun little maths problem. Our recent work on u-μP does...
image-generation
Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation
Real-time interactive video generation requires (1) low latency (ideally frame generation requires a single model evaluation) and (2) the model can only use ...
unit-scaling
Scale-preserving nonlinearities for u-μP
My colleagues and I always get excited when, every once in a while, deep learning research throws up a fun little maths problem. Our recent work on u-μP does...
Almost-scaled dot-product attention
TL;DR: Scaled dot product attention isn’t properly scaled, and that’s a good thing!
materials
Generative Hierarchical Materials Search
The key idea
Scaling Deep Learning for Materials Discovery
The key idea
active-learning
ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization
This is a Graphcore co-authored paper.
position-embeddings
sparse-attention
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
not-transformers
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
The key idea
xLSTM: Extended Long Short-Term Memory
The key idea
self-correction
optimisation
ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization
This is a Graphcore co-authored paper.
SOAP: Improving and Stabilizing Shampoo using Adam
The key idea
generative-models
Motion Prompting: Controlling Video Generation with Motion Trajectories
The key idea The central concept of Motion Prompting is to gain fine-grained control over video generation by conditioning a video diffusion model on spatio-...
test-time-compute
dataset
ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization
This is a Graphcore co-authored paper.
DataRater: Meta-Learned Dataset Curation
The key idea
chip-design
ChipNeMo: Domain-Adapted LLMs for Chip Design
The key idea
fp8
FP8-LM: Training FP8 Large Language Models
The key idea
DFT
Scaling Deep Learning for Materials Discovery
The key idea
multi-modality
automated-theorem-proving
Back to Top ↑synthetic-data
Back to Top ↑distributed-training
DiPaCo: Distributed Path Composition
The key idea
local-updates
DiPaCo: Distributed Path Composition
The key idea
optimization
The Road Less Scheduled
The key idea
learning-rate-schedules
The Road Less Scheduled
The key idea
RNNs
xLSTM: Extended Long Short-Term Memory
The key idea
state-space-models
hallucinations
hiring
Graphcore Research is hiring!
We are pleased to have announce we have open positions for Research Scientists and Engineers to join our team.
batch-size
How Does Critical Batch Size Scale in Pre-training?
The key idea
byte-level
Back to Top ↑language-models
Back to Top ↑embedding-models
Back to Top ↑training
Phi-4 Technical Report
The key idea
synthetic data
Phi-4 Technical Report
The key idea
VLMs
Llama 3.2 Vision — A Deep Dive
Vision-Language Models (VLMs) allow LLMs to “see”, but how do they work? In this post, we’ll walk through the model changes needed to turn an LLM into a VLM ...
reinforcement learning
DeepSeek-V3 & DeepSeek-R1 Technical Reports
With their V3 and R1 models, DeepSeek sets a new state-of-the-art in open-weight models and trades benchmark to benchmark with the best models from Anthropic...
memory
Titans: Learning to Memorize at Test Time
The key idea
flow-matching
Back to Top ↑drug-design
Back to Top ↑normalisation
Transformers without Normalisation
The key idea
activation-functions
Transformers without Normalisation
The key idea
reward-modeling
Back to Top ↑inference-time-compute
Back to Top ↑mamba
M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models
The key idea Language models applied to reasoning have recently been shown to benefit from longer chain-of-thought sequences, which require the model to proc...
AGI
AlphaEvolve: A coding agent for scientific and algorithmic discovery
AlphaEvolve, evolves (no pun intended) the seminal method FunSearch introduced in late 2023. Powered by a frontier model rather than a smaller LLM, it levera...
Evolutionary Algorithms
AlphaEvolve: A coding agent for scientific and algorithmic discovery
AlphaEvolve, evolves (no pun intended) the seminal method FunSearch introduced in late 2023. Powered by a frontier model rather than a smaller LLM, it levera...
RAG
AlphaEvolve: A coding agent for scientific and algorithmic discovery
AlphaEvolve, evolves (no pun intended) the seminal method FunSearch introduced in late 2023. Powered by a frontier model rather than a smaller LLM, it levera...
video-generation
Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation
Real-time interactive video generation requires (1) low latency (ideally frame generation requires a single model evaluation) and (2) the model can only use ...