The Super Weight in Large Language Models
The key idea
Posts by Tag
- LLMs 30
- efficient-inference 16
- training-dynamics 10
- efficient-training 9
- quantisation 9
- transformers 6
- computer-vision 4
- fine-tuning 4
- GNNs 3
- scaling-laws 3
- diffusion 3
- long-context 3
- mixture-of-experts 3
- sparsity 3
- mup 3
- unit-scaling 2
- number-formats 2
- materials 2
- inference 2
- position-embeddings 2
- speculative-decoding 2
- retrieval-augmented-generation 2
- sparse-attention 2
- not-transformers 2
- self-correction 2
- image-generation 2
- chip-design 1
- fp8 1
- DFT 1
- active-learning 1
- multi-modality 1
- automated-theorem-proving 1
- synthetic-data 1
- distributed-training 1
- local-updates 1
- optimization 1
- learning-rate-schedules 1
- RNNs 1
- state-space-models 1
- hallucinations 1
- reinforcement-learning 1
- optimisation 1
- hiring 1
- reasoning 1
- batch-size 1
LLMs
How Does Critical Batch Size Scale in Pre-training?
The key idea
Generative Hierarchical Materials Search
The key idea
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability
The key idea
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
The key idea
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
The key idea
Scalable MatMul-free Language Modeling
The key idea
xLSTM: Extended Long Short-Term Memory
The key idea
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
The key idea
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
The key idea
A transformer walk-through, with Gemma
Transformer-based LLMs seem mysterious, but they don’t need to. In this post, we’ll walk through a modern transformer LLM, Google’s Gemma, providing bare-bon...
G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering
The key idea
FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design
The key idea
Massive Activations in Large Language Models
The key idea
Simplifying Transformer Blocks
The key idea
ChipNeMo: Domain-Adapted LLMs for Chip Design
The key idea
efficient-inference
Scaling Laws for Precision
The key idea
Scaling FP8 training to trillion-token LLMs
The key idea
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
The key idea
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
The key idea
Scalable MatMul-free Language Modeling
The key idea
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
The key idea
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
The key idea
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
The key idea
training-dynamics
Scaling Laws for Precision
The key idea
How Does Critical Batch Size Scale in Pre-training?
The key idea
Scale-preserving nonlinearities for u-μP
My colleagues and I always get excited when, every once in a while, deep learning research throws up a fun little maths problem. Our recent work on u-μP does...
Sparse maximal update parameterization: A holistic approach to sparse training dynamics
The key idea
The Road Less Scheduled
The key idea
Massive Activations in Large Language Models
The key idea
Simplifying Transformer Blocks
The key idea
Almost-scaled dot-product attention
TL;DR: Scaled dot product attention isn’t properly scaled, and that’s a good thing!
efficient-training
Scaling Laws for Precision
The key idea
How Does Critical Batch Size Scale in Pre-training?
The key idea
Mixture of a Million Experts
The key idea
quantisation
The Super Weight in Large Language Models
The key idea
Scaling Laws for Precision
The key idea
Scaling FP8 training to trillion-token LLMs
The key idea
Scalable MatMul-free Language Modeling
The key idea
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
The key idea
transformers
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
A transformer walk-through, with Gemma
Transformer-based LLMs seem mysterious, but they don’t need to. In this post, we’ll walk through a modern transformer LLM, Google’s Gemma, providing bare-bon...
Massive Activations in Large Language Models
The key idea
Simplifying Transformer Blocks
The key idea
computer-vision
fine-tuning
GNNs
Generative Hierarchical Materials Search
The key idea
G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering
The key idea
Scaling Deep Learning for Materials Discovery
The key idea
scaling-laws
How Does Critical Batch Size Scale in Pre-training?
The key idea
diffusion
Generative Hierarchical Materials Search
The key idea
long-context
mixture-of-experts
Mixture of a Million Experts
The key idea
DiPaCo: Distributed Path Composition
The key idea
sparsity
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
Mixture of a Million Experts
The key idea
mup
Scale-preserving nonlinearities for u-μP
My colleagues and I always get excited when, every once in a while, deep learning research throws up a fun little maths problem. Our recent work on u-μP does...
unit-scaling
Scale-preserving nonlinearities for u-μP
My colleagues and I always get excited when, every once in a while, deep learning research throws up a fun little maths problem. Our recent work on u-μP does...
Almost-scaled dot-product attention
TL;DR: Scaled dot product attention isn’t properly scaled, and that’s a good thing!
number-formats
Scaling Laws for Precision
The key idea
FP8-LM: Training FP8 Large Language Models
The key idea
materials
Generative Hierarchical Materials Search
The key idea
Scaling Deep Learning for Materials Discovery
The key idea
inference
position-embeddings
speculative-decoding
retrieval-augmented-generation
sparse-attention
Speeding up LLM inference using SparQ Attention & llama.cpp
With the rapid advances in the capabilities of large language models (LLMs), there is an increasing need for efficient inference platforms that would enable ...
not-transformers
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
The key idea
xLSTM: Extended Long Short-Term Memory
The key idea
self-correction
image-generation
chip-design
ChipNeMo: Domain-Adapted LLMs for Chip Design
The key idea
fp8
FP8-LM: Training FP8 Large Language Models
The key idea
DFT
Scaling Deep Learning for Materials Discovery
The key idea
active-learning
multi-modality
automated-theorem-proving
Back to Top ↑synthetic-data
Back to Top ↑distributed-training
DiPaCo: Distributed Path Composition
The key idea
local-updates
DiPaCo: Distributed Path Composition
The key idea
optimization
The Road Less Scheduled
The key idea
learning-rate-schedules
The Road Less Scheduled
The key idea
RNNs
xLSTM: Extended Long Short-Term Memory
The key idea
state-space-models
hallucinations
reinforcement-learning
Back to Top ↑optimisation
SOAP: Improving and Stabilizing Shampoo using Adam
The key idea
hiring
Graphcore Research is hiring!
We are pleased to have announce we have open positions for Research Scientists and Engineers to join our team.
reasoning
Back to Top ↑batch-size
How Does Critical Batch Size Scale in Pre-training?
The key idea