Unit Scaling

Welcome to the unit-scaling library. This library is designed to facilitate the use of the unit scaling method, as outlined in the paper Unit Scaling: Out-of-the-Box Low-Precision Training (ICML, 2023).

For a demonstration of the library, see Out-of-the-Box FP8 Training — a notebook showing how to unit-scale the nanoGPT model.

Installation

To install unit-scaling, run:

pip install git+https://github.com/graphcore-research/unit-scaling.git

Getting Started

We recommend that new users get started with Section 1. User guide.

A reference outlining our API can be found at Section 4. API reference.

The following video gives a broad overview of the workings of unit scaling.

Note

The library is currently in its beta release. Some features have yet to be implemented and occasional bugs may be present. We’re keen to help users with any problems they encounter.

Development

For those who wish to develop on the unit-scaling codebase, clone or fork our GitHub repo and follow the instructions in our developer guide.

Contents