Transformers 5.22 brings Mistral Large 4 support and makes FP8 adapter training more practical

Hugging Face has released Transformers v5.22.0 with first-party integration for Mistral Large 4, the newly available trillion-parameter, multimodal mixture-of-experts model. This is a tooling story rather than a second announcement of the weights themselves: the model checkpoint was already published, while the October 29 library update makes its architecture available through the widely used Transformers integration path. For teams evaluating whether an open-weight model can move beyond a download into an application or fine-tuning workflow, runtime compatibility is a separate and important part of the release.

Transformers identifies Mistral Large 4, also called Le Chonk, as an approximately one-trillion-parameter model with around 49 billion routed active parameters per token, or 52 billion when embeddings and output layers are counted. Its published architecture includes 128 routed experts with four selected at each token and one shared expert, multimodal text-and-image inputs and a configured million-token context. Those are model specifications from the maintainers, not Open Model Weights measurements of achievable throughput, real context retrieval or server requirements.

The more general infrastructure improvement is training support for FP8-weight checkpoints. The release adds backward computation for dense and grouped FP8 matrix operations and marks the relevant trainability path as available for adapter training, including LoRA-style workflows using PEFT. Hugging Face says that chunking expert dequantization and avoiding unnecessary autograd work at inference lowers peak memory in the mixture-of-experts layer's backward phase from roughly 28.9 GiB to 2.8 GiB for an 8,000-token example. That highly specific figure is a maintainer measurement of one layer operation, not a claim that an entire trillion-parameter model can be fine-tuned on a 3 GiB GPU.

The same release includes model-specific support for NVIDIA Nemotron 3.5 Transcribe, improves loading configuration handling and fixes other processors and numeric edge cases. Operators should still validate their exact model revision, precision format, runtime dependencies and deployment hardware. Mistral Large 4's separate Ecosystem Open-Weight License, including its paid Model-as-a-Service condition, is not replaced by the Transformers package license. Open Model Weights records this release as a meaningful bridge between downloadable model artifacts and real adaptation workflows, while distinguishing implementation availability from independently reproduced quality or speed.