A new November 18 engineering write-up on the Hugging Face Blog shows a path to running a model with roughly 180 billion parameters on a Mac with 32GB of unified memory. The key is not squeezing every weight into RAM. Instead, contributor Tarek Ziadé describes a prototype that keeps frequently used mixture-of-experts modules in a bounded memory cache while streaming other expert weights and an oversized n-gram table from disk. This is an inference systems experiment, not the release of a new Qwen foundation checkpoint. Primary source ↗
Mixture-of-experts models activate only a subset of their specialist networks for each token. But a conventional runtime can still load or repeatedly transfer a huge bank of inactive experts, defeating the promise of sparse computation on smaller machines. The proposed Transformers path uses a reusable expert-slot pool and remaps router selections to slots so existing grouped matrix multiplication kernels can operate on cached experts rather than reading whole banks every time.
The test target was Qwen3.8-Flash-Next, which the report describes as having 48 layers with 512 experts each and ten activated per token. Its FP8 experts alone occupy around 121GB; the author's converted arrangement stores 3-bit experts in roughly 49GB on disk, keeps a 12GiB expert cache and streams a separate 51GB n-gram table. Dense components are quantized as well. The write-up reports about 18GiB after loading and a peak around 25.4GiB for a 6,000-token prompt. The files remain substantially larger than the physical memory available.
After targeted Metal kernels, prefetch improvements and changes to data movement, the author measured 13.9 decode tokens per second and 228 prefill tokens per second on the selected M5 configuration. The compared Strata setup delivered 14.8 and 220 respectively under the listed prompts and settings. This is promising, but the gap and numbers belong to one published test arrangement; they do not establish faster Transformers performance across other laptops, SSDs or workloads.
The implementation is also not yet a normal supported library feature. The `experts_offload` interface is in a proposed Transformers pull request, with model-specific conversion code and Metal kernels maintained separately. The author explicitly notes that the performance remains too slow for his own coding-agent workflow and that API discussions are ongoing. Trying it today requires custom files, a substantial conversion step and tolerance for experimental software.
For the open-weight ecosystem, the important idea is separating model availability from working-set memory: local users may be able to test massive sparse models without buying hardware capable of holding their entire weight set at once. The trade-off is storage, latency and implementation complexity. Teams should reproduce the published setup before drawing conclusions about local production readiness or cost.
What this changes for the evidence layer
This is a contributor's technical prototype, not a production Transformers release or a new base-model checkpoint. Published tokens-per-second, memory use and comparisons are author-run on specified hardware, not OMW independent benchmarks. Do not imply a full 180B BF16 model fits wholly in 32GB RAM: quantized experts and n-gram tables stream from disk.
Tarek Ziadé — first-person engineering report on the Hugging Face Blog
This brief is based on the cited primary source. Performance, benchmark and comparative claims remain attributed unless Open Model Weights publishes an independent measurement.
Open source article ↗