Open-weight model adaptation and mixture-of-experts research

TNG demonstrates modular MoE experts for Rust and Haskell

Researchers add independently trained experts to a frozen Qwen3.5 MoE and combine the skills in one checkpoint, avoiding a standard adapter-merging bottleneck.

Open Model Weights published15 Nov 2026
Primary sourceTNG Technology Consulting — Bring Your Own Expert! Extending MoE Models with New Domain Expertise
Source published2026-11-10
FOLLOW OUR COVERAGE

Follow Open Model Weights on Google See more of our open-weight AI reporting in Google Search.

Add as a Preferred Source on Google
Original premium 3D concept of modular crystal-and-chrome expert cartridges being installed into a neural processor under luminous teal studio lighting.
Original conceptual illustration of composable mixture-of-experts weight modules; not an actual checkpoint visualization or independent performance measurement. AI-generated editorial illustration; not a documentary photograph.

Researchers at TNG Technology Consulting have described a way to extend an existing open-weight mixture-of-experts model with new, independently trained domain specialists. Their November 10 report uses Qwen3.5-35B-A3B-Base and adds experts for Rust and Haskell programming without retraining the original expert bank. The practical attraction is modularity. Rather than adjusting shared base parameters for every capability, the method adds separate new weight blocks that can later be assembled into one larger checkpoint. Primary source ↗

The researchers freeze the pretrained model, append new expert weights and train the small router rows that decide when to use them. A cold-start technique initializes each added expert from an active existing expert and temporarily freezes routing, so the new component is not immediately ignored. On the reported 35-billion-parameter system, each additional expert involves about 125 million trainable parameters. Training a domain expert used eight B200 accelerators and took roughly half a day, according to the project; this is not a low-cost consumer-hardware recipe.

The authors report Rust code-generation pass@1 increasing from 0.62 in the base model to 0.73 with a Rust expert, and Haskell increasing from 0.32 to 0.51 with its specialist. Combining independently trained experts yielded 0.70 for Rust and 0.50 for Haskell in their tests. General benchmark averages stayed close to the starting model, but individual settings varied; these data should be read as experiment-specific, not independent proof that modular expansion never degrades other skills.

The result is notable because combining ordinary LoRA adapters trained for separate specialties can introduce interference. In the authors' comparison, averaging or summing those adapters did not retain both domain gains, whereas concatenating new experts largely did. Important questions remain: router load balancing, expert-parallel inference efficiency and retention after further instruction training. Open Model Weights should track these as adaptation and training-evidence questions, separate from a claim that any particular derivative checkpoint is freely downloadable.

OMW REGISTRY WATCH

What this changes for the evidence layer

This is a domain-adaptation experiment on a previously published Qwen3.5-35B-A3B-Base checkpoint, not evidence of a wholly new foundation model. The article reports author-run code benchmarks; check whether resulting trained-expert checkpoints, training scripts, datasets and precise licenses are independently obtainable before listing a particular derivative as reproducibly open weight. Inference parallelism and scalability require further testing.

PRIMARY SOURCE

TNG Technology Consulting — Bring Your Own Expert! Extending MoE Models with New Domain Expertise

This brief is based on the cited primary source. Performance, benchmark and comparative claims remain attributed unless Open Model Weights publishes an independent measurement.

Open source article ↗