Researchers at TNG Technology Consulting have described a way to extend an existing open-weight mixture-of-experts model with new, independently trained domain specialists. Their November 10 report uses Qwen3.5-35B-A3B-Base and adds experts for Rust and Haskell programming without retraining the original expert bank. The practical attraction is modularity. Rather than adjusting shared base parameters for every capability, the method adds separate new weight blocks that can later be assembled into one larger checkpoint. Primary source ↗
The researchers freeze the pretrained model, append new expert weights and train the small router rows that decide when to use them. A cold-start technique initializes each added expert from an active existing expert and temporarily freezes routing, so the new component is not immediately ignored. On the reported 35-billion-parameter system, each additional expert involves about 125 million trainable parameters. Training a domain expert used eight B200 accelerators and took roughly half a day, according to the project; this is not a low-cost consumer-hardware recipe.
The authors report Rust code-generation pass@1 increasing from 0.62 in the base model to 0.73 with a Rust expert, and Haskell increasing from 0.32 to 0.51 with its specialist. Combining independently trained experts yielded 0.70 for Rust and 0.50 for Haskell in their tests. General benchmark averages stayed close to the starting model, but individual settings varied; these data should be read as experiment-specific, not independent proof that modular expansion never degrades other skills.
The result is notable because combining ordinary LoRA adapters trained for separate specialties can introduce interference. In the authors' comparison, averaging or summing those adapters did not retain both domain gains, whereas concatenating new experts largely did. Important questions remain: router load balancing, expert-parallel inference efficiency and retention after further instruction training. Open Model Weights should track these as adaptation and training-evidence questions, separate from a claim that any particular derivative checkpoint is freely downloadable.
What this changes for the evidence layer
This is a domain-adaptation experiment on a previously published Qwen3.5-35B-A3B-Base checkpoint, not evidence of a wholly new foundation model. The article reports author-run code benchmarks; check whether resulting trained-expert checkpoints, training scripts, datasets and precise licenses are independently obtainable before listing a particular derivative as reproducibly open weight. Inference parallelism and scalability require further testing.
TNG Technology Consulting — Bring Your Own Expert! Extending MoE Models with New Domain Expertise
This brief is based on the cited primary source. Performance, benchmark and comparative claims remain attributed unless Open Model Weights publishes an independent measurement.
Open source article ↗