Open-model training infrastructure, fine-tuning efficiency and GPU scheduling

Dharma AI finds GPU FLOP estimates miss real fine-tuning time across 272 runs

A study of 7,382 GPU-hours finds deployment and checkpoint overheads matter: a workload-specific CatBoost predictor cuts reported mean error from 12.41 to 1.71 hours.

Open Model Weights published19 Nov 2026
Primary sourceDharma AI — original analysis of production H200 fine-tuning execution time
Source published2026-11-18
FOLLOW OUR COVERAGE

Follow Open Model Weights on Google See more of our open-weight AI reporting in Google Search.

Add as a Preferred Source on Google
Original minimalist editorial illustration representing compute in AI research; conceptual, not actual model output.
AI-generated editorial illustration · Not official model imagery.

An analysis by Dharma AI asks a practical question for teams adapting open-weight models: can operators predict how long a fine-tuning job will really occupy their GPUs before dispatching it? The authors examined 272 supervised fine-tuning jobs from a shared NVIDIA H200 cluster, covering about 7,382 GPU-hours. Their results suggest that estimates based mainly on theoretical floating-point operations can badly underestimate end-to-end runtime because training includes much more than matrix multiplication. Primary source ↗

The study includes LoRA and full fine-tuning runs involving models of up to 15 billion parameters, conducted with Hugging Face Transformers and Unsloth. Individual jobs lasted from 1.6 to 172.4 hours, with an average around 23.4 hours. Runs used one to four H200s within a larger shared pool. This gives the report operational texture, although the machines and task mix all come from a single infrastructure environment.

Dharma AI reports that the best of several FLOP-based reference assumptions had a mean absolute prediction error of 12.41 hours and an R-squared value of 0.31. A CatBoost regression model using configuration information available before the job started reduced reported mean absolute error to 1.71 hours, with R-squared 0.98, under stratified five-fold cross-validation. Its median absolute error was around half an hour. These are the authors' cross-validation scores, not independent prospective deployment results.

The key difference lies in the work outside the main training loop. Downloading and loading weights, preparing datasets, evaluating checkpoints, uploading model snapshots and competing for network or storage resources all consume wall-clock time. An operator who only divides estimated training FLOPs by nominal GPU capacity misses those fixed and configuration-dependent costs. Dharma's predictor therefore includes scheduled evaluation volume, checkpoint frequency, per-GPU data load and the fine-tuning regime alongside the theoretical baseline.

For the open-model ecosystem, such estimates could support more reliable job packing, reservations and experiment scheduling on shared accelerators. The broader implication is not a magic GPU speedup or direct evidence of cheaper tokens; the computation itself has not become faster. More accurate duration forecasts might instead reduce idle intervals and scheduling conflicts, provided the predictor generalizes to the operator's own models, storage topology and policies.

The study has important boundaries. It covers supervised fine-tuning on H200 hardware, not reinforcement-learning methods such as RLHF, DPO or RLVR. The sample includes relatively few extremely long jobs, and prediction error grows above 60 hours. The article previews a forthcoming paper and says the full feature construction and search details will be published later. Before trusting the reported performance on a different GPU cluster, researchers would need to reproduce the method using local execution logs and an independent holdout period.

OMW REGISTRY WATCH

What this changes for the evidence layer

Original first-party quantitative report, not an open-weight model launch or a released forecasting checkpoint. The author says the full paper is forthcoming; neither reusable trained predictor weights nor independent cross-cluster validation are established by the announcement. Do not make universal time/cost-savings claims.

PRIMARY SOURCE

Dharma AI — original analysis of production H200 fine-tuning execution time

This brief is based on the cited primary source. Performance, benchmark and comparative claims remain attributed unless Open Model Weights publishes an independent measurement.

Open source article ↗