Why your fine-tuning job takes longer than the FLOPs say
Summary
- Researchers analyzed 272 production supervised fine-tuning jobs on NVIDIA H200 GPUs to understand why actual runtime deviates from FLOP-based estimates.
- Traditional FLOP calculations and analytical models poorly predict end-to-end duration because they omit non-compute overheads like evaluation cadence, checkpointing, and I/O contention.
- A CatBoost regressor using pre-dispatch features achieved an R² of 0.98 and a mean absolute error of 1.71 hours, vastly outperforming theoretical compute formulas.
- Features with high predictive importance include evaluation sample volume, trainable parameter counts, and checkpoint frequencies.
- The current model is limited to supervised fine-tuning on H200 hardware, but offers a reliable scheduling framework for shared GPU clusters.
Why it matters
Accurately predicting LLM fine-tuning duration helps cluster administrators optimize expensive GPU utilization and prevents downstream queue delays. By accounting for non-compute overheads, machine learning teams can schedule workloads more reliably.