Why your fine-tuning job takes longer than the FLOPs say

Impact 7 Hugging Face Blog

Summary

  • Researchers analyzed 272 production supervised fine-tuning jobs on NVIDIA H200 GPUs to understand why actual runtime deviates from FLOP-based estimates.
  • Traditional FLOP calculations and analytical models poorly predict end-to-end duration because they omit non-compute overheads like evaluation cadence, checkpointing, and I/O contention.
  • A CatBoost regressor using pre-dispatch features achieved an R² of 0.98 and a mean absolute error of 1.71 hours, vastly outperforming theoretical compute formulas.
  • Features with high predictive importance include evaluation sample volume, trainable parameter counts, and checkpoint frequencies.
  • The current model is limited to supervised fine-tuning on H200 hardware, but offers a reliable scheduling framework for shared GPU clusters.

Why it matters

Accurately predicting LLM fine-tuning duration helps cluster administrators optimize expensive GPU utilization and prevents downstream queue delays. By accounting for non-compute overheads, machine learning teams can schedule workloads more reliably.

Who should care

manager 75