SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints
Summary
- Amazon SageMaker Inference now supports NVIDIA Multi-Instance GPU (MIG) for hardware-isolated GPU partitioning.
- Users can run separate models in each partition or utilize a right-sized fraction of newer accelerators.
- The integration uses new optional API fields to assign one model per partition.
Why it matters
Hardware partitioning via NVIDIA MIG on SageMaker helps AI teams improve GPU utilization and cost efficiency for model endpoints by running multiple models or right-sized fractions on a single accelerator.