Compression Technique Interactions in Small Language Models: A Factorial Ablation Study

Impact 7 arXiv cs.AI

Summary

  • A factorial ablation study evaluates 72 compression configurations and ordering across small language models ranging from 2.7B to 7B parameters.
  • INT8 quantization delivers lossless memory reduction of 40-45%, whereas combining NF4 quantization with L1 magnitude pruning triggers model collapse across all tested combinations.
  • LoRA-based fine-tuning recovers up to 97% of degraded perplexity following aggressive pruning, driven by adaptation rather than a teacher distillation signal.
  • On-device CPU deployment shows 4-bit quantization runs 46-54% faster than 8-bit with minimal perplexity loss, while unstructured pruning yields no CPU latency or memory gains.

Why it matters

Provides practical guidance for deploying small language models on resource-constrained hardware by detailing how quantization, pruning, and fine-tuning recovery interact.

Who should care