Compression Technique Interactions in Small Language Models: A Factorial Ablation Study
Summary
- A factorial ablation study evaluates 72 compression configurations and ordering across small language models ranging from 2.7B to 7B parameters.
- INT8 quantization delivers lossless memory reduction of 40-45%, whereas combining NF4 quantization with L1 magnitude pruning triggers model collapse across all tested combinations.
- LoRA-based fine-tuning recovers up to 97% of degraded perplexity following aggressive pruning, driven by adaptation rather than a teacher distillation signal.
- On-device CPU deployment shows 4-bit quantization runs 46-54% faster than 8-bit with minimal perplexity loss, while unstructured pruning yields no CPU latency or memory gains.
Why it matters
Provides practical guidance for deploying small language models on resource-constrained hardware by detailing how quantization, pruning, and fine-tuning recovery interact.