Quantisation

Reducing the numerical precision of model weights (for example from 16-bit to 4-bit) to lower memory requirements and speed up inference.

The trade is precision for efficiency. A 70B model at FP16 needs about 140 GB of memory; at 4-bit quantisation, about 35 GB. Quality degradation is measurable but often acceptable, especially for larger models. GPTQ, AWQ and GGUF are common quantisation formats, each with different trade-offs between speed, quality and hardware support.