A research paper co-authored by Miniml highlights a key idea for enterprise AI adoption: modern language models do not use all of their internal capacity on every request.

This behavior, known as activation sparsity, appears across major model families and tends to become stronger as models scale. In practice, large parts of the model can remain inactive depending on the input.

For deployment teams, that can translate into lower inference cost, lower latency, and better scalability. If sparsity is systematically leveraged, larger frontier models may become more operationally efficient than expected.

Paper: https://arxiv.org/pdf/2509.00454

Abstract

Input-dependent activation sparsity is a notable property of deep learning models, which has been extensively studied in networks with ReLU activations and is asso- ciated with efficiency, robustness, and interpretability. However, the approaches developed for ReLU-based models depend on exact zero activations and do not transfer directly to modern large language models (LLMs), which have abandoned ReLU in favor of other activation functions. As a result, current work on acti- vation sparsity in LLMs is fragmented, model-specific, and lacks consensus on which components to target. We propose a general framework to assess sparsity robustness and present a systematic study of the phenomenon in the FFN layers of modern LLMs, including diffusion LLMs. Our findings reveal universal patterns of activation sparsity in LLMs, provide insights into this phenomenon, and offer practical guidelines for exploiting it in model design and acceleration.