Hugging Face announced on September 22, 2026 that Transformers now supports GGUF models. Developers can select a GGUF checkpoint on the Hub, load it with from_pretrained and generate text through Transformers APIs, according to the company’s announcement.
A more direct route into Transformers
GGUF packages model weights and metadata in a single file. That can include tokenizer information and, when available, a conversation model. In Hugging Face’s loading example, users provide the Hub model identifier and the filename through gguf_file. The example requires no additional GGUF-specific configuration, bringing the format closer to a conventional Transformers workflow.
The change also makes quantized local models easier to compare by storage size. In the Qwen3.5-4B example from Unsloth, the BF16 GGUF file is listed at 8.42 GB, while the Q4_K_M quantized variant is listed at 2.74 GB. These are different model representations, so the smaller file should be understood as a storage comparison rather than a like-for-like performance or capability result.
The integration uses llama.cpp kernels
The compact path reuses ggml kernels from llama.cpp through the kernels library. Hugging Face says the goal is to move performance closer to llama.cpp while reducing the overhead of generate.
Its comparison places Transformers close to llama.cpp on three GGUF checkpoints, but the measurements do not use identical methods. The Transformers figure includes prefilling, whereas llama-bench reports decoding throughput without prefilling. That makes the comparison useful for showing the intended direction of the integration, but it is not a universal performance guarantee for every model or device.
Apple Silicon is the initial target
The first focus is local inference on Apple Silicon, with the compact inference path limited to MPS. GGUF support alone does not mean that compact kernels will be available on every device. Importing GGUF through dequantization remains a separate option.
Current compact-loader coverage includes dense and mixture-of-experts Qwen3.5 architectures, as well as compatible Qwen3.8 checkpoints. The same GGUF checkpoint can also be served with transformers serve, which exposes an API compatible with the OpenAI API.
Batching is not yet covered to the same extent. Hugging Face says it intends to extend the MPS work to generate_batch. For now, the release is more clearly suited to local single-request inference than to a fully mature batched serving path.
For developers using Apple Silicon with supported Qwen checkpoints, the update reduces the friction of bringing a quantized GGUF model into a Transformers-based workflow. The practical limits remain hardware support, architecture coverage and batching. The file-format integration simplifies model loading, but it does not remove those constraints.