Hugging Face published details of ShadowPEFT on September 15, 2026. The parameter-efficient fine-tuning method is presented as natively supported by the 🤗 PEFT library, with its implementation already merged into the main branch and publication planned for the next PEFT release. The article is attributed to Zongxi Li, Xianming Li, Tsz-fung Andrew Lee, Jing Li, Haoran Xie and Qing Li.
ShadowPEFT changes how a task is represented during fine-tuning. Rather than adding independent low-rank updates at different layers, it introduces a compact shadow model with a persistent hidden state that is updated throughout the decoder. At each Transformer layer, the method sends the difference between the base model's hidden state and the shadow state through a low-rank bottleneck. It then runs the frozen Transformer layer before updating the shadow state with two small MLPs and a controlled residual. Information consequently moves in both directions between the base model and the shadow model.
The integration keeps the familiar PEFT workflow
ShadowPEFT uses ShadowConfig, BaseTuner and BaseTunerLayer. The documented call, get_peft_model(model, ShadowConfig(...)), follows the same general pattern as LoRA, as do save_pretrained and from_pretrained. Once training is complete, users can leave the shadow model attached to its base model or call unload_shadow() to obtain a DetachedShadowModel for independent inference.
The example uses Qwen/Qwen3-8B with r=8, one shadow hidden layer, the mirror shadow model and the CAUSAL_LM task type. That setup points to developers already working with PEFT-style model adaptation. ShadowPEFT is presented as a method for modifying existing models, not as a standalone chatbot or end-user application.
The reported language-model result comes with a memory cost
In the MetaMathQA-to-GSM8K experiment, the methods were compared on Llama-3.2-3B using exact match, PEFT's default hyperparameters and an 80 GB NVIDIA A100. ShadowPEFT used 8.66 million trainable parameters and reached 48.1% exact match after 17 minutes. Its peak memory use was 28.2 GB, and its checkpoint measured 26.0 MB.
LoRA reached 46.9% exact match with 22.3 GB of memory and a 36.7 MB checkpoint. DoRA reached 46.2% with 24.5 GB of memory and a 37.2 MB checkpoint. The ShadowPEFT page attributes the higher memory use to running the shadow model for every token and keeping two KV caches. The result therefore describes a trade-off rather than a universal replacement for the alternatives: the reported exact-match score and checkpoint size are better in this experiment, while the runtime memory requirement is higher.
Image-generation results favor the same design in this comparison
The image-generation comparison used DreamBooth with FLUX.2-klein-base-4B and around 20 photos of the same cat. ShadowPEFT recorded a DINOv2 cosine similarity of 0.717 and drift of 0.244, alongside 31.2 million trainable parameters, eight minutes of training, 10.3 GB of peak memory and a 74.5 MB checkpoint.
LoRA recorded 0.671 for similarity and 0.274 for drift, while DoRA recorded 0.682 and 0.250. The page reports ShadowPEFT ahead of both methods on every listed metric in this comparison. These figures come from the presented DreamBooth experiment; they do not establish that the same balance will apply to every model, dataset or fine-tuning task.
For developers already using PEFT, the practical change is an additional stateful architecture that preserves the library's familiar configuration and save/load pattern. The presented results suggest potential gains across the two tasks, but the persistent shadow computation and dual KV caches make hardware capacity part of the decision. The planned next PEFT release will determine when this merged implementation becomes available through the released library.