[Question]: Can internal features be used as effective semantic guidance signals to improve the training of DiT models?
[Answer]: Yes! It can even provide better feature guidance than external pretrained DINO features used in REPA.
[Question]: Could any feature be effective guidance?
[Answer]: No, not any feature is suitable. The desired internal guidance features should meet two requirements:
(1) They should structurally clean to help shallow blocks separate noise from signal.
(2) They should be semantically discriminative to help shallow layers learn effective representations.
To obtain such features without external supervision, we propose Self-Transcendence, which aligns DiT features with clean VAE latents for structure and applies classifier-free guidance for semantics. These internal features then supervise a new DiT from scratch, leading to better generation quality and faster convergence than REPA.
REPA leverages pre-trained DINO model features to effectively highlight semantically meaningful regions. In comparison, SRA and LayerSync rely on internal features from the in-training model, whose weak semantic representation limits their guidance for shallow layers. Instead, our approach generates features with clearer structural organization and richer semantics.
We propose a two-stage training framework, namely Self-Transcendence.
Firstly, we use clean VAE features as guidance to help the model distinguish useful information from noise in shallow layers. After a certain number of iterations, the model has learned more meaningful representations.
We then freeze this model and use its representation as a fixed teacher. To enhance the semantic expression in the features, we build a self-guided representation that better aligns with the target conditions.
The framework of our proposed Self-Transcendence. The spark icon indicates that the parameters of this layer are trainable, while the snowflake icon indicates that they are frozen.
Despite the powerful semantic separability obtained by REPA, it heavily relies on extensive and time-consuming pre-training with external data, which may not always be feasible and desirable. Though some recent methods such as LayerSync and SRA achieve self-acceleration, their features lack stable structural guidance and semantic separability for training. Instead, our Self-Transcendence not only achieves self-acceleration, but also demonstrates comparable semantic separability with REPA.
t-SNE visualizations of the guiding features extracted from (a) REPA, (b) LayerSync, (c) VAE features, and (d) our Self-Transcendence with t = 0.4 in the 200K iteration of SiTXL/2. Different colors represent different classes. As REPA, our internal guiding features demonstrate superior class separability.
Compared to methods using external features like REPA, our Self-Transcendence is easier and more efficient without using external data. And still, it outperforms all other self-contained techniques, achieving results comparable to or even better than REPA.
Comparisons with acceleration methods across different backbones on ImageNet. ↓ and ↑ indicate whether lower or higher values are better, respectively
We also conduct text-to-image (T2I) generation experiments using the same setting (including training dataset evaluation protocol) as REPA. We can see that REPA improves over original MMDiT with cleaner structure and fewer artifacts. Our method further improves prompt alignment and visual performance, yielding more realistic results with faithful structure and illumination.
Qualitative comparison on text-to-image generation (MS-COCO).
Self-transcendence: Is External Feature Guidance Indispensable for Accelerating Diffusion Transformer Training?
@misc{sun2026selftrans,
title={Self-transcendence: Is External Feature Guidance Indispensable for Accelerating Diffusion Transformer Training?},
author={Lingchen Sun and Rongyuan Wu and Zhengqiang Zhang and Ruibin Li and Yujing Sun and Shuaizheng Liu and Lei Zhang},
year={2026},
eprint={2601.07773},
archivePrefix={European Conference on Computer Vision},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2601.07773},
}