Neural Collapse (Project Website)

ICASSP’24 Tutorial on Understanding Deep Representation Learning via Neural Collapse (Lecture 1, Lecture 2)

Neural Collapse in Last-Layer

• Zhihui Zhu*, Tianyu Ding*, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, Qing Qu. A Geometric Analysis of Neural Collapse with Unconstrained Features. Neural Information Processing Systems (NeurIPS’21), 2021. (spotlight, top 3%)
  Preprint – PDF –  Slides –  BibTex – Code – Video
• Jinxin Zhou*, Xiao Li*, Tianyu Ding, Chong You, Qing Qu*, Zhihui Zhu*. On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features. International Conference on Machine Learning (ICML’22), 2022.
  Preprint – PDF –  BibTex – Code
• Peng Wang*, Huikang Liu*, Can Yaras*, Laura Balzano, Qing Qu. Linear Convergence Analysis of Neural Collapse with Unconstrained Features. OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022.
  Preprint – PDF –  BibTex
• Can Yaras*, Peng Wang*, Zhihui Zhu, Laura Balzano, Qing Qu. Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold. Neural Information Processing Systems (NeurIPS’22), 2022.
  Preprint – PDF –  BibTex – Code
• Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, Zhihui Zhu. Are All Losses Created Equal: A Neural Collapse Perspective. Neural Information Processing Systems (NeurIPS’22), 2022.
  Preprint – PDF –  BibTex – Code
• Pengyu Li*, Xiao Li*, Yutong Wang, Qing Qu. Neural Collapse in Multi-label Learning with Pick-all-label Loss. International Conference on Machine Learning (ICML’24), 2024.
  Preprint – PDF –  BibTex – Code
• Jiachen Jiang*, Jinxin Zhou*, Peng Wang, Qing Qu, Dustin Mixon, Chong You*, Zhihui Zhu*. Generalized Neural Collapse for a Large Number of Classes. International Conference on Machine Learning (ICML’24), 2024.
  Preprint – PDF –  BibTex

Feature Separation & Neural Collapse Across Intermediate Layers

• Alec S. Xu, Can Yaras, Peng Wang, Qing Qu. Understanding How Nonlinear Layers Create Linearly Separable Features for Low-Dimensional Data. Arxiv Preprint arXiv:2501.02364, 2025.
  Preprint – PDF –  BibTex
• Peng Wang*, Xiao Li*, Can Yaras, Zhihui Zhu, Laura Balzano, Wei Hu, Qing Qu. Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination. ArXiv Preprint arXiv:2311.02960, 2023.
  Preprint – PDF –  BibTex – Code
• Can Yaras*, Peng Wang*, Wei Hu, Zhihui Zhu, Laura Balzano, Qing Qu. The Law of Parsimony in Gradient Descent for Learning Deep Linear Networks. ArXiv Preprint arXiv:2306.01154, 2023.
  Preprint – PDF –  BibTex – Code – Slides

Representation Learning in Diffusion Models

• Xiao Li, Zekai Zhang, Xiang Li, Siyi Chen, Zhihui Zhu, Peng Wang, Qing Qu. Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling. Arxiv Preprint arXiv:2502.05743, 2025.
  Preprint – PDF –  BibTex

Multimodal Learning

• Can Yaras*, Siyi Chen*, Peng Wang, Qing Qu. Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learning. Arxiv Preprint arXiv:2412.07909, 2024.
  Preprint – PDF –  BibTex

Maximizing Rate Reduction

• Peng Wang, Huikang Liu, Druv Pai, Yaodong Yu, Zhihui Zhu, Qing Qu, Yi Ma. A Global Geometric Analysis of Maximal Coding Rate Reduction. International Conference on Machine Learning (ICML’24), 2024.
  Preprint – PDF –  BibTex

Applications

• Xiao Li*, Sheng Liu*, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda, Zhihui Zhu, Qing Qu. Understanding and Improving Transfer Learning of Deep Models via Neural Collapse. Transactions on Machine Learning Research (TMLR), 2024.
  Preprint – PDF –  BibTex
• Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, Yi Ma. Investigating the Catastrophic Forgetting in Multimodal Large Language Models. Conference on Parsimony and Learning (CPAL’24), 2024.
  Preprint – PDF –  BibTex
• Shuo Xie, Jiahao Qiu, Ankita Pasad, Li Du, Qing Qu, Hongyuan Mei. Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning. Findings of Empirical Methods in Natural Language Processing (EMNLP), 2022.
  Preprint – PDF –  BibTex – Code