Congratulations to Professor Surya Ganguli, et al, on their Outstanding Paper Honorable Mention!
"Disentanglement with Biological Constraints: A Theory of Functional Cell Types," by Stanford postdoc James C. R. Whittington, Will Dorrell (UCL), Surya Ganguli (Stanford), Timothy Behrens (Oxford).
This work shows an interesting connection between machine learning and neuroscience. This paper introduced biologically inspired constraints, nonnegativity and energy efficiency, and mathematically proved these constraints lead to linear network disentanglement. It also shows empirically that the same constraints are effective for non-linear cases. It contains other contributions summarized at the end of the introduction section, which are not trivial either. The committee believes this work has the potential to give a big impact on future research in diverse fields of machine learning.
Paper Abstract: Neurons in the brain are often finely tuned for specific task variables. Moreover, such disentangled representations are highly sought after in machine learning. Here we mathematically prove that simple biological constraints on neurons, namely nonnegativity and energy efficiency in both activity and weights, promote such sought after disentangled representations by enforcing neurons to become selective for single factors of task variation. We demonstrate these constraints lead to disentanglement in a variety of tasks and architectures, including variational autoencoders. We also use this theory to explain why the brain partitions its cells into distinct cell types such as grid and object-vector cells, and also explain when the brain instead entangles representations in response to entangled task factors. Overall, this work provides a mathematical understanding of why single neurons in the brain often represent single human-interpretable factors, and steps towards an understanding task structure shapes the structure of brain representation.
Excerpted from "Announcing the ICLR 2023 Outstanding Paper Award Recipients"
