Princeton researchers trained neural networks from scratch on abstract sequence tasks to determine what factors drive the emergence of symbolic mechanisms. They found that greater data diversity produced stronger symbolic signatures, while architectural inductive biases were not necessary for their development. These signatures appeared in both transformer-based models and multilayer perceptrons, indicating that data distribution is the primary driver. The study used representational, attentional, and causal mediation analyses to confirm these findings across multiple axes of data diversity.
Read the full story at alphaxiv.org.
Photo: Alexandra_Koch / Pixabay
