“Mind reading” study looks inside coders’ brains

What do you think of when you visualize a list? Is your brain behaving the same when you consider adding something to an array and removing a node from a tree? Data structures are some of the key building blocks to any computer science problem, from the introductory to the complex, but coders don’t often give much thought to how the “computers” in their heads are handling these tools.

A group of researchers led by Prof. Westley Weimer and CSE grad student Yu Huang set out to do just that, using real-time fMRI readings to look inside coders’ brains while they program. This “mind reading” project studied whether manipulating data structures in a program (like balancing a tree) uses the same parts of the brain as spatial reasoning (rotating a map or turning a key in a lock). The paper, “Distilling Neural Representations of Data Structure Manipulation using fMRI and fNIRS,” earned a Best Paper Award at the ACM/IEEE International Conference on Software Engineering. 

To gather this data, the group conducted a human study with 76 participants completing several tasks involving data structures and spatial reasoning. Spatial reasoning is a person’s capacity to think about objects in three dimensions and draw conclusions about them from limited information. This type of thinking is typically measured by tests involving rotating or otherwise manipulating images of 3D objects.

Psychology research has shown that spatial ability can be a major factor in proficiencies such as mathematics, engineering, and the natural sciences – but despite these relationships, it has rarely been studied within software engineering. In fact, this was the first study focused on the relationship between software engineering tasks and spatial ability that didn’t rely on interviews.

Study participants were tasked with mentally manipulating lists, arrays, and trees. Afterwards, they completed mental rotation tasks such as determining if two perspective drawings portrayed the same 3D shapes.

The group then used two medical imaging techniques, functional magnetic resonance imaging (fMRI) and functional near-infrared spectroscopy (fNIRS), to provide objective measurements of active brain function during each of the activities.

Ultimately, researchers concluded that data structure and spatial operations are related but distinct neural tasks, using the same regions of the brain but to different degrees. In fact, 95%  of the study’s tree problems and mental rotation problems were statistically indistinguishable. 

The study also found that harder computer science problems have a bigger effect on neural activity than harder spatial reasoning problems, with more complex stimuli inducing a relatively higher cognitive load in data structure tasks than in mental rotation.

The paper was co-authored by Huang, CSE undergrads Xinyu Liu and Ryan Krueger, Weimer, Assistant Research Scientist of Biologic and Materials Sciences, CSE Research Fellow Kevin Leach, and Postdoctoral Researcher of Psychological and Brain Sciences Tyler Santander at the University of California, Santa Barbara.