The creation of a virtual child

Justine Cassell programs virtual peers for children to help them learn
Illustration: Mrzyk & Moriceau for BOLD
Illustration: Mrzyk & Moriceau for BOLD

The development of new technology is often followed quickly by claims about how it will revolutionize education for the masses. After World War II, the filmstrip projector gained popularity in classrooms and promised to change forever the way students learn. Then videocassettes, laserdiscs, and DVDs became the latest and greatest in educational technology — that is, until computers arrived on the scene.

While every advance brings forward a better, faster machine, many experts would argue that the accompanying educational revolutions have failed to happen as predicted. According to Justine Cassell, the Associate Dean for Technology Strategy and Impact in the School of Computer Science at Carnegie Mellon University, the aspect of social interaction has been one of the most important missing links.

“What is the problem with these educational revolutions? They are new technologies but rely on extremely old, and I would say outdated, models of pedagogy,” said Cassell during her keynote talk at the Society for Research in Child Development‘s special topic meeting, Technology and Media in Children’s Development. “Because as we all know, we do not learn by having knowledge poured into us. We learn by having a belief that is confronted by an alternative belief, and one that has more evidence in its favor.”


Her research has shown that the types of social interactions important to learning — for instance, storytelling or discussion of the material — don’t have to be strictly human-to-human. For over two decades, Cassell has built virtual humans with conversational, social, and narrative intelligence called Embodied Conversational Agents (ECAs). Although ECAs are restricted to a screen, they are capable of interacting with humans using both language and nonverbal behavior.

How to build a virtual peer

More recently, her work has focused on the ECA as a “virtual peer” for children to help them learn. In this application, ECAs can collaborate with or be tutored by students to enhance their grasp of skills and domains. The hope is that teaching computers how to think will allow children to delve into the material on a deeper level themselves. Virtual peers are based on extensive study of human peers engaged in collaborative tasks, both in and out of the classroom. That study includes analysis of children’s language, as well as of their nonverbal behavior and the ways in which they collaborate (or don’t) and learn (or don’t).

For instance, Alex is a virtual peer based on an ethnographic and sociolinguistic analysis of African American children between the ages of 8 and 10. In a 2013 study, Cassell and her colleagues found that children who natively spoke African American Vernacular English (AAVE) demonstrated the strongest performance on a science task when the virtual peer used AAVE rather than the standard English dialect.

“We do not learn by having knowledge poured into us. We learn by having a belief that is confronted by an alternative belief, and one that has more evidence in its favor.”

More recently, a longitudinal study demonstrated that gains in science reasoning over a five-week period were stronger for children who interacted with an AAVE-speaking virtual peer than they were for children who interacted with a virtual peer that spoke MAE. In addition, children who worked with the AAVE-speaking peer had more positive language ideologies by the end of the study – that is, they did not share the negative attitudes towards their own dialect shown by children who worked with the MAE-speaking virtual peer.

Such results challenge the cultural assumptions made when developing educational technologies and suggest that a more tailored learning experience makes children better able to grasp concepts. Instead of assuming that all children grow up in a standard English dialect household, educational software that uses their native AAVE in some contexts could help boost African American students’ learning.

Cassell began creating virtual humans in 1993. After graduate school, as a professor of linguistics, psychology, and French at Penn State University, she struggled to model the relationship between verbal and nonverbal behavior in adults and children. Cassell decided to seek answers from the world-renowned computer science department at the University of Pennsylvania, where she went to spend a sabbatical year. Starting from scratch, she began to cobble together a virtual human.

Then in 2000, after having moved to a faculty position at the MIT Media Lab, she began building virtual peers after reading research by psychologist Marjorie Taylor on children’s creation of imaginary companions.

“I was really affected by her research showing that children with imaginary playmates did better at all kinds of abstract tasks,” Cassell said. “And in a sense, a virtual peer is an imaginary playmate that is given some kind of graphical existence on the screen.”

“The creation and use of virtual peers may preserve the value of social interaction and engagement in education tech — while also teaching us about what makes us human.”

The first step in making a virtual peer is the painstaking cataloging and scrutiny of interactions, both verbal and nonverbal, between real children. The methodology involves witnessing child-child interactions, building formal models based on the data, and then using those models to develop algorithms that govern a virtual peer. Subsequently, once a prototype of the virtual peer is built, it continues to be tweaked and refined as real children interact with the computer program.

“We look at every video that we’ve collected in one-thirtieth of a second granularity. So we look at every eye gaze shift, every smile, every head nod. We look at every posture shift and hand movement,” she said. “We look at all the nouns the children produce, we look at conversational strategies and their functions. We end up with a multilevel, multimodal model of dyadic peer interaction.”

Using virtual peers to help children with autism

One of Cassell’s interests is the idea of rapport. Her lab first developed a theory of how rapport is built, maintained, and destroyed among teens by annotating interactions between high school student pairs during peer tutoring tasks. A 2014 study outlined a model of rapport management, which included elements of mutual attentiveness and coordination, while another paper that same year presented a computational architecture that allows virtual agents to enhance, maintain, and destroy long-term rapport with their users.

Outside of science and literacy skills, Cassell has also explored the use of virtual peers to help children with autism practice social interaction.

“One of the things that we found quite by accident was that children with high-functioning autism or Asperger’s are more likely to show sophisticated social skills interacting with a virtual peer than with a real peer,” she said.

However, Cassell’s goal is not to consign children with high-functioning autism or Asperger’s to a life of interacting only with virtual peers. Her 2008 study had children with autism engage with both a virtual and a human peer in a collaborative narrative intervention. Contingent discourse and better topic management were more likely to occur over the course of interaction with a virtual peer than with a human peer. A study published in 2015 shows that children who author social behaviors for a virtual peer and watch the resulting interaction with real children show a transfer effect – subsequently producing more contingent discourse with their human peers.

In all cases, the hope is that this technology will push the capability of computers in the classroom above and beyond the passive digital lecture. Indeed, the creation and use of virtual peers may preserve the value of social interaction and engagement in education tech — while also teaching us about what makes us human.