Since the launch of ChatGPT in late 2022, millions of people have started using large language models to access knowledge. And it’s easy to understand its appeal: ask a question, get a polished summary, and move on: it seems like effortless learning.
However, a new paper I co-authored provides experimental evidence that this facility can come at a cost: When people rely on large language models to summarize information about a topic, they tend to develop more superficial knowledge about it compared to learning through a standard Google search.
Co-author Jin Ho Yun and I, both marketing professors, reported this finding in a paper based on seven studies with more than 10,000 participants. Most of the studies used the same basic paradigm: Participants were asked to learn about a topic—such as how to garden—and were randomly assigned to do so using an LLM like ChatGPT or the “old fashioned” way of browsing links through a standard Google search.
No restrictions were placed on the use of the tools; they could Google as long as they wanted and keep requesting ChatGPT if they wanted more information. Once their research was completed, they were asked to write advice to a friend about the topic based on what they had learned.
The data revealed a consistent pattern: people who learned about a topic through an LLM versus a web search felt they learned less, invested less effort in subsequently writing down their advice, and ultimately wrote shorter, less factual, and more generic advice. In turn, when this advice was presented to an independent sample of readers, who did not know which tool had been used to learn about the topic, they found the advice to be less informative, less useful, and were less likely to adopt it.
We find that these differences are robust across a variety of contexts. For example, one possible reason LLM users wrote shorter, more generic advice is simply that LLM results exposed users to less eclectic information than Google results. To control for this possibility, we conducted an experiment in which participants were exposed to an identical set of facts in the results of their Google and ChatGPT searches. Likewise, in another experiment we kept the search platform – Google – constant and varied whether participants learned from standard Google results or from Google’s AI Summary feature.
The findings confirmed that, even holding facts and platform constant, learning from synthesized LLM responses led to more superficial knowledge compared to collecting, interpreting, and synthesizing information by oneself through standard web links.
You may be interested in: Google plans to invest $40 billion in Texas data centers amid AI boom
Why is it important
Why did the use of LLMs seem to decrease learning? One of the most fundamental principles of skill development is that people learn best when they are actively engaged with the material they are trying to learn.
When we learn about a topic through Google search, we face much more “friction”: we must navigate different web links, read informative sources, and interpret and synthesize them ourselves.
Although more challenging, this friction leads to the development of a deeper and more original mental representation of the topic at hand. But with LLMs, this entire process is done on behalf of the user, transforming learning from a more active process to a passive one.
What’s next?
To be clear, we do not believe that the solution to these problems is to avoid the use of LLMs, especially given the undeniable benefit they offer in many contexts. Rather, our message is that people simply need to become smarter or more strategic users of LLMs, which starts with understanding the areas where LLMs are beneficial and where they hurt their goals.
Do you need a quick, factual answer to a question? Feel free to use your favorite AI co-pilot. But if your goal is to develop deep, generalizable knowledge in an area, relying only on LLM syntheses will be less useful.
As part of my research into the psychology of new technologies and new media, I am also interested in whether it is possible to make learning in LLM a more active process. In another experiment we tested this by having participants interact with a specialized GPT model that offered real-time web links along with their synthesized responses. However, there we found that once participants received an LLM summary, they were not motivated to delve deeper into the original sources. The result was that participants developed more superficial knowledge compared to those who used standard Google.
Building on this, in my future research I plan to study generative AI tools that impose healthy frictions for learning tasks, specifically, examining what types of barriers or obstacles most successfully motivate users to actively learn beyond easy, synthesized answers. These tools would seem especially critical in secondary education, where a major challenge for educators is how to best equip students to develop foundational reading, writing, and mathematics skills, while preparing for a real world where LLMs will likely be an integral part of their daily lives.
*Shiri Melumad is an associate professor of Marketing at the University of Pennsylvania.
This article was originally published on The Conversation
Do you use Facebook more? Follow us to always be informed
