The powerful new AI chatbot tool recently passed law exams in four courses at the University of Minnesota and another exam at University of Pennsylvania’s Wharton School of Business.
Professors at the University of Minnesota Law School recently graded the tests blindly to test how well ChatGPT could generate answers on exams. The bot performed on average at the level of a C+ student after completing 95 multiple-choice questions and 12 essay questions, earning a low but passing mark in all four courses.
ChatGPT performed better on a business management course exam at Wharton, where it received a B to B- grade. In a paper detailing the experiment, Professor Christian Terwiesch claimed ChatGPT did “an amazing job” when answering questions about operations management and process analysis, but struggled with more advanced prompts and made “surprising mistakes” with basic math.
These findings coincide with an increase in the number of schools and teachers voicing their worries about ChatGPT’s direct effects on students and their propensity to cheat. Even though it is still unknown how frequently students use ChatGPT and how detrimental it might actually be to learning, some educators are moving quickly to reevaluate their assignments in response to the AI tool.
Since it was made available in late November, ChatGPT has been used to generate original essays, stories and song lyrics in response to user prompts. It has written research paper summaries that deceived several scientists. Even some CEOs have used it for email writing and accounting tasks.
ChatGPT is trained on vast amounts of online data in order to respond to use queries. Although it has gained popularity among users, it has also sparked certain concerns, including those regarding errors and the possibility of perpetuating biases and spreading misinformation.
One of the University of Minnesota’s law professors, Jon Choi, told CNN that the tests’ purpose was to examine how ChatGPT might help attorneys in their work and help students in exams, regardless of their professors’ permission. This is because the questions frequently resemble the writing that attorneys do in real life.
“ChatGPT struggled with the most classic components of law school exams, such as spotting potential legal issues and deep analysis applying legal rules to the facts of a case,” Choi said. “But ChatGPT could be very helpful at producing a first draft that a student could then refine.”
He contends that the most promising use case for ChatGPT and related technology is human-AI collaboration.
“My strong hunch is that AI assistants will become standard tools for lawyers in the near future, and law schools should prepare their students for that eventuality,” Choi said.
Likewise, Wharton’s professor Terwiesch found that the chatbot was “remarkably good” at modifying its responses after being pointed out an error by a human, suggesting the potential for people to collaborate with artificial intelligence.
However, in the short term, there is still immediate uncertainty around whether and how students should use ChatGPT. For instance, public schools in New York City and Seattle have already banned students and teachers from using ChatGPT on the district’s networks and devices.
Considering ChatGPT performed above average on his exam, Terwiesch told CNN he agrees restrictions should be put in place for students while they’re taking tests: “Bans are needed. After all, when you give a medical doctor a degree, you want them to know medicine, not how to use a bot. The same holds for other skill certification, including law and business.”
Despite everything, Terwiesch believes this technology still has a place in the classroom: “If all we end up with is the same educational system as before, we have wasted an amazing opportunity that comes with ChatGPT.”
