TL;DR
Turn the school run and nap time into listening time
- Thousands of audiobooks, podcasts and originals
- Listen on your phone, tablet or Echo — also offline
- Cancel anytime
A randomized trial involving 1,179 students in 10 U.K. public schools found that students receiving AI or human tutoring answered follow-up math questions correctly more often than students receiving standard practice. After 10 weeks, however, tutoring did not improve scores on a standardized math test, and tentative subgroup findings suggested girls may have benefited while tutored boys did worse. Researchers also reported occasional AI math errors and said one in five students never replied to the tutor.
Eedi and Google DeepMind reported results from a randomized trial of 1,179 students at 10 U.K. public schools, finding that an AI math tutor performed about as well as expert human tutors on immediate follow-up questions. After 10 weeks, however, neither form of tutoring improved scores on a broad standardized math test compared with standard practice, leaving the study with a more mixed result than a simple comparison of tutor performance suggests.
From April through July, secondary school students used Eedi’s math platform for one class period a week, lasting about 30 to 35 minutes, with a teacher present. After a student answered incorrectly, the platform assigned them to one of four conditions for the duration of the study: standard in-app remediation, text chat with an expert human tutor, chat with an AI tutor whose messages a human checked, or an AI tutor given a summary of the student’s past misconceptions and work habits.
On similar follow-up questions, students in the tutoring groups answered correctly 56% to 58% of the time, compared with 43% after standard remediation. Eedi Chief Impact Officer Bibi Groot said the AI and human tutors were “basically on par.” The study’s standardized-test results told a different story: among the 916 students who took both Star Maths tests, AI tutoring did not outperform static practice. Human-tutored students scored slightly lower than the other groups, which researchers said could relate to covering less material while exploring topics in greater depth.
The researchers also examined gender and safety findings. On Star Maths, tutored girls tended to score slightly better than girls receiving static practice, while tutored boys scored worse, particularly those assigned human tutors. The authors described these subgroup findings as tentative. Human reviewers approved 94.7% of AI draft messages without changes; researchers found 11 messages containing false math statements, seven of which reached students. The source report also says one in five students never replied to the tutor, though it does not provide a detailed breakdown of that finding in the supplied material.
Immediate Help Did Not Raise Test Scores
The results distinguish between solving the next problem and improving performance on a wider assessment. AI and human tutoring were associated with better answers to similar follow-up questions than standard remediation, but that advantage did not appear on Star Maths after 10 weeks. Schools weighing tutoring tools therefore have evidence of a short-term instructional benefit, but this trial does not show a corresponding gain on the broad test it used.
Cost and staffing are part of the question. Groot said human tutors are expensive and that having experts review every AI interaction is not a scalable arrangement. The study tested AI with a human checking its messages, so its findings do not establish how well a less-supervised system would perform in everyday school use. The reported math errors and the unanswered student messages also point to practical issues that matter if schools consider using such tools at scale.
The possible difference by gender merits further study, not a firm conclusion. Researchers suggested that being prompted into a conversation after each mistake could feel discouraging to some boys, and that text chat may favor students more comfortable with reading and writing. Those are proposed explanations, not demonstrated causes, and the reported subgroup pattern is tentative.
Products Worth Considering
LEGO Art Hokusai The Great Wave Building Set (31208)
LEGO Icons Tranquil Garden Building Set – Japanese Zen Garden DIY Kit for Adults, 10315
Graco TriRide 3-in-1 Convertible Car Seat, Kipling
As an affiliate, we earn on qualifying purchases.
How This Trial Expanded Earlier Research
The new trial follows a smaller study reported the previous December, in which Eedi said its human-supervised AI tutor slightly outperformed human tutors on new problems in later topics: 66.2% compared with 60.7%. The larger trial found the AI and human groups closer together on follow-up questions, and it added a standardized test administered before and after the intervention.
The design also compared two AI setups. One provided the system with a summary of each student’s previous misconceptions and work habits; the other did not. According to Groot, the extra personal context did not meaningfully improve outcomes, possibly because the AI already received information about the misconception behind the student’s incorrect answer. The trial found no clear differences by family income or age in the reported test results. Researchers also said lower-achieving students assigned human tutors scored lower than comparable students receiving standard practice, possibly because they covered less material.
These findings concern one platform, one term-long trial, and the particular tutoring and assessment conditions used. They do not establish that all AI tutors have the same performance or that results would carry over to different subjects, schools, or longer periods.
“The AI and human tutor are basically on par.”
— Bibi Groot, Eedi’s chief impact officer
Questions About Impact and Safeguards
The source report does not provide confidence intervals or enough statistical detail to determine how robust the subgroup patterns are. Researchers characterized the gender findings as tentative; the trial does not establish why tutored boys scored lower or whether the pattern would persist in other groups of students. The report also does not show that the tutoring caused the difference.
It remains unclear whether the absence of Star Maths gains reflects the 10-week duration, the amount of time students spent tutoring, the test’s scope, or another feature of the intervention. Researchers suggested that students in tutoring groups may have covered less material, but the supplied account does not establish that as the explanation for the overall flat results. The study also found false math statements in AI drafts, with seven reaching students, while human tutors’ messages contained seven math errors as well; the report does not provide enough detail to compare the severity or consequences of those errors.
The source says that one in five students never replied, but does not specify the denominator, how nonresponses were distributed across study groups, or whether they affected outcomes. It also does not report the full results of the company’s later, larger trial.
A Larger Trial Tests Direct AI Use
Eedi began another trial in September with more than 15,000 students, according to Groot. In that study, the AI interacts directly with students, with human support used if conversations go off track. This differs from the reported trial, where a human approved each AI message, and may offer evidence about a less labor-intensive model.
Eedi said it would release the current study’s data on Thursday, including subgroup results, conversation-level analysis, and findings the company did not expect. Those materials could help researchers examine the tentative gender patterns, the frequency and handling of errors, and the students who did not respond. The supplied report does not give a publication date for the full dataset or a timeline for results from the larger trial.
Key Questions
Did the AI tutor outperform human tutors?
Not clearly. Students assigned to AI and human tutoring answered similar follow-up questions correctly at comparable rates: 56% to 58%. The study reported no broad standardized-test gain from AI tutoring over standard practice.
Did tutoring improve students’ standardized math scores?
The trial found no improvement in Star Maths scores from AI tutoring compared with standard practice after 10 weeks. Students assigned human tutors scored slightly lower than the AI and standard-practice groups, according to the report.
What did the study find about girls and boys?
Girls who received AI or human tutoring tended to do slightly better on Star Maths than girls receiving static practice, while tutored boys did worse, especially those assigned human tutors. Researchers called these findings tentative; the study did not establish a cause.
Was the AI tutor error-free?
No. Human reviewers approved 94.7% of AI drafts without changes, but researchers found 11 AI messages with false math statements, seven of which reached students. Human tutors’ messages also contained seven math errors, according to the report.
What is being tested next?
Eedi began a separate trial with more than 15,000 students, in which the AI communicates directly with students and humans are brought in if conversations go off track. Results and a timeline for their release were not provided in the source report.
Source: the74million.org
