Raising Australian Writing Outcomes: A Way Forward
Effectiveness of AI-supported Educational Programme: a Four-Year Study
One of the research principles at Writer’s Toolbox is that our work is evidence backed. This means we test the validity and reliability of our work in the field. So, we know if our AI and our approach to writing instruction is really making a difference. We believe that ideas must be backed by evidence, and that applies to us, first. Consequently, a team of researchers at Writer's Toolbox have been monitoring the longitudinal impact of the Writer's Toolbox writing programme. This study has been in place for over four years—monitoring the impact on schools and students using our AI technology to affect educational outcomes.
Robust research. Empirical evidence.
Education is rich with research. And in education research many studies will monitor the impacts in a classroom over a ten-week period. Or a year level group over a semester, or perhaps over a year. Few educational research projects will monitor the impact of educational interventions over several years, let alone dozens of schools. This is what we’ve done. Our longitudinal educational impact study has been underway now for four years, and involves 71 schools in the Australian state of Queensland, and approximately 79,000 students.
What’s the question?
Our hypothesis is simple: Could the Writer’s Toolbox programme using AI-supported learning trigger an obvious lift in writing improvement across an entire schooling system? This question could then be broken down into several research aims. The first was to see if such a system was robust across an entire eco-system of schools, rather than just one or two pilot schools. Secondly, could this approach work in schools of any type or socio-economic status? Thirdly, and related to this: could the system work across all age levels? That is: could we build a system that would work with students of any age or ability, and work to improve any type of writing? Finally, could we prove any outcome statistically? So we would know—using confidence intervals—that the results were not down to chance or a person, but something significant had occurred.
Method and approach
Teachers attended professional development workshops where they were trained in the Writer's Toolbox writing system. Here, they learned how to use the AI-supported writing programme. They also received instruction in explicit writing methods from the Writer’s Toolbox programme, including sentence styles, paragraph models, and composition skills including fluency, expansion, precision, thematic development, and structure.
Teachers then integrated these methods into their classroom practice, and had coaches from Writer’s Toolbox come and support them in their schools. At the same time, school leadership were advised on the effective integration of the Writer's Toolbox writing system into their school. In their daily writing, students used the online tool, which gave students immediate feedback on writing quality and improvement, and differentiated experiences according to their age and writing ability. In addition, teachers used real time writing data to make educational decisions in their school. The results were tracked at an individual, school, and cohort level.
Engagement, confidence, and better writing quality
We couldn't be more excited by the results
Teachers reported higher levels of student engagement, student confidence, and better writing quality. As a result, they also reported less time spent marking. In school internal and external examinations, schools in the study reported higher outcomes and increased performance in writing related tasks. In addition, anecdotally, teachers reported they felt higher levels of capability teaching writing.
At least double the state NAPLAN results
NAPLAN writing data was also used as an objective measure of school performance and value add in writing performance. The schools in the study all had students undertake the NAPLAN standardised writing test each year at Years 3, 5, 7, and 9. The results were available by school and collated. At every year level, schools using the Writer's Toolbox AI-supported writing programme outperformed the state of Queensland.
Across the 70-plus schools, writing improvement growth was at least double the state in every year level. In some year levels, rate of growth was five times the state. Perhaps the most profound result was in boys’ writing. Year 9 boys saw a rate of writing improvement 9.6 times greater than the state. And at all year levels the results were statistically significant, using a P value less than 10 per cent.
