An exploration of writing, grading, and AI scoring
Writing Solutions and Auto Scoring
This application allows the instructor to create writing assignments with well-managed scoring rubrics. Special features that were added include the ability for peers to review each other’s work, and an adaptive AI scoring tool that learns an instructor’s method of scoring and then uses that pattern to auto-score papers in the future.
One early challenge in creating this application was streamlining the assignment creation form. It had previously been a much bulkier and complex form which was not user friendly, and made the task of assignment creation a hassle. It is now intuitive and simple.
This is the dashboard view of a learner’s Writing Solutions assignments in a date / importance sorted view. It’s been tailored to give the learner a view into what is most pressing, what is coming up, and what’s already been achieved. The approach was to greet the user with a tailored snapshot of their world within the app. It could easily be expanded to accommodate other deadlines from other applications.
Writing Solutions has many features to provide extremely detailed feedback to a learner about his or her work. Here, individual comments can be added to issues that have already been flagged by the auto-scoring system.
Each assignment has scoring based on rubrics that can be set when the assignment is created. Under the Rubric Scoring tab, the pre-set rubrics function in two ways - they provide detailed instructions about how the score should be applied, and an affordance for choosing a score. Changes in these “sub scores” impact the overall score at the top. This type of scoring involved careful consideration of where to put all the mechanics of the rubrics. They are complex, and involve a lot of consideration for the grader and for the designer who makes the form. Keeping them to one panel on the right, although visually dense, is key to understanding how they all relate toward the final score.
An important experimental feature of Writing Solutions is the AI Scoring functionality. This is a dashboard view of the AI scoring as it is in progress. The function requires that the system “watches and learns” from the instructor as he or she grades papers. After a certain pre-set amount of papers has been analyzed by the system, it is now ready to start auto-grading papers based the patterns it has observed from the instructor.
The AI scoring tool is something that has undergone extensive testing, since there are still some psychological barriers to machine learning and AI functions. However, instructors often voice their concern about the amount of time it takes to grade papers, particularly in a very large class, and so they are looking for solutions that will help them do this faster. Our findings show that opinions about AI and learning are still evolving.
This is a view of a completed assignment in Writing Solutions just before it is ready to be submitted by the student for feedback. It’s important to point out many design decisions: The fidelity of the right rail is preserved in both the student and instructor views. The same right rail is cleared of any extraneous data when it's not in use. Status of the assignment is limited to one message at the top (in this case the status of the assignment is “Open.”) Editing functions for the copy are limited to a simple componentized text editing tool.
This video walks us through the creation of an assignment in the latest version of the form made for Writing Solutions. The toggles for using different auto grading options are also discussed.