Automated marking performs well for constrained response items such as multiple choice and short answer questions, but reliably assessing extended responses and questions involving multimedia remains challenging. Recent advances in computer vision and large language models (LLMs) now enable richer semantic and multimodal understanding, offering new possibilities for more transparent and trustworthy automation.

The EPSRC supported Prosperity project, undertaken in collaboration between AQA and King’s College London, aims to develop a virtual AI examiner assistant founded on a modular architecture. The project aims to provide a reliable and transparent baseline for quality assurance, while also modelling examiner cognition to identify where AI can assist rather than replace human judgement. Drawing on principles of Human Centered AI and distributed cognition, we explore how expert markers attend to, interpret, and integrate evidence across the marking process.

We will present early conceptual models for the virtual examiner assistant and preliminary findings from a behavioural eye gaze study with experienced markers. These findings inform the design of our modular AI system and guide the development of support tools that reduce ambiguity and cognitive load in high stakes decision making. Overall, Prosperity seeks to augment examiners by improving consistency, limiting bias, and enhancing transparency while preserving human agency and oversight.

Computer-based assessments offer unprecedented opportunities to capture rich, granular log data through human–machine interactions, providing deeper insights into test-takers’ strategies and behaviors. This multidimensional sequential log data, encompassing actions, time intervals, eye-tracking, and more, poses challenges for traditional unidimensional sequence models. This talk presents an overview of sequence mining techniques applied to unstructured process data, with the goal of advancing item design and measurement frameworks.

Two case studies are highlighted to illustrate the application of sequence mining in educational assessments. The first explores missing response patterns in a computational thinking assessment using Dynamic Time Warping (DTW) method, identifying potential causes of nonresponse and predicting missing values based on process data. The second investigates behavioral features of resilient students in scientific inquiry tasks, applying the Time-Warped Longest Common Subsequence (T-WLCS), a method originally developed for musical retrieval—to uncover interpretable patterns in student interactions.

The presentation will also discuss the broader potential of integrating AI techniques with log data in large-scale assessments, exploring opportunities for more adaptive, inclusive, and behaviorally informed measurement systems.

Assessment theory and practice have developed in relation to particular contexts of practice, and with technological modes.  In the context of new and emergent digital technologies, including sensor-based, process oriented, interactive and adaptive designs, and with computational psychometrics and artificial intelligence we ask: to what extent to the axioms of assessment theory and organising concepts and principles that used to underpin assessment design and validity practice hold in relation to emergent, and future assessment designs and assessment practice?

To answer we draw on methods and concepts in anticipatory futures thinking, and the influential field of French anthropology of technology (Andre Leroi-Gournan 1943, Gilbert Simondon, 2017 Bernard Steigler, 1994). The phrase – ‘weak signals and wild cards’ – indicates both the relevance of anticipatory futures think though. It imagines and considers radically different assessment possibilities and scenarios in rapidly changing and highly uncertain technological, ecological and political contexts and the increased significance of digital ‘sensor’ technologies.