A few months ago, DEFI and CEDiR hosted a webinar that I was part of entitled Designing and Facilitating Effective Online and Blended Teacher Professional Development: A Discussion. With a continued focus in the sector on identifying ingredients for successful technology-enhanced TPD courses, I’ve found myself reflecting on some of the areas of our discussion. The webinar classically felt far too short to respond to all of the excellent questions posed by participants and my fellow panellists, and so I thought I would revisit the discussion through this post, which I hope will stoke further conversation and reflection. Please keep hold of your comments (especially the provocative ones! My favourite!) because you’re very welcome to share your opinions, insights and experiences in response to this blog post.

The full webinar is available here if you were unable to attend and would like to view it. The panellists were Farah Ahmed, Sara Hennessy and me, and you can find links to our research projects at the bottom of this post that we drew on throughout the webinar. A few main threads came up during our discussion, which are worth thinking through further. I’ve grouped these thematically, but note that this is not an exhaustive list of considerations for designing and facilitating technology-enhanced TPD. Rather, these are areas that I’ve been reflecting on since our webinar and that I believe are worth highlighting.

Don’t lead with the tech: First (and I always like to mention this first), technology is not a silver bullet for enacting change. Providing teachers with access to an online learning platform alone will not result in an improvement in their students’ learning outcomes. Pedagogy should come first; the needs of the community for whom the technology is being provided should come first.

The actual technology including all those fancy bits of functionality (although they’re fun and should indeed be leveraged!) should come last. Technology should also not seek to replace in-person alternatives to learning as they will likely have different objectives that need to be articulated before the selection of a format or platform. And regardless of the platform (whether it’s online, blended, in-person or a variation therein), the tech should be selected according to the context and in conversation with potential participants.

Buy-in is hard to achieve online, but it’s not impossible. I’m thinking about buy-in in two different ways: buy-in for the pedagogy the course strives to impart, and buy-in for engaging in and completing the online course itself. Regarding buy-in for the pedagogy, courses can’t assume that everyone who registers is automatically sold on the potential impact of the content.

I found particular success through using reflective activities for participants to have a structured space to assess their practice and the ways in which the course content may apply (more on these activities and others in a theme below).

I also found success in having local facilitators as pedagogical champions. The first Massive Open Online Course (MOOC) I designed was a space for local facilitators, which really bridges that online/offline divide. By facilitators, I refer to practitioners who take on a leadership role in their setting whereby they coordinate and support their colleagues in developing their practice.

The course provided guidance and tools for achieving buy-in with their colleagues, as well as opportunities for them to share successes and challenges with their course peers. Having a colleague describe the pedagogy, and being able to do it in a contextualised way according to the setting, is much more likely to result in its uptake than if I (a person they’ve never met who isn’t from the same setting) tried to convince them that dialogic pedagogy is something they should spend their limited time learning about and using in their practice.

Regarding buy-in for the engagement and completion of the MOOC itself, the inclusion of participant voice in the course design goes a long way to help with this, as does meaningful accreditation, and having opportunities to connect with other practitioners globally.

Collaborate with key stakeholders throughout all phases of the intervention: One seminar audience member asked how we ensure that teacher needs remain front and centre in our course design and facilitation. I am unwavering in my commitment to the use of user centred design and collaboration with participants and key sector stakeholders from the conception of a course.

Who this includes will of course depend on the course, but it would likely include learners (i.e. potential participants), community members, government stakeholders and other course providers. Regardless, this collaboration should continue through the different iterations of the intervention and beyond.

Position access considerations centrally in the design, facilitation and evaluation of the course: This comes from a design principle in our research, which asks who the participants really are and whether there are any differences in their ability to meaningfully participate and engage in the course. By access, I’m referring to a number of areas, which should be unpacked separately during the design, facilitation and evaluation of the course.

It can refer to electricity, internet connectivity, device availability, gender, disability, affordability, digital skills and prior experience in online learning, and prior experience with the subject matter to name a few. Doing an in-depth context analysis and utilising user-centred design through collaborating with potential participants from the outset are necessary practices.

Success has also been seen in using technology that participants already use and have access to (e.g. use WhatsApp instead of creating a new discussion-based platform if that’s what is already used by teachers in the community), which is particularly relevant for blended programmes.

Create opportunities for reflection, creation, and contextualisation: If I was asked for content design considerations that enhance impact to teacher practice, these are the three areas I would highlight:

(1) Include reflective activities throughout the course, where possible – e.g. videos, weekly tasks, discussion forum contributions. Through reflecting (both independently and collectively) on the materials, practitioners will be better equipped to apply the theory to their settings and daily practice.

(2) I found particular success through including opportunities for participants to create and share artefacts. This can take different forms but examples from my course series include reflective reports, inquiry objectives and findings, and facilitation plans and models. This sought to enhance the relevance of the content, the ability to meaningfully connect with other participants, and accountability.

(3) Having an adaptive and responsive course to participant needs, including resources that can easily be contextualised (with guidance on how to do so) goes a long way to sustaining impact. This means that international course content can be made relevant to local contexts, for example through offering a range of scaffolding or support; providing a library with additional open access resources; and/or ensuring that the monitoring and evaluation process collects accurate and meaningful information from participants in order for positive adaptations to be made. 

These strategies additionally address a question from a seminar audience member who asked how we can make research accessible and digestible for teachers who are already under pressure in terms of time and workload. Research is perhaps most digestible when there are opportunities to contextualise it to their own practice through reflection, and when they can engage in a dialogue with other practitioners about the subject area.

Build in monitoring and evaluation mechanisms during the design of the course. Feedback needs to be systematic and built-in to the course structure while also allowing for its own adaptations where required, which is one way to ensure the meaningful inclusion of participant voices. I would suggest a mixed methods approach for this, leveraging technology to collect data on user analytics (e.g. scroll maps that indicate where participants spend the most/least amount of time on the course), as well as collecting more specific demographic and engagement data via questionnaires and interviews or focus group discussions if appropriate and possible. I would also suggest that EdTech interventions need more evidence on access-related constraints and opportunities in particular. Related to the collection of data, careful considerations need to be made regarding data protection and the collection of personal data.

As always, I invite your feedback and comments regarding these reflections! You can respond below this post, or you can also get in touch with me directly (see email address included below).

Sincere and very big thank yous to Dr Alison Twiner at CEDiR and Jude Hannam at DEFI for their support! Note that this blog post is also hosted on the Cambridge Teacher Research Exchange – Camtree – website. This theme of designing for effective technology-enhanced TPD will be revisited in future webinars so be sure to look out for the next one!

Links for further exploration:

Digital Education Futures Initiative (DEFI)

DEFI’s vision is to create a more inclusive global education system by building a community that offers a space for open dialogue, research, knowledge creation, reflection and thought leadership on digital education’s role in shaping a sustainable future.

To receive notification of similar events both online and in-person, please sign up to DEFI’s mailing list at Home – DEFI

Camtree

Camtree provides a global platform for close-to-practice research in education to improve teaching and lead to better outcomes for learners worldwide. It is the world’s first platform that helps teachers and educational leaders both to improve learning and share their knowledge.

To find out more visit Home – Camtree

Alina von Davier is a keynote speaker at DEFI’s annual event which this year is titled: AI-Powered Pedagogy: Designing the Future of Education.

As Chief of Research at Duolingo English Test, she and her team are pioneering new assessment methods.

Here she tells us a little about her work. 

If you want to hear more from her, consider signing up to attend our in-person event on Tuesday 26^th March.

Digital-first Assessments in the Age of Artificial Intelligence

By Dr. Alina von Davier

Recent advances in artificial intelligence (AI) and intelligent automation have revolutionized educational assessments, with a focus on scalable content generation. AI-powered content generation allows the development of diverse test items and personalized assessments, making high-quality assessments more affordable and accessible. However, a careful balance is needed between technological progress and human-centered design. This approach emphasizes the need for interdisciplinary collaboration and a human-centered AI framework that integrates AI with human expertise to create assessments that adapt to test-takers’ needs. The Duolingo English Test (DET), a research-based digital-first, high-stakes language test, is a trailblazer in the thoughtful adoption of AI throughout the test development and administration processes.

We believe that technology in general, and AI in particular, is the best way to scale education and level the playing field for all, regardless of their circumstances. It can help us create educational and testing content quickly, and curate and recommend resources for learners that are at a specific difficulty level given a learner’s ability. In the context of assessment, AI and computational psychometrics create adaptive test experiences that hone in on a test taker’s proficiency more efficiently than previously possible, making for a quicker, more streamlined testing experience while adhering to the high standards of reliability and validity for high-stakes tests.

We launched the DET as an extension of our mission to break down barriers to education. Our co-founders, Luis von Ahn and Severin Hacker, both immigrated to the US and have experienced first hand the arduous and expensive process of certifying their English proficiency. Duolingo was already using technology expertise to make language learning accessible for people across the globe, and we sought to apply some of the same techniques to language assessment.

Interactive tests for interactive skills

As a research-based assessment, our first priority is the theoretical alignment of the test to frameworks such as the Common European Framework of Reference for Languages (CEFR, 2020). This is achieved by our language experts designing the task types using an extended evidence-centered design framework (Arieli-Attali et al., 2019, Mislevy et al., 2014). 

AI comes into play next helping us generate and review interactive items and pilot new items that can further improve the quality of the test and the DET’s Test Taker Experience, or TTX. As a digital-first assessment, we embrace new technology to enhance how we measure language ability. Our latest items use generative AI to simulate real-time conversations and writing tasks, allowing us to test language skills, including interactional competence (see Galaczi & Taylor, 2018), like never before in the assessment industry.

A digital environment allows for interactivity, which allows us to build more authentic tasks, that is, tasks whose characteristics correspond to relevant activities in the real world (Bachman & Palmer, 2010). Last year, we introduced Interactive Listening, which puts test takers in a chat with an animated conversation partner. They then conduct a multi-turn conversation to achieve a certain communication goal, such as following up with a professor for more information on a topic from class, or asking a friend to review a paper. Such interactions align with the CEFR in terms of the listening purposes of gist, specific details, and rhetorical purpose, and can be seen to assess underlying listening purposes; test takers must use top-down and bottom-up listening processes to understand the individual utterances, as well as interactional competencies to manage the topic and turn taking.

In addition to assessing the skill of listening in isolation, the Interactive Listening task, like many DET tasks, measures multiple language skills at once—which is how they’re used in real life! By listening and responding to a conversation partner, test takers demonstrate conversation skills (integrated listening and speaking), not just comprehension. In addition, the follow-up summary writing task further allows test takers to demonstrate comprehension of the conversation while simultaneously producing a written response. This type of integrated listening/writing skill aligns with ‘Mediation’ in the current CEFR.

Optimizing test development with human in the loop AI

After the items have been designed and pre-tested, we use the latest machine learning and software engineering technology to automatically generate test content, making room for more innovation in the test development process. While the LLMs are able to produce natural, coherent texts on a wide range of topics, in virtually any genre, we ensure that each item is reviewed by human experts, for quality, fairness and bias, as well as for factual accuracy.

For years, the only way to generate items according to the specifications and design was for teams of expert test developers to write them. This process involves people finding source material, researching, and of course writing, all of which takes time and money. And those costs are then passed on to test takers, which is why most high-stakes English proficiency tests cost several hundred dollars.

Instead of spending months writing items by hand, our test developers use a human-in-the-loop AI process to generate a far greater range of content, much faster! We then filter, edit, and review AI-generated content to produce test items that are indistinguishable from something written by actual humans. This ensures that we have a wide variety of content on the test—at a volume necessary to support our computer adaptive test format.

CATs are nimble, CATs are quick

Computer adaptive tests (CATs) are efficient assessments that match the items’ difficulty to each test taker’s performance. These tests necessitate a flexible delivery platform, an item selection algorithm based on psychometric models, and a large item bank (see, for example,  Magis, Yan & von Davier, 2017).  Our approach to automatic item generation ensures that we have a wide variety of appropriate content on the test at a volume necessary to support our computer adaptive test format. Leveraging AI to create a large pool of items is what allows us to administer the test adaptively at scale—another innovation for the test-taker experience, because it means both that the exam can be completed in about an hour, and it can be taken at home.

In a traditional fixed-form exam, test takers are given questions at every proficiency level, regardless of their own proficiency. Not only are these test items demoralizing for the test taker, they are also not “informative”—that is, their response to the item doesn’t help much to estimate their true proficiency.

We designed the DET to quickly adapt to a test taker’s learning level, withholding content that is likely too difficult or too easy, not only because this is a better experience for the test taker but because it’s a more efficient way to assess their proficiency. Because they encounter fewer items far above or below their proficiency level, test takers may find that the test feels less stressful, and perhaps easier, than a longer, fixed-form exam.

Adaptively administering test items also enhances the security of our test: because each test session is uniquely administered and draws from such a large pool of items, it’s extremely difficult for test takers to take advantage of leaked items. In addition, each test session records the webcam, microphone, keyboard, and cursor actions, and is first reviewed by our AI algorithm for potential signs of rule breaking or malicious behavior before it is reviewed by human proctors. These proctors are language experts who use the AI flags and established proctoring guidelines to determine whether any rules have been broken, before arriving at a certification decision.

AI must be used responsibly

The element that unites this entire ecosystem is the test-taker experience, or TTX. TTX is distinct from UX, or user experience; while UX is typically associated with design elements related to visuals and navigation in digital platforms, we use TTX to refer to the full test-taker experience, from test administration to score reporting.

We aim to provide a positive TTX at every stage of the test taker’s journey, including free test-readiness resources, a more affordable price point, intuitive UX design, shorter testing time, and fast score turnaround processes. Fairness plays a role, too: a test taker’s experience can vary across culture, first language, and computer setup. We consider these and other factors when developing test items, and when running quality control evaluations on the exam, to ensure that the test is fair for everyone.

As leaders in AI, we feel an obligation to set the standard. The most recent standards of the assessment community were published in 2014 and do not include guidelines to how AI may be used in assessments. That’s why we developed new Responsible AI standards (Burstein, 2023) to ensure accountability, transparency, fairness, privacy, and security in testing. By sharing them with the world, we hope to continue to lead the way in digital-first assessment.

Human expertise + AI’s power = a winning combination

From item generation to proctoring, we use human experts and machines in places they both perform best, always leaving final decisions to the human experts. And we’re not trying to minimize our need for human collaboration and innovation—rather, we’re using the technology to supplement all of the hard work our test creators do! By streamlining the test development process, we’re able to offer a faster, more innovative test, at a much more affordable price point, making it possible for more people to access high-stakes testing.