Our guest blogger for this post is Dr. Steve Watson, associate professor at the Faculty of Education, University of Cambridge, and a fellow at Wolfson College, Cambridge.

Learn more about Steve at the bottom of this page.

Capabilities and limitations of ChatGPT

I have been experimenting with ChatGPT in my work since its release last year. At the same time, it prompted me to reflect on how the technology works. In this blog, I want to illustrate the use of ChatGPT by outlining some of the uses I have put ChatGPT to in my teaching and research; what I have learned from thinking about the use of this technology in teaching and research practice; and finding practical ways for it to contribute to my work, as well as thinking about how the technology works as I engage with it. While I have spent a lot of time exploring ChatGPT since its release last year, I have realized that this technology had many potential applications for my teaching and research, and I wanted to realize these possibilities. However, ChatGPT is not an out-of-the-box solution. Like any technology, it takes time to learn how to use it effectively and understand its capabilities and limitations.

While this provides a simplified overview of deep learning, it makes the point that ChatGPT is not looking up or referencing information sources but is based on its training of publicly available text on the internet (around 300 billion words) with a statistical model that involves some 195 billion parameters. This is really about the probability of the next word based on recognizing the context. Within these structures, there is knowledge and information, but there is also the possibility of factual inaccuracy or misinformation. This is especially pronounced when the input prompt is open-ended, like, for instance, “what is the meaning of life?” or “how much will it cost to repair my roof?” ChatGPT is concerned with structure and form rather than factuality.

And no doubt, like many others, I began by asking fairly banal questions as if I was typing into a search engine but expecting a considered and factual response from the AI. This was disappointing because while the output text was logically, grammatically, and stylistically consistent, there were often things that were simply untrue. Having reflected on how the technology actually works, I began to work in a different way with the technology, and this is where I have had the most promising results. Not that ChatGPT really ‘creates’ anything for me; it simply assists me in developing richer interpretations through a closer and more thorough assisted reading of a text.

Use cases – automated assistive reading and writing

I get pieces of text sent to me by students and colleagues. I receive texts from prospective PhD students, other academics, and as an editor of a journal. This can be a problem in that I don’t have the time to always engage in a meaningful way with all that is sent to me. So, I began experimenting by inputting some of the anonymized texts I received into ChatGPT.

I have tried different approaches to assessment and even the analysis of interview transcripts using ChatGPT. The capability to summarize is clear, but asking questions about the forms and structures of the text is possible and leads to some good results. However, this is not in a reliably automated way and requires manual checking for fidelity and plausibility. Nonetheless, this is certainly more thorough than what I could achieve manually. In this sense, ChatGPT becomes a reading assistant that helps me clarify the questions and feedback that I want to respond to the author with.

Using ChatGPT as a writing assistant

As a writing assistant, understanding the importance of prompt engineering in ChatGPT has helped me generate content in considered ways. There are two main ways in which I use ChatGPT. The first one is when I have a few ideas, and I am not sure how to structure them. I can type something like “write three paragraphs that explain/describe/argue the following…” and then list the ideas. The output from this is treated as an iteration, which provides me with a structure and form that I can then adapt and develop manually. I may also input this output again into ChatGPT to consider different structures, wordings, and phrasings. I have realized that writing iteratively, i.e., drafting and redrafting, is something I do manually. ChatGPT simply automates some of these iterations. Effectively, it can help me overcome obstacles to writing, such as writer’s block.

The second way I use it is when I am writing more fluidly, but probably not so fluently. Ideas and words are flowing, the articulation of the ideas is good, but I feel I can lose a sense of structure, or the overall argument becomes lost to me. I then input the text and ask ChatGPT to describe the argument and structure and/or the claims and assumptions that I am making. This can be very revealing, as it becomes much more apparent what I am trying to say and how the structure might be improved. In terms of the use of language, both formal and informal emergent grammar are recognized. Therefore, feedback on presentation is pragmatic rather than pedantic.

As a writing assistant, then, I feel ChatGPT deals with issues of structure and form, which frees me up to address the more nuanced and conceptual aspects of the content, knowing that there is going to be a logical organization with ‘i’s and ‘t’s dotted and crossed appropriately.

Conclusion

While ChatGPT has limitations, it has the potential to be a useful tool for teaching and research. By experimenting with different use cases, I have found practical ways for ChatGPT to contribute to my work. One of the key elements of this is recognizing the importance of the prompt so that the technology is clear about what you want it to do. This is being referred to as ‘prompt engineering’. As a writing assistant, ChatGPT deals with issues of structure and form, which frees me up to address the more nuanced and conceptual aspects of the content. While it is important to be aware of the technology’s limitations, ChatGPT can be a powerful tool in both teaching and research.

My next step is to look into developing apps based on ChatGPT that refine and automate some of the practices I have developed through this early experimentation. Such apps could help users quickly and accurately summarize large amounts of text, generate outlines or drafts for written pieces, and analyze complex interview transcripts, among other things. However, as with any technology, it is crucial to exercise caution and critically evaluate the output to ensure the accuracy and reliability of the results.

In summary, I believe that ChatGPT has the potential to be a valuable tool for researchers and educators alike, and I am excited to continue exploring its applications and potential.