Detecting Artificial Intelligence (AI)

Can Turnitin detect AI-produced writing?

Turnitin will release a preview of their AI-detection tools from April 4, 2023 through January 1, 2024. These two videos explain what to expect from the tool and how it will handle false positives.

Sneak preview of Turnitin’s AI writing and ChatGPT detection capability for Education

Understanding false positives within Turnitin’s AI writing detection capabilities

Additional information and resources from Turnitin on catching AI-based writing are also available.

Because the technology is changing so rapidly, and this is a preview, we would like to learn from your experiences and hear your concerns. Please send them to teachingtools@umsystem.edu. Please add "AI Detector" to the subject line, that will help us track the response to the tool.

What is an AI detector?

These are applications that predict the likelihood that writing was created by an AI or a human. Typically, these look at characteristics of the writing, particularly how random it is. Humans tend to use a greater variety of words, have more randomness in their spelling, grammar, and syntax, and generally more complexity than an AI. Some will give a verbal or graphical indication of how strongly it finds the text to be from a human or an AI. Others return results in terms of perplexity (a measure of randomness) and burstiness (a measure of randomness between sentences) with scores, graphs, or color coding. Lower perplexity and burstiness scores are more likely to be from an AI, with higher ones pointing toward human authorship. 

Another technique some experts are advocating is feeding suspect passages back to the AI, (such as ChatGPT) and asking if it was written by a human or an AI. 

Are AI detectors reliable? 

No, at best they are indicative. Published claims to reliability vary greatly, between about 26 and 80%. In other words, expect them to be wrong between a fifth and three-quarters of the time. Those figures apply to the free detectors already available in early 2023. We do not have data on Turnitin currently (March 21, 2023).  

It is possible they will improve, but this should be viewed as an arms race rather than a stable situation. For instance, recent advances with ChatGPT (particularly using GPT-4) show that it is possible to coach it to write with more complexity and fluency, making it harder to detect. This is particularly true of students using well-engineered prompts. By prompt engineering, we mean the creation of fully developed questions and instructions (sometimes including data) to elicit the desired kind of results from the AI. 

If you are going to use these tools, we recommend that you check your sample with more than one.

Are there privacy or other issues? 

Free AI detectors have not been vetted by the University. This is strictly a case of use at your own risk for now. You should never feed them any content allowing identification of the student. We also do not know the specifics of if or how they store or use content. The same applies to feeding text from student papers back into an AI for evaluation.  

Many faculty will enter parts of student papers into Google or other search engines to try to find matches, so this may not seem so different to you. You may wish to consider the differences between that and pasting or uploading all (or large parts of) a paper into an application. Beyond privacy, there may be questions of student copyright to consider. 

Because we have a license and agreements with Turnitin that covers FERPA and meet the University's interpretation of student copyright, these considerations should not apply to the new Turnitin AI tool. 

What should I look for when reading a paper? 

There are also telltale signs to look for, though, again, as the technology evolves, these may change. This is based on ChatGPT.

• Look at the complexity and repetitiveness of the writing. AI's are more likely to write less complex sentences and to repeat words and phrases more often. 
• If the assignment has a bibliography, look for made-up or mixed-up entries. Some will be obviously odd, such as having authors writing an article after their death, or a German writing in English, but publishing in a German publication. One of the simplest things, if it provides URLs (particularly with DOI references), try them. In one test we tried, of eight DOI references, two pointed to other articles and six were nonexistent. 
• Look for egregious factual errors. At least on some subjects, ChatGPT will insert information that is flatly impossible. While students might do this, in combination with other factors, the errors are often things that are unlikely to be made by a human. Remember, AIs do not understand what they are writing. The phrase "stochastic parrots" is often used to describe them, as they work out what words are most likely to follow other words and string them together. 
• Look for grammatical, syntax, and spelling errors. These are more likely to be mistakes a human author would make. 
• If you have a writing sample from the student that you know is authentic, compare the style, usage, etc. to see if they match up or vary considerably. 
• Does the paper refer directly to or quote the textbook or instructor? The AI is unlikely to have textbook access (yet) or know what is said in class (unless fed that in a prompt).
• Does the paper have self-references that refer back to the AI by name or kind? In some cases students have left references to ChatGPT made to itself in the text. 
• Consider giving ChatGPT your writing prompt and see how it compares to student submissions. 

What are some free AI classifiers? 

OpenAI Classifier is from OpenAI, the company behind ChatGPT, and is perhaps the frankest in explaining its limitations. It provides a verbal indication of the likelihood of human or AI authorship 

GPTZero was developed specifically to target ChatGPT. It gives perplexity and burstiness scores and flags the most suspicious sentences. 

GLTR (Giant Language model Test Room) is a research project from MIT. It uses color-coding and histograms to show how random the language is in the piece. 

AI Writing Check is probably best with shorter passages, as it has a 400-word limit.  It provides a verbal indication of the likelihood of human or AI authorship 

Hugging Face OpenAI Detector gives a percentage likelihood of the text being written by an AI and human. 

Content at Scale's AI Detector gives a percentage likelihood of the text being written by an AI and human.

CopyLeaks AI Detector gives a percentage likelihood that a piece of text is written by a human or AI. However, it can give wildly different answers if individual paragraphs are pasted in, as opposed to longer blocks of writing. It has a Chrome extension allowing you to paste text in on the fly. 

What are some ways I can structure writing assignments to discourage bad or prohibited uses of AI?

• Consider requiring students to quote from specific works, such as the textbook or from class notes. AI is unlikely to have access to either, though students might include quotes in the writing prompt. 
• Add reflective features to the assignment, these could be written or non-written, such as having students discuss live, or record (e.g., VoiceThread, Panopto), on what they have found in their research and reflect on their writing. 
• Use ChatGPT or other tools as part of the writing process, for instance, brainstorming, but also have students critique the work, consider the ethics of using it, etc.
• Create a writing assignment with scaffolding (including outlines, rough drafts, annotated bibliographies, incorporating feedback from peer reviews and the instructor, etc.) could help with some aspects of AI. It might not be very effective on the outline or first draft as those could be still be generated by the AI. Asking for an annotated bibliography would, with the current limitations of the software, be something it could do with much success. Peer review and instructor feedback needs to be detailed and substantive. Otherwise, students could give those as further prompts to the AI, generating new versions of the paper in an iterative fashion. 
• Contact your campus teaching-learning center, writing program, or Missouri Online for ideas and help working out assignments that promote use of AI in positive ways or mitigate possible harms.

What other resources are available? 

• ChatGPT & Generative AI: Missouri Online blog post from February, 2023.
• Cultivating a Culture of Academic Integrity in your Classroom and on Campus: recording of a session on academic integrity (including ChatGPT) from the January, 2023 MU Teaching Renewal Week 2023. (Requires UM Panopto login.)
• How To Check If Something Was Written with AI: long blog post on ways to detect AI writing. This is not focused on education, but some of the suggestions are useful. The author also discusses a mixture of for-fee and freely-available tools. 
• A Toolkit for Addressing AI Plagiarism in the Classroom: this has many good suggestions and a Conversation Template: To Discuss AI-Plagiarism With Students with useful ideas on how to approach a student you believe is using AI in unacceptable ways. 
• AI Writing Detection: A Losing Battle Worth Fighting: Inside Higher Education article covering on-going attempts to detect AI writing, the challenges, and the pitfalls.