Detecting Artificial Intelligence (AI) Plagiarism

Tags

Turnitin AI Artificial inteliigence

Can Turnitin detect AI-produced writing?

You may have heard that Turnitin released a preview of their AI-detection tool from April 4, 2023 through January 1, 2024. Due to concerns about some of its features, the University is reviewing it in more detail prior to deciding on whether to release it. Please be aware that the existing version of Turnitin cannot detect content from ChatGPT or other AI tools. 

These two videos explain what to expect from the tool, if adopted, and how it handles false positives.

Sneak preview of Turnitin’s AI writing and ChatGPT detection capability for Education

Understanding false positives within Turnitin’s AI writing detection capabilities

Additional information and resources from Turnitin on catching AI-based writing are also available.

AI writing detection update from Turnitin's Chief Product Officer explaining what they found in the first few weeks of real-world use and specifying that ~4% of individual sentences that are flagged may be false positives.

What is an AI detector?

These are applications that predict the likelihood that writing was created by an AI or a human. Typically, these look at characteristics of the writing, particularly how random it is. Humans tend to use a greater variety of words, have more randomness in their spelling, grammar, and syntax, and generally more complexity than an AI. Some will give a verbal or graphical indication of how strongly it finds the text to be from a human or an AI. Others return results in terms of perplexity (a measure of randomness) and burstiness (a measure of randomness between sentences) with scores, graphs, or color coding. Lower perplexity and burstiness scores are more likely to be from an AI, with higher ones pointing toward human authorship. 

Do not use ChatGPT or similar AIs (e.g., Bing AI Chat, Bard, Claude) to determine if a paper was written by an AI or human. They produce false results at a very high rate regardless of who or what wrote the paper. They will also produce plausible rationalizations to defend their answers if asked. This is the worst way to determine AI plagiarism. 

Are AI detectors reliable? 

No, at best they are indicative. Published claims to reliability vary greatly, between about 26 and 80%. In other words, expect them to be wrong between a fifth and three-quarters of the time. Those figures apply to the free detectors already available in early 2023. Turnitin claims a false positive rate of 1% (1% of human-created submissions are expected to be flagged as AI-generated) and adjusts its threshold for what it flags as AI-created high in order to avoid more false positives. Those figures may be many times higher for non-native writers of English.

It is possible they will improve, but this should be viewed as an arms race rather than a stable situation. For instance, recent advances with ChatGPT (particularly using GPT-4) show that it is possible to coach it to write with more complexity and fluency, making it harder to detect. This is particularly true of students using well-engineered prompts. By prompt engineering, we mean the creation of fully developed questions and instructions (sometimes including data) to elicit the desired kind of results from the AI. 

If you are going to use these tools, we recommend that you check your sample with more than one.

Are there privacy or other issues? 

Free AI detectors have not been vetted by the University. This is strictly a case of use at your own risk for now. You should never feed them any content allowing identification of the student. We also do not know the specifics of if or how they store or use content. The same applies to feeding text from student papers back into an AI for evaluation.  

Many faculty will enter parts of student papers into Google or other search engines to try to find matches, so this may not seem so different to you. You may wish to consider the differences between that and pasting or uploading all (or large parts of) a paper into an application. Beyond privacy, there may be questions of student copyright to consider. 

Because we have a license and agreements with Turnitin that covers FERPA and meet the University's interpretation of student copyright, these considerations should not apply to the new Turnitin AI tool. 

What should I look for when reading a paper? 

There are also telltale signs to look for, though, again, as the technology evolves, these may change. This is based on ChatGPT.

  • Look at the complexity and repetitiveness of the writing. AI's are more likely to write less complex sentences and to repeat words and phrases more often.  
  • If the assignment has a bibliography, look for made-up or mixed-up entries. Some will be obviously odd, such as having authors writing an article after their death, or a German writing in English, but publishing in a German publication. One of the simplest things, if it provides URLs (particularly with DOI references), try them. In one test we tried, of eight DOI references, two pointed to other articles and six were nonexistent. Note that it is now possible to force GPT (such as Bing AI or ChatGPT with some browser plugins) to search Google Scholar or PubMed when generating citations. In many cases, this will produce real, plausible articles.  Look for egregious factual errors. At least on some subjects, ChatGPT will insert information that is flatly impossible. While students might do this, in combination with other factors, the errors are often things that are unlikely to be made by a human. Remember, AIs do not understand what they are writing. The phrase "stochastic parrots" is often used to describe them, as they work out what words are most likely to follow other words and string them together.  
  • Look for grammatical, syntax, and spelling errors. These are more likely to be mistakes a human author would make. 
  • If you have a writing sample from the student that you know is authentic, compare the style, usage, etc. to see if they match up or vary considerably.  
  • Does the paper refer directly to or quote the textbook or instructor? The AI is unlikely to have textbook access (yet) or know what is said in class (unless fed that in a prompt). 
  • Does the paper have self-references that refer back to the AI by name or kind? In some cases students have left references to ChatGPT made to itself in the text.  
  • Consider giving ChatGPT your writing prompt and see how it compares to student submissions. 

What are some free AI classifiers?

GPTZero was developed specifically to target ChatGPT. It gives perplexity and burstiness scores and flags the most suspicious sentences. 

GLTR (Giant Language model Test Room) is a research project from MIT. It uses color-coding and histograms to show how random the language is in the piece. It is best used not to detect AI writing per se, but to take a closer look at the variation in word choice in a suspect passage, as it gives information word-by-word, rather than by sentence. 

AI Writing Check is probably best with shorter passages, as it has a 400-word limit.  It provides a verbal indication of the likelihood of human or AI authorship 

Hugging Face OpenAI Detector gives a percentage likelihood of the text being written by an AI and human. 

Content at Scale's AI Detector gives a percentage likelihood of the text being written by an AI and human.

CopyLeaks AI Detector gives a percentage likelihood that a piece of text is written by a human or AI. However, it can give wildly different answers if individual paragraphs are pasted in, as opposed to longer blocks of writing. It has a Chrome extension allowing you to paste text in on the fly. 

ZeroGPT is a detector for ChatGPT and Bard. It gives a percentage likelihood and highlights suspect passages. 

What are some ways I can structure writing assignments to discourage bad or prohibited uses of AI?

  • Consider requiring students to quote from specific works, such as the textbook or from class notes. AI is unlikely to have access to either, though students might include quotes in the writing prompt. 
  • Add reflective features to the assignment, these could be written or non-written, such as having students discuss live, or record (e.g., VoiceThread, Panopto), on what they have found in their research and reflect on their writing. 
  • Use ChatGPT or other tools as part of the writing process, for instance, brainstorming, but also have students critique the work, consider the ethics of using it, etc.
  • Create a writing assignment with scaffolding (including outlines, rough drafts, annotated bibliographies, incorporating feedback from peer reviews and the instructor, etc.) could help with some aspects of AI. It might not be very effective on the outline or first draft as those could be still be generated by the AI. Asking for an annotated bibliography would, with the current limitations of the software, be something it could do with much success. Peer review and instructor feedback needs to be detailed and substantive. Otherwise, students could give those as further prompts to the AI, generating new versions of the paper in an iterative fashion. 
  • Contact your campus teaching-learning center, writing program, or Missouri Online for ideas and help working out assignments that promote use of AI in positive ways or mitigate possible harms.

What other resources are available? 

Modified on: Fri, Sep 1, 2023 at 4:09 PM

Did you find it helpful?

More articles about Artificial Intelligence (AI)

Can't find the information you need?

Help us improve the site.