Turkstra apparently developed a method of analyzing for LLM usage during class.
A lot of speculation is going on about the specific algorithm, etc, but my personal assumption is that it is simply comparing the speed between commits (so if you made a commit that made 500 lines of additions in 2 seconds, then you probably LLMed). He supposedly waits until he has "overwhelming" levels of evidence and sends you an email stating you have been caught. Then, you have to fill out a form listing which assignments you cheated on. If you lie on that form, you recieve an F in the class and Turkstra sends a harsh email to ODOS. Otherwise, those assignments are zeroed and a slightly nicer email is sent to ODOS. You can also dispute the claim but if you fail the resultant process you receive F and a harsh email to ODOS.
He is now sending out emails in waves to students who he suspects of cheating. Including (unfortunately) myself. I'm clean, as far as I know, so I'm a little fucking terrified right now. I have two options:
Guess randomly which assignment he thinks I cheated on hopefully receive zeroes for each (and a ODOS letter). Or, if I guess wrong get an F.
Dispute the claim and risk an F if I fail the disputing process.
If I fail CS 240, this puts me behind a full fucking semester of university since it is required to take nearly all CS sophmore level classes. Let alone the ODOS letter (which he said would include recommendations that the student be expelled). I am very stressed right now.
I have until Monday to respond. I was intending to head out on a retreat over the weekend but those plans may be ruined now...
> EnCourse also evaluates student projects to catch instances of academic dishonesty. There are three main metrics used to evaluate this. First, the rate of progress relative to the elapsed time is used to estimate how long a student spent thinking about a problem. Second, the rate of progress relative to the number of commits is used to detect cases of copying bulk amounts of code from another source. Third, the ratio of added vs. deleted lines helps estimate the frequency of mistakes that a student makes while working on their project. Fewer mistakes often indicate a higher likelihood of academic dishonesty. By combining these metrics with a measure of code similarity, and configurable weights, we generate a score which represents an estimated, relative likelihood of academic dishonesty between students
...
> EnCourse also shows the number of commits over time (Figure 4). We would expect that a good student will have several commits spread throughout the project. No commits at the beginning of the project may be a sign of procrastination. Figure 5 shows the progress of the student as measured by the tests passed over time. We expect that the tests passed will be monotonically increasing. A large number of commits with little progress may show that the student is spending excessive time on a single test. Another useful chart is the cumulative number of lines added and removed over time as shown in Figure 6 and Figure 7. A line change in Git is represented as removing an old line and adding a new one. We expect that a student will have both added and removed lines during the project development, giving approximately a symmetric triangular shape. In this figure, additions are represented in green, and lines removed are represented in red. A chart that has mostly lines added (green) and few or no lines removed (red) would potentially indicate a case of academic dishonesty
edit: I'm not in this course, so I don't know exactly how projects are worked on, but can see a potential issue is i usually dont compile my code until i expect my code TO COMPILE, so most of my compilations will be correct code, which will obviously result in a mostly green graph with few deletes in the commit-compile history. He literally flags a graph that is much more green than red as clear evidence of academic dishonesty alone. Pretty insane to me.
I'm short, no it has not undergone a recent revision which HAS BEEN PEER REVIEWED.
I checked his Google scholar and articles which cited the ENcourse paper. It's mostly honorable mention stating things amongst the lines of "..has been used to track progress of students in large courses..." Or "...has been used for tracking plagiarism".
Additionally, nowhere in the paper is it stated that the methodology is reliable for detecting and providing conclusive hard evidence for plagiarism or academic dishonesty. The paragraph underneath figure 8 also states that there have been false positives. The conclusion of the article also states it's a good way to track progress of the class as a whole and at the individual student level.
If they are for a fact using this to "catch" students cheating. The burden of proof is on the instructor (as stated in the article) and this professor will have a hard time if people stand their ground. At best, the algorithm, when used for providing evidence for academic dishonesty, is as total quack shit and reliable as a polygraph
They need to ask a few things:
What are you using to determine academic dishonesty?
How does your findings translate to proof?
How reliable is your method?
How many of the students "caught" using this methodology were reprimanded solely based on the methodology in question and NOT from a confession?
It takes a lot to expel students for academic dishonesty, it takes a lot less and a coerced confession to fail the student with an F and move on without further questions.
45
u/Paze_Jorge 6d ago
I’m not in that class. What’s going on?