The online assessment Turing test

In the back-channel for yesterday's Transforming Assessment webinar (which I would recommend) Geoff Crisp asked me:

"Tim - what about the Turing test - what if a student could not tell the difference between a computer giving them feedback and the teacher?"

I think this is a really nice question. Food for some quite wide-ranging thoughts about what online assessment should be.

On the whole, I stand my the snap answer I gave at the time. Computers and human markers (at least currently) have different strengths. The computer (having been set up by a human teacher) can be there at any time the student wants, able to give immediate feedback on a range of more or less basic practice activities. A human teacher in only available at certain times, but is able to give feedback in a more holistic way. They may know the student, and have some concept about how their subject is best learned, on on that bases give the student some really meaningful advice about how best to improve.

I know there is adaptive learning hype about computers being able to know the students and therefore offer contextual advice, but I will believe that when (if) I see it.

If you are thinking about designing a course today, you are much better off understanding the strengths and weaknesses of both computer-marked and conventional assessment, and using each for where they work best. There is currently nothing to be gained by trying to hide where you are using computer marking.

I think a reasonable analogy is with searching for information. You might do a Google search, which will give you the kind of results that a computer can give. Alternatively, you could ask a friend who knows more about the subject, and they will give you a different sort of advice about what to read. Neither is necessarily better. In some cases one of the two approaches might be clearly more appropriate. In other cases either would do. If you really want to understand something in depth, you probably want use both approaches, and it is an advantage that each will give different results that help in different ways.

If we are trying to create self-regulating learners, then it can be a merit that a computer only gives basic templated feedback, which could be as little as just right/wrong. The learner needs to do more work themself to get from the feedback to an action to take to improve. This is not always a benefit, but it could be.

So, while the idea of an assessment Turing test is usefully thought provoking, I don't think it is educationally useful, at least not for the foreseeable future. Having said that, the nicest thing anyone said about an online assessment system I helped build is still

"It's like having a tutor at your elbow."

The key word there is "like", which is not the same as "indistinguisable from".