Assessment in Higher Education conference 2013

Last week I attended the Assessment in Higher Education conference in Birmingham. This was the least technology and most education conference that I have been to. It was interesting to learn about the bigger picture of assessment in universities. One reason for going was that Sally Jordan wanted my help running a 'masterclass' about producing good computer-marked assessment on the first morning. I may write more about that in a future post. Also I presented a poster about all the different online assessment systems the OU uses. Again a possible future topic. For now I will summarise the other parts of the conference, the presentations I listed to.

One thing I was surprised to discover is how much the National Student Survey (NSS) is influencing what universities do. Clearly it is seen as something that prospective students pay attention to, and attracting students is important. However, as Margaret Price from Oxford Brookes University, the first keynote speaker said, the kind of assessment that students like (and so rate highly in NSS) is not necessarily the most effective educationally. That is, while student satisfaction is something worth considering, students don't have all the knowledge to evaluate the teaching they receive. Also, she suggested that the NSS ratings have made universities more risk-averse in trying innovative forms of assessment and teaching.

The opening keynote was about "Assessment literacy", making the case that students need to be taught a bit about how assessment works, so they can engage with it most effectively. That is, we want the students to be familiar with the mechanics of what they are being asked to do in assessment, so those mechanics don't get in the way of the learning; but more than that, we want the students to learn the most from all the tasks we set them, and assessment tasks are the ones students pay the most attention to, so we should help the students understand why they are being asked to do them. I dispute one thing the Margaret Price said. She said that at the moment, if assessment literacy is developed at all, that only happens serendipitously. However, in my time as a student, there were plenty of times when it was covered (although not by that name) in talks about study skill and exam technique.

Another interesting realisation during the conference was that, at least in that company (assessment experts), the "Assessment for learning" agenda is taken as a given. It is used as the reason that some things are done, but there is no debate that it is the right thing to do.

Something that is a hot topic at the moment is more authentic assessment. I think it is partly driven by technology improvements making it possible to capture a wider range of media, and to submit eportfolios. It is also driven by a desire for better pedagogy, and assessments that by their design make plagiarism harder. If you are being asked to apply what you have learned to something in your life (for example in a practice-based subject like nursing) it is much harder to copy from someone else.

I ended up going to all three of the talks given by OU folks. Is it really necessary to go to Birmingham to find out what is going on in the OU? Well, it was a good opportunity to do so. The first of these was about an on-going project to review the OU's assessment strategy across the board. So far a set of principles have been agreed (for example affirming the assessment for learning approach, athough that is nothing new at the OU) and they are about to be disseminated more widely. There was an interesting slide (which provoked some good discussion) pointing out that you need to balance top-down policy and strategy with bottom up implementation that allows each faculty use assessment that is effective for their particular discipline. There was another session by people from Ulster and Liverpool Hope universities that also talked about the top-down/bottom-up balance/conflict in policy changes.

In this OU talk, someone made a comment along the lines, "why is the OU re-thinking its assessment strategy? You are so far ahead of us already and we are still trying to catch up." I am familiar with hearing comments like that at education technology conferences. It was interested to learn that we are also held in similarly high for policy. The same questioner also used the great phrase "the OU effectively has a sleeper-cell in every other university, in the associate lecturer you employ". That makes what the OU does sound far more excitingly aggressive than it really is.

In the second OU talk, Janet Haresnape described a collaborative online activity in a third level environmental science course. These are hard to get right. I say that having suffered one as a student some years ago. This one seems to have been more successful, at least in part because it was carefully structured. Also, it started with some very easy tasks (put your name next to a picture and count some things in it), and the students could see the relationship between the slightly artificial task and what would happen in real fieldwork. Janet has been surveying and interviewing students to discover their attitudes towards this activity. The most interesting finding is that weaker students comment more, and more favourably, on the collaboration than the better students. They have more to learn from their peers.

The third OU talk was Sally Jordan talking about the ongoing change in the science faculty from summative to formative continuous assessment. It is early days, but they are starting to get some data to analyse. Nothing I can easily summarise here.

The closing keynote was about oral assessment. In some practice-based subjects like law and veterinary medicine it is an authentic activity. Also, a viva is a dialogue, which allows the extent of the student's knowledge to be probed more deeply than a written exam. With an exam script, you can only mark what is there. If something the student has written is not clear, then there is no way to probe that further. That reminded me of what we do in the Moodle quiz. For example in the STACK question type, if the student has made a syntax error in the equation they typed, we ask them to fix it before we try to grade it. Similarly, in Pattern-match questions, we spell check the student's answer and let them fix any errors before we try to grade it. Also, with all our interactive questions, if the student's first answer is wrong, we give them some feedback then let them try again. If they can correct their mistake themselves, then they get some partial credit. Of course computer-marked testing is typically used to assess basic knowledge and concepts, whereas an oral exam is a good way to test higher-order knowledge and understanding, but the parallel of enabling two-way dialogue between student and assessor appealed to me.

This post is getting ridiculously long, but I have to mention two other talks. Calum Delaney from Cardiff Metropolitan University reported on some very interesting work trying to understand what academics think about as they mark an essays. Some essays are easy to grade, and an experienced marker will rapidly decide on the grade. Others, particularly those that are partly right and partly wrong, take a lot longer weighing up the conflicting evidence. Overall though, the whole marking process struck me, a relative outsider, as scarily subjective.

John Kleeman, chair of QuestionMark, UK, summarised some psychology research that shows that the best way to learn something so that you can remember it again is to test yourself on it, rather than just reading it. That is, if you want to be able to remember something, then practice remembering it. It sounds obvious when you put it that way, but the point is that there is strong evidence to back up that statement. So, clearly you should all now go and create Moodle (or QuestionMark) quizzes for your students. Also, in writing this long rambling blog post I have been practising recalling all the interesting things I learned at the conference, so I should remember them better in future. If you read this far, thank you, and I hope you got something out of it too.

Open University question types ready for Moodle 2.5

This is just a brief note to say that Colin Chambers has now updated all the OU question types to work with Moodle 2.5. Note that we are not yet running this code ourselves on our live servers, since we are on Moodle 2.4 until the autumn, but Phil Butcher has tested them all and he is very thorough.

You can download all these question types (and others) from the Moodle add-ons database.

Thanks to Dan Poltawski's Github repository plugin, that is easier than it used to be. Still, updating 10 plugins is pretty dull, so I feel like I have contributed a bit. I also reviewed most of the changes and fixed the unit tests.

I hope you enjoy our add-ons. I am wondering whether we should add the drag-and-drop questions types to the standard Moodle release. What do you think? If that seems like a good idea to you, I suggest posting something enthusiastic in the Moodle quiz forum. It will be easier to justify adding these question types to standard Moodle if lots of non-OU Moodlers ask for it.

Book review: Computer Aided Assessment of Mathematics by Chris Sangwin

Chris is the brains behind the STACK online assessment system for maths, and he has been thinking about how best to use computers in maths teaching for well over ten years. This book is the distillation what he has learned about the subject.

While the book focusses specifically on online maths assessment, it takes a very broad view of that topic. Chris starts by asking what we are really trying to achive when teaching and assessing maths, before considering how computers can help with that. There are broadly two areas of mathematics: solving problems and proving theorems. Computer assessment tools can cope with the former, where the student performs a calculation that the computer can check. Getting computers to teach the student to prove theorems is an outstanding research problem, which is touched on briefly at the end of the book.

So the bulk of the book is about how computers can help students master the parts of maths that are about performing calculations. As Chris says, learning and practising these routine techniques is the un-sexy part of maths education. It does not get talked about very much, but it is important for students to master these skills. Doing this requires several problems to be addressed. We want randomly generated questions, so we have to ask what it means for two maths questions to be basically the same, and equally difficult. We have to solve the problem of how students can type maths into the computer, since traditional mathematics notation is two dimensional, but it is easier to type a single line of characters. Chris precedes this with a fascinating digression into where modern maths notation came from, something I had not previously considered. It is more recent than you probably think.

If we are going to get the computer to automatically assess mathematics, we have to work out what it is we are looking for in students' work. We also need to think about the outcomes we want, namely feedback for the student to help them learn; numerical grades to get a measure of how much the student has learned; and diagnostic output for the teacher, identifying which types of mistakes the students made, which may inform subsequent teaching decisions. Having discussed all the issues, Chris them brings them together by describing STACK. This is an opportune moment for me to add the dislaimer that I worked with Chris for much of 2012 to re-write STACK as a Moodle question type. That was one of the most enjoyable projects I have ever worked on, so I am probably biassed. If you are interested, you can try out a demo of STACK here.

Chris rounds off the book with a review of other computer-assissted assessment systems for maths that have notable features.

In summary, this is a facinating book for anyone who is interested in this topic. Computers will never replace teachers. They can only automate some of the more routine things that teachers do. (They can also be more available than teachers, making feedback on their work available to students even when the teacher is not around.) To automate anything via a computer you really have to understand that thing. Hence this book about computer-assessted assessment gives a range of great insights into maths education. Highly recommended. Buy it here!

Performance-testing Moodle

Background

The Open University is moving from Moodle 2.3.x to Moodle 2.4.3 in June. As is usual with a major upgrade, we (that is Rod and Derek) did some load testing to see if it still runs fast enough on our servers.

The first results were spectacularly bad! Moodle 2.4 was ten times slower. We were expecting Moodle 2.4 to be faster than 2.3. The first step was easy.

Performance advice: if you are running Moodle 2.4 with load-balanced web servers, don't use the default caching option that stores the data in moodledata on a shared network drive. Use memcache instead.

Take 2 was a lot better. Moodle 2.4 was now only about 1.5 times slower. Still not good enough, but in the right ball park. This blog post is about what we did next, which was to use the tools Moodle provides to work out what was slow and fix it.

Moodle's profiling tool

When your software is too slow, you need measurements to tell you which are the slow bits. Tools that do that are called profilers. One of the better profiling tools for PHP is called XHProf. The good news is that it has already been integrated into Moodle, and there is documenation about getting it working. Basically, you just need to install a PHP extension and turn on some options under Admin -> Development -> Profiling.

Since we already had the necessary PHP extension on our severs, that was really easy. The option I chose was to profile a page when &PROFILEME was added to the end of the URL, but there are several ways to control it.

Profiling output

Once you have profiled a page, the results appear under Admin -> Development -> Profiling runs.

This just lists the runs you have done. You need to click through to see the details of one run. That looks like a big table of all the function that were called as part of rendering the page, how many times each one was called, and how much time each function was responsible for.

Inclusive time is the amount of time taken by that function, and all the other functions it called. Exclusive time is the time taken by that function itself. Some people, like sam, seem to like that tabular view. I am a more visual person, so I tend to click on the [View full callgraph] link. That produces a crazily big image, showing graphically which functions call which other functions, and how much time is spent in each one. Here is the image for the run we are looking at:

You can click for the full-sized image. The yellow and red highlighting is applied automatically to try to highlight places where a lot of time is being spent. Sometimes it is helpful. Sometimes not. The red box in the bottom right is where we do database queries. No suprise there. We know calling the database is one of the slowest things you can do in Moodle code. The other red box is fetching data from memcache, which also involves connecting to another server.

What you have to look for is somewhere on the diagram that makes you go "What! We are spending how much time doing that?! That's surely not necessary." In this case, my eye was drawn to the far right of the yellow chain. When viewing this small course, we are fetching the course format object 134 times, and doing that is accounting for about 9% of the page-load time. There is no way we need to do that.

Fixing the problem

Once you have identified what appears to be a gross inefficiency, then you have to fix it. Mostly that follows the normal Moodle bug-fixing mechanics, but it is worth saying a bit about the different approaches you could take to changing the code:

1. You might work out that what is being done is unnecessary. Then you can just remove it. For example MDL-39452 or MDL-39449. This is the best case. We have both improved performance and simplified the code.
2. The next option is to take an overview of the code, and re-organise it to be more sensible. For example, in the course format case, we should probably just get the course format object once, and then use it. However, that would be a big risky change, which I did not want to do at this time (just before the Moodle 2.5 release). This approach does, however, also have the potential to simplify the code while improving performance.
3. The next option is some other sort of refactoring. For example get_plugin_list was getting called a lot, and it in turn was calling the generic function clean_param to validate something. clean_param is quite fast, but not when you call it a thousand times. Therefore, it was worth extracting a simpler is_valid_plugin_name function. Doing that (MDL-39445) reduced the page load time by about 2%, but did make the code slighly more complex. Still, that is a worth-while trade off.
4. The last option is to add caching. If you are doing the same thing repeatedly, and it is slow, and you can't avoid doing it repeatedly, then remember the answer the first time you compute it, and reuse it later. This should be the option of last resort because caches definitely increase the code complexity, and if you forget to clear them when necessary you introduce bugs. However, as in the course formats example we are looking at they can make a big difference. This fix reduced page-load times by 8%.

So far, we have found nine speed-ups we can make to Moodle 2.4 in the core Moodle code, and about the same in OU plugins. That is probably a 10-20% speed-up on most pages. Some of those are new problems introduced in Moodle 2.4. Others have been there since Moodle 2.0. We could really benefit from more people looking at Moodle profiling output often, and that is why I wrote this article.

Do different media affect the effectiveness of teaching and learning

Here is some thirty-year-old research that still seems relevant today:

Richard E. Clark, 1983, "Reconsidering Research on Learning from Media", Review of Educational Research, Vol. 53, No. 4 (Winter, 1983), pp. 445-459.

This paper reviews the the seemingly endless research trying to ask whether teaching using Media X inherrently more effective than the same instruction in Media Y. Given the age of the paper, you will not suprised to learn that the research cited covers media like Radio for education (hot research topic in the 1950s), Television (1960s) and early computer-assessted assessment (1970s). Clark's earliest citation, however, is "since Thorndike (1912) recommended pictures as a labor saving device in instruction." Images as novel educational technology! Well, they were once. The point is  that basically the same reserach was done for each new media to come along, and it was all equally inconclusive.

Here are some choice quotes that nicely summarise the article:

Based on this consistent evidence, it seems reasonable to advise strongly against future media comparison research. Five decades of research suggest that there are no learning benefits to be gained from employing different media in instruction, regardless of their obviously attractive features or advertised superiority.

Where learning benefits are at issue, therefore, it is the method, aptitude, and task variables of  instruction that should be investigated.

The best current evidence is that media are mere vehicles that deliver instruction but do not influence student achievement any more than the truck that delivers our groceries causes changes in our nutrition

Clark does not miss out on the fact that effectiveness of the learning is the only problem in education:

Of course there are instructional problems other than learning that may be influenced by media (e.g., costs, distribution, the adequacy of different vehicles to carry different symbol systems, equity of access to instruction).

Since this paper is a thorough review of a lot of the available literature, it contains a number of other gems. For example:

Ksobiech (1976) told 60 undergraduates that televised and textual lessons were to be (a) evaluated, (b) entertainment, or (c) the subject of a test. The test group performed best on a subsequent test with the evaluation group scoring next best and the entertainment group demonstrating the poorest performance.

Hess and Tenezakis (1973) ... Among a number of interesting findings was an unanticipated attribution of more fairness to the computer than to the teacher.

I wonder how much later research fell it to the trap outlined in this paper. I am not familiar enough with the literature, but presumably there was lots of papers about the world-wide web, VLEs, social media, mobiles and tablets for education. I wonder how novel they really were?

Today, computers and the internet have made media cheaper to produce and more readily accessible than ever before. This removes many constraints on the instructional techniques available, but what this old paper is reminding us is that when it comes to teaching, it is not the media that matters, but the instructional design.

Automating git

This is a long-overdue follow-up to my previous post about using git to fix Moodle bugs. Thanks to Andrew Nichols of LUNS for nudging me in to writing this.

Git has an efficient command-line interface, but even so, there are some sequences of commands that you find yourself typing repeatedly. Git provides a mechanism called aliases which can be used to reduce this repetitive typing. This post explains how I use it in my Moodle development.

Basic usage

Let us start with the simplest possible example. I get bored typing git cherry-pick in full all the time. The solution is to edit the file .gitconfig in my home directory, and add

[alias]
        cp     = cherry-pick

Then git cp … is equivalent to git cherry-pick …. That saves 9 characters every time I have to copy a bug fix to a stable branch.

Simple aliases like this can also be used to to supply options. Another one I have set up is

        ff     = merge --ff-only

I use that when I need to update one of my local branches to match a remote branch. Suppose I think I are on the master branch, and I want to update that to the latest moodle/master. Normally one would just type git merge moodle/master and it would look like this:

timslaptop:moodle_head tim$ git merge moodle/master
Updating ddd84e9..b658200
Fast-forward

Suppose, however, that I had made a mistake, and I was actually on some other branch. Then git would try to do a merge between master and that branch, which is not what I want. The --ff-only option tells git not to do that. Instead it will stop with an error if it can't do a fast forward. So, to prevent mistakes, I normally use that option, and I do it frequently enough I found it worthwhile to create the alias.

Getting more ambitious

Sometimes the repeated operation you want to automate is a sequence of git commands. For example, when a new weekly build of Moodle comes out, I need to type a sequence of commands like this:

git checkout master
git fetch moodle
git merge --ff-only moodle/master
git push origin master

That updates my local copy of the master branch with the latest from moodle.org and then copies that to my area on github. To automate this sort of thing, you have to start using the power of Unix shell scripting. (If you are on Windows, don't worry, because you typically get the bash shell when you install git.)

Fortunately, you don't need to know much scripting, and you can probably just copy these examples blindly. The first thing to know is that you can put two commands on one line if you separate them using a semi-colon (just like in PHP). The previous sequence of commands could be typed on one line as

git checkout master; git fetch moodle; git merge --ff-only moodle/master; git push origin master

(Note that these lines of code are getting quite long, and will probably line-wrap. It should, however, be a single line of code.)

Doing it this way turns out to be a bad idea. What happens if one of the commands gives an error? Well, the system will just move on to the next command, even though the error from the previous command probably left things in an unknown state. Dangerous! Fortunately there is a better way. If you use && instead of ; then any error will cause everything to stop immediately. If you are familiar with PHP, then just image that every command is a function that returns true or false to say whether it succeeded or not. That is not so far from the truth. So, the right way to join the commands together looks like this:

git checkout master && git fetch moodle && git merge --ff-only moodle/master && git push origin master

Now we know what we want to automate, we need to teach this to git. It is a bit tricky because we don't just want to convert one single git command into another single git command. Instead we want to convert one git command into a sequence of shell commands. Fortunately this is supported, you just need to know the right syntax:

        updatemaster = !sh -c 'git checkout master && git fetch moodle && git merge --ff-only moodle/master && git push origin master' -

Now I just have to type git updatemaster to run that sequence of commands.

Parameterising your aliases

That is all very well for master, but what about all the stable branches? Do I have to create lots of separate aliases like update23, update22, update21, …? Of course not. Git was created by and for computer programmers. Shell scripts can take parameters, and the solution is an alias that looks like

        update = !sh -c 'git checkout MOODLE_$1_STABLE && git fetch moodle && git merge --ff-only moodle/MOODLE_$1_STABLE && git push origin MOODLE_$1_STABLE' -

With that alias, git update 23 will update my MOODLE_23_STABLE branch, git update 22 will update my MOODLE_22_STABLE, and so on.

You can use any number of parameters. If you remember my previous blog post, typically I will create the bug fix on a branch with a name like MDL-12345 that starts from master, and then I will want to copy that to a branch called MDL-12345_23 branching off MOODLE_23_STABLE. With the following alias, I just have to type git cpfix MDL-12345 23 in my Moodle 2.3 stable check-out:

        cpfix = !sh -c 'git fetch -p origin && git checkout -b $1_$2 MOODLE_$2_STABLE && git cherry-pick origin/master..origin/$1 && git push origin $1_$2' -

One final example that belongs in this section:

        killbranch = !sh -c 'git branch -d $1 && git push origin :$1' -

That deletes a branch both in the local repository and also from my area on github. That is useful once one of my bug fixes has been integrated. I then no longer need the MDL-12345 branch and can eliminate it with git killbranch MDL-12345.

To boldly go …

Of course, all this automation comes with some risk. If you are going to screw up, automation lets you screw up more things quicker. I feel obliged to emphasis that at this point. If you are going to shoot yourself in the foot, a machine gun gives the most spectacular results, and we are about to build one, at least metaphorically.

We just saw the killbranch command that can be used to clean up branches that have been integrated. What happens if I submitted lots of branches for integration last week. I have to delete lots of branches. Can that be automated? Using git I can at least get a list of those branches:

timslaptop:moodle_head tim$ git checkout master
Already on 'master'
timslaptop:moodle_head tim$ git branch --merged
  MDL-12345
  MDL-23456
* master

Those are the branches that are included in master, and so are presumably ones that have already been integrated. It is a bit irritating that the master branch itself is included in the list, but I can get rid of it using the standard command grep:

timslaptop:moodle_head tim$ git branch --merged | grep -v master
  MDL-12345
  MDL-23456

I have a list of branches to delete, but how can I actually delete them? I need to execute a command for each of those branch names. Once again, we find that shell scripting was developed by hacker, for hackers. The command xargs does exactly that. xargs executes a given command once for each line of input it receives. Feeding in the list of branches, and getting it to execute the killbranch command looks like this:

git branch --merged | grep -v $1 | xargs -I "{}" git killbranch "{}"

Now to make that into an alias

        killmerged = !sh -c 'git checkout $1 && git branch --merged | grep -v $1 | xargs -I "{}" git killbranch "{}"' -

With that in place, git killmerged master will delete all my branches that have been integrated into master. Note that you can use one alias (killbranch) inside another (killmerged). That makes it easier to build more complex aliases.

Once I have deleted all the things that were integrated, I am left with the branches I have in progress that have not been integrated yet. Those all need to be rebased, and that can be automated too:

        updatefix = !sh -c 'git checkout $1 && git rebase $2 && git checkout $2 && git push origin -f $1' -
        updatefixes = !sh -c 'git checkout $1 && git branch | grep -v $1 | xargs -I "{}" git updatefix "{}" $1' -

With those in place, I just just type git updatefixes master, and that will rebase all my branches, both locally and on github. Use at your own risk!

Thats all folks

To summarise, here is the whole of the alias section of my .gitconfig file:

[alias]
        cp     = cherry-pick
        ff     = merge --ff-only
        cpfix  = !sh -c 'git fetch -p origin && git checkout -b $1_$2 MOODLE_$2_STABLE && git cherry-pick origin/master..origin/$1 && git push origin $1_$2' -
        update = !sh -c 'git checkout MOODLE_$1_STABLE && git fetch moodle && git merge --ff-only moodle/MOODLE_$1_STABLE && git push origin MOODLE_$1_STABLE' -
        killbranch = !sh -c 'git branch -d $1 && git push origin :$1' -
        killmerged = !sh -c 'git checkout $1 && git branch --merged | grep -v $1 | xargs -I "{}" git killbranch "{}"' -
        updatefix = !sh -c 'git checkout $1 && git rebase $2 && git checkout $2 && git push origin -f $1' -
        updatefixes = !sh -c 'git checkout $1 && git branch | grep -v $1 | xargs -I "{}" git updatefix "{}" $1' -

There is limited documentation for this on the git config man page. There is more on the git wiki.

Standards

Standardisation efforts are odd things. Most successful standards seem to have come out of one or a few brilliant individuals, and the standardisation committees only took over after the thing in question became widely adopted. Think of C, C++, Java, HTML, HTTP, JavaScript, SQL… Of course, it is only with hind-sight that we know those were successful things, that the people who created them were brilliant, and that it was worth investing effort in a standardisation committee to get different implementations to be interoperable. There are many fewer examples of successful standards that started with a committee. I am sure there are some, but I am failing to think of any right now.

Even when there are standards, that does not magically solve all your problems. Ask any developer about the problems of getting their web site to work on all browsers, particularly Internet Explorer, despite the existence of the HTML, CSS and JS standards; or look at the work Moodle has to do to work with the four databases we support, even though SQL is supposed to be a standardised language.

In theory a standard makes sense. If you have n different systems you want to move data between, then

• If you go directly from system to system, you would have to write ½n*(n-1) different importers.
• Given a common standard, you only need to write n different importers.

In practice, different systems have slightly different features, so you cannot perfectly copy data from one system to another. An importer from X to Y is not a perfect thing, it has to fudge some details. Now compare the two ways of handling import:

• An importer for System Y that directly imports the files output by System X can know all about the details of System X, so it can do the best possible job of dealing with the incompatible features.
• Using Standard S, System X has to save its data in format S dealing with any incompatibilities between what X supports and what S supports. Then System Y has to take file S and import it, dealing with any incompatibilities between what S supports and what Y supports, and it has to do that without the benefit of knowing that the data originally came from System X.

Therefore, going for direct import is likely to give better results, although at the cost of more work.

The particular case I am thinking about is, of course, moving questions between different eAssessment systems. The only standard that exists is IMS QTI, which has always struck me as the worst sort of product of a committee. It is not widely adopted and it is horribly complicated. Also, if we wanted to make Moodle support QTI, we would have to completely rewrite the Moodle to work the way QTI specifies. That is sort-of fair enough. If you want to display HTML like a web browser, you basically have to start from scratch and write you code from the ground up to work the way the HTML, CSS and JavaScript standards say. These standards are not designed to make content interoperate between different existing systems. You need only look at the horrible mess you get when you do Save as… -> HTML in MS Word, or even just copy-and-paste from Word to Moodle, to be convinced of that.

So, QTI is trying to solve the wrong problem. It is trying to be a full-featured standard that you can only support by basing your whole software around what the standard says. We don’t want to rewrite the whole Moodle question engine just to support some standard that hardly anyone else uses yet. We just want to be able to import 99%+ of questions from other systems, and from publishers, that Teachers can get access to. The kind of standard we want is more like CSV files. CSV is a nice simple standard to transfer data between spreadsheets and other applications.

In the past, it has always been easier to write separate importers for each other system Moodle wants to import from, rather than trying to deal with one very complex generic standard like QTI. See the screen-grab of Moodle's import UI to the right. To write a new importer, you just need some example files in the format you want to support, containing a few questions of each type. Then it is easy to write the code to parse that sort of file, and converting the data to the format Moodle expects.

Having said that, the current situation is not perfect. The problem is that most of these other file formats are output by commercial software. Therefore, many developers cannot easily get sample files in those formats to use for developing and testing code. As a result, some of the importers are buggy. We have to rely on people in the community who care enough, and who have access to the software, to create example files for us. There was a good example of that recently: Someone called Rick Jerz from Le Claire, Iowa produced a good example Examview file, and long-time quiz contributor Jean-Michel Vedrine from St Etienne, France used that to fix the bugs in the Examview importer.

On the standardisation front, there is a glimmer of hope. IMS Common Cartridge is a standard for moving learning content from one VLE to another. It uses a tiny, and hence manageable, subset of QTI that tries to solve the “transfer 99%+ of the questions teachers use” problem. It should be possible to get Moodle to read and write that QTI subset. We just need someone with the time and inclination to do the work. It is even possible that the OU's OpenLearn project will be that someone, but QTI import/export is just one of many things on their to-do list.