Moodle bug tracker

Today, between fixing bugs and reviewing code, I spent a bit of time tinkering with my dashboard in the Moodle bug tracker. I was trying to make it as clear as possible which issues need my attention. I am quite pleased with the result:

The issue statistics widget does not just show you the pretty graphs, it also makes it easy to get at those issues. For example, if I click on 1.9.12 in the My targetted issues box, then I am taken to a list of those 11 issues. That particular widget I have used for a while, the new parts are the boxes just under there.

My: Ongoing pull requests I added to make it easy to find the things I have submitted for inclusion in next week's weekly build (hopefully). Thanks to Eloy, that filter is now available to everyone in the jira-developers group.

The next two boxes let me quickly get to issues with patches attached. There is an emerging convention of adding the label patch to such issues, where the attached code needs to be reviewed. This makes finding such issues very much easier. The whole point of the new development processes is to encourage more people to contribute patches, and then ensure those patches get looked at, rather than just sitting there for years. (Here is an example I found yesterday of what used to happen: MDL-13983). Therefore, as quiz maintainer, I need to be able to see easily if anyone has submitted any relevant patches. I also want easy access to bugs with patches that I created or commented on.

Having brought it up, can I say that I am quite happy with how the new processes are working so far. My impression is that since they were introduced, I have received more usable bug fixes for the quiz that in the past. I am not sure how much causality one can claim there, however, since as well as the new processes, we also had the Moodle 2.0 release. Moodle 2.0 has plenty of minor bugs that are ripe for fixing. So, it may just be that we are seeing lots of bug fixes because there are lots of bugs.

At the other end, it has made it a bit easier to get my code reviewed. Well, finished code where I have created a PULL request certainly gets is reviewed. It is still sometimes a problem to get comments on work-in-progress, because everyone is so busy.

Etiquette for questions

I have been working hard at converting the new Moodle question engine to work in Moodle 2.0, aiming at a deadline this Friday (25th February). On Friday we should have a first OU version of Moodle 2.0 with all the key features so that we can start testing, even though students won't get onto the new system before July. I have have basically finished the question engine, give or take a few features that are not needed for testing, and this week I am just doing some final tidying up of the code.

Hopefully, next week I can start the process of getting it reviewed for inclusion in Moodle 2.1. As I say, there are some gaps in the functionality that will need to be filled in before it can actually be committed, but there is a lot of code to be reviewed (lucky Eloy!) and so I hope we can kick off the process.

So, my excuse for not blogging about the new question engine recently is that I have been too busy working on it to write about it. In the last few days, however, I encountered a couple of nice ideas that would be easy to implement using the flexibility the new question engine gives, and I want to describe them. First, I need to remind you of one key point about the new system:

Question behaviours

As I explained before a key idea in the new question engine is that of question behaviours. Whereas a question type lets you have either a multiple-choice, a drag and drop, or a short-answer question, a behaviour controls how the student interact with the questions, of whatever type. For example, the student may have to type in answers to each question in the quiz, then submit everything, and only then are the questions marked. This is known as the "Deferred feedback" behaviour. Alternatively, the student may answer one question, have their answer marked immediately. If they are wrong, they get a hint and can then immediately have another go. If they get it right on the second or third try, they get fewer marks. This is called the "Interactive with multiple tries" behaviour.

When I was first working on this, I did wonder whether it was perhaps over-kill to make behaviours fully-fledged Moodle plugins. It seemed to me that I had already implemented all the types of behaviour anyone was likely to want. It turns out I was wrong. Here are three ideas for new behaviours have I have come across since had that naive thought.

Explain your thinking behaviour

The concept here is that, in addition to presenting the question to the student for them to answer in the usual way, you also give them a text-area with the prompt "Explain your answer". When the submit the question is graded as usual. Moodle does not do anything with the explanation, other than to store it, and re-display it later when the student or their teacher reviews their attempt. The point is that the student should reflect upon and articulate their thought processes, and the teacher can then see what they wrote, which might be useful for diagnosing what problems the students are having.

I'm not sure that this would really work. Would the students really bother to write thoughtful comments if there were no marks to be had? However, this would be relatively easy to implement, so we should build it and see what happens in practice. The teacher could always manually adjust the marks based on the quality of the reflection, if that was necessary to incentivise students.

I'm afraid I cannot remember who suggested this idea. It was a post in the Moodle quiz forum some time ago, just after I had implemented the behaviour concept and was thinking that my initial list of behaviours was all anyone could possibly want.

gnikram desab-ytniatreC

This idea I only came across yesterday evening, in a blog post from people in the OU's technology faculty. It is a slightly strange twist on certainty-based marking.

With classic CBM, the students answers the questions, and also says how certain they are they got it right (for example, on a three-point scale). The student will only get full marks if they get the question right, and are very certain that they were right. If, however, they express high certainty and are wrong, they are penalised heavily with a big negative mark. To maximise their score, the student must accurately gauge their level of knowledge. This hopefully promotes reflection, and self awareness by the student of their level of knowledge.

The idea from the OU technology faculty is to do this backwards, for multiple choice questions. Rather than getting the student to answer the question and then select a certainty, you first show them just the question stem without the choices, and get them to express a certainty. Only then do you show them the choices and let them chose what they think is the right answer.

Again, I am not sure if this would work, but it is sufficiently easy to do by creating a new behaviour plug-ing (and a some change to the multiple-choice question type so that you can output just the question, without the choices) that it has to be worth a try.

Free text responses with a chance to act on feedback

This last idea I only heard about this morning. There was a session of the OU's "eLearning community" all about eAssessment, which naturally I attended. This is a monthly gathering with a number of different presentations on some eLearning topic. The first three talks were about specific courses that have recently adopted eAssessment, and how students had engaged with that, what effect the effect had been on retention and pass rates, and so on. That was interesting, but not what I want to talk about here. The final talk was by Denise Whitelock from the OU's Institute of Educational Technology who has just completed a review of recent research into technology-enhance assessment for HEA that should be published soon. Here, I just want to pick up on one specific idea from her talk.

I'm afraid that again, I don't recall who deserves credit for this idea. (Once Denise's review is published, I will have a proper reference, but I did not take notes this morning.) It was another UK university that had done this. It was in the context of language teaching. The student had to type a sentence in answer to the question, then the computer graded that attempt and gave some feedback. Then, the student was immediately allowed to revise their sentence in light of the feedback, and get it re-marked. The final grade for the question is then a weighed sum of the first mark and the second mark. You need to get the weights right. The weight for the first try has to be big enough that the student tries hard to get the question right on their own before seeing the hints, and the weight for the second try, though smaller, also has to be big enough so that the student bothers to revise their response.

Now, the OU is currently creating a Moodle question type that can automatically grade sentence length answers using an algorithm that my colleague Phil Butcher implemented the first version of in 1978! (When I say we are creating this, what I actually mean is that we have contracted Jamie Pratt a free-lance Moodle developer to implement it to our specification.) Anyway, once you have that, the idea of allowing two tries, with feedback after the first try, and a final grade that is a weighted sum of the marks for the two tries, is just another behaviour.

So, my initial thought that people would not have many ideas for interesting new behaviours seems to have been wrong. The flexibility I built into the system is worth having.

Should you listen to futurologists?

Educause just published their annual survey describing "six areas of emerging technology that will have significant impact on higher education and creative expression over the next one to five years".

This got circulated round the team at work and I rather cynically asked "so, what did they predict last year then?" My colleague Pete Mitton took that question and ran with it to produce the following analysis:

OK, as I have a full set of Horizon Reports on my hard disk, here's a summary of their predictions for the years 2004-11.

I've pushed some titles together where the wording is different but the intent is the same (for example they've used mobile computing/mobiles/mobile phones in the past with the same meaning).

The numbers in the table are the time-to-adoption horizon in years.

	2004	2005	2006	2007	2008	2009	2010	2011
User-created content			1	1	1
Social Networking		4-5	1	1
Mobiles			2-3	2-3		1	1	1
Virtual Worlds				2-3
New Scholarship and Emerging forms of publication				4-5
Massively Multiplayer Educational Gaming				4-5
Collaboration Webs					1
Mobile Broadband					2-3
Data Mashups					2-3
Collective Intelligence					4-5
Social Operating Systems					4-5
Cloud Computing						1
The Personal Web						2-3
Semantic-Aware Applications						4-5
Smart Objects						4-5
Open Content							1
Electronic Books							2-3	1
Simple Augmented Reality		4-5	4-5				2-3	2-3
Gesture-based computing							4-5	4-5
Visual Data Analysis							4-5
Game-based learning		2-3	2-3					2-3
Learning analytics								4-5
Learning Objects	1
Scaleable Vector Graphics	1
Rapid Prototyping	2-3
Multimodal Interfaces	2-3
Context Aware Computing aka Geostuff	4-5		4-5			2-3
Knowledge Webs	4-5
Extended Learning		1
Ubiquitous Wireless		1
Intelligent Searching		2-3

Of course, the purpose of a report like this is not to accurately predict the future. The aim is rather to stimulate informed debate about the technologies that are coming up. Within our team, at least, they seem to have succeeded.

I thought, however, that this analysis was interesting enough to share. It provides some context for year's predictions. More generally it shows how difficult it is predict future technology trends.

How (not) to sell your open source development services

It must be terrible trying to sell software development services if you work in Open Source. The restrictions of the GPL appear to make it almost impossible to demonstrate your previous work to prospective clients. I would like to offer the following advice, which is my attempt to distil what appears to be current best practice in dealing with this tricky situation.

Do not make it possible for your potential client to find any examples of code you have written. If you cannot avoid it completely, make it as difficult for them as possible. For example, if you have created a certain sort of plugin, just say "We have created a new Moodle block." Do not give any clues as to which block that might be. If the open source project provides a convenient place to upload and share your contributions, try to avoid making any code you have produced easy to find there. Even if you have been forced to share your code in these places, do not on any account provide your potential client with a link to the example of work that you are most proud of. It is fairer to let them search and find a representative sample of your work, if they are clever and patient enough to do so.

Your development staff are, or course, all faceless drones who are nothing to be proud of. Your potential client does not care who will actually be doing the work they are requesting. This is particularly important when the open source project has a strong community. It must not be possible to identify your staff in the recognised list of project contributors; nor should it be easy to discover how they have contributed to the on-going development of the project's code by looking in the project's issue tracker.

It is important is to dress your proposal up in meaningless marketing-speak. If your client cannot complete their buzzword bingo card while reading your document, they just won't hire you. Including a PowerPoint presentation with impressive diagrams and more platitudes about your company can create a particularly strong impression.

The new question engine - how it works

In my last blog post I promised more details of the new question engine "in a week or so". Unfortunately, things like work (mainly fixing minor bugs in the aforementioned question engine); rehearsing for a rather good concert that will take place this Friday; and buying curtains for my new flat, have been rather getting in the way. Now is the time to remedy that.

Last time I explained roughly what a question engine was, and that I had made big changes to the one in Moodle. I now want to say more about what the question engine has to do, and how the new one does it.

The key processes

There are three key code-paths:

To display a page of the quiz:

1. Load the outline data about the student's attempt.
2. Hence work out which questions are on this page.
3. Load the details of this student's attempt at those questions within this quiz attempt.
4. Get the display settings to use (should the marks, feedback, and so on be visible to this user).
5. Taking note of the state of each question (from Step 3) update the display options. For example, if the student has not answered the question yet, we don't want to display the feedback now, even if we will later.
6. Using the details of the current state of each question, and the relevant display options, output the HTML for the question.

The bits in italic are the bits done by the quiz. The rest is done by the question engine.

To start a new attempt at the quiz:

1. From the list of questions in the quiz, work out the layout for this attempt. (This is only really interesting if you are shuffling the order of the questions, or selecting questions randomly.)
2. Create an initial state for each question in the quiz attempt. (This is when things like the order of multiple choice options are randomised.)
3. Write the initial states to the database.

To process a student's responses to a page of the quiz.

1. From the submitted data, work out which attempt and questions are affected.
2. Load the details of the current state of those question attempts.
3. Sort all the submitted data, into the bits belonging to each question. (The bits of data have names like 'q188:1_answer'. The prefix before the '_' identifies this as data belonging to the first question in attempt 188, and the bit after the '_' identifies this as the answer to that question.)
4. For each question, process its data to work out whether the state has changed and, if so, what the new state is. This is really the most important procedure, and I will talk more about it in the next section.
5. Write any updated states to the database.
6. Update the overall score for the attempt, if appropriate, and store it in the database.

These outlines are, of course, oversimplifications. If you really want to know what happens, you will have to read the code.

There are other processes which I will not cover in detail. These include finishing a quiz attempt, re-grading an attempt, fetching data in bulk for the quiz reports, and deleting attempts.

The most important procedure

This is the step where we take the data submitted by the student for one particular question and use it to update the state of that question.

Moodle has long had the concept of different questions types. It can handle short-answer questions, multiple-choices questions, matching questions, and so on. Naturally, what happens when updating the state of the question depends on the question type. That is true for both the old and new code.

Now, however, there is a new concept in addition to question types. The concept of 'question behaviours'.

In previous versions of Moodle, there was a rather cryptic setting for the quiz: Adaptive mode, Yes/No. That affected what happened as the student attempted the quiz. When adaptive mode was off, the student would go through the quiz entering their response to each question. Those responses were saved. At the end, they would submit the quiz and everything would be marked, all at once. Then the student could review their attempt (either immediately, or later, depending on the quiz settings) to see their marks and the feedback. When adaptive mode was on, the student could submit each question individually during the attempt. If they were right first time, they got full marks. If they were wrong, the got some feedback and could try again for reduced marks.

The problem with the previous version of Moodle was the way this was implemented. There was a single process_responses function that was full of code like "if adaptive mode, do this, else do that". It was a real tangle. It was very common to change the code to fix a bug in adaptive mode (for example), only to find that you had broken non-adaptive mode. Another problem was the essay question type, which has to be graded manually by the teacher. It did not really follow either adaptive or non-adaptive mode, but was still processed by the same code. That lead to bugs.

A very important realisations in the design of the new question engine was identifying this concept of a 'question behaviour' as something that could be isolated. There is now a behaviour called 'Deferred feedback' that works like the old non-adaptive mode; there is an 'Adaptive' behaviour; and there is a 'Manually graded' behaviour specially for the essay question type. Since these are now separate, you can alter one without risking breaking the others. Of course, the separate behaviours still share common functions like 'save responses' or 'grade responses'. We now also have a clean way to add new behaviours. I made a 'certainly-based marking' behaviour, and a behaviour called 'Interactive', which is a bit like the old Adaptive mode but modified to work exactly how the Open University wants.

It takes two to tango, and three to process a question

In order to do anything, there now has to be a three-way dance between the core of the question engine, the behaviour and the question type. Does this just replace the old tangle with a new tangle (of feet)? Fortunately there is a consistent logic. The request arrives at the question engine. The question engine inspects it, and passes it on to the appropriate behaviour. The behaviour inspects it in more detail, to work out exactly what need to be done. For example is the student just saving a response, or are they submitting something for grading. The behaviour then asks the question type to do that specific thing. All this leads to a new state that is passed back to the question engine.

So, the flow of control is question engine -> behaviour -> question type except, critically, in one place. When we start a new attempt, we have to choose which behaviour to use for each question. At this point the question engine directly asks the question type to decide. Normally, the question type will just say "use whatever behaviour the quiz settings ask for", but certain question types, like the essay, can instead say "I don't care about the quiz settings, I demand the manual grading behaviour."

If you like software design patterns, you can think of this as a double application of the strategy pattern. The question engine uses a behaviour strategy, which uses a question type strategy (with the subtlety that the choice of behaviour strategy is made by the question type).

Summary

So that is roughly how it works. A clear separation of responsibility between three separate components. Each component focussing on doing one aspect of the processing accurately, which makes the system highly extensible, robust and maintainable. Of course, everyone says that about the software they design, but based on my experiences over the last year, first of building all the parts of the system, and then of fixing the bugs that were found during testing, I say it with some confidence.

Introducing the new Moodle question engine

The rest of the (Moodle) world is eagerly anticipating Moodle 2.0, but I would like to tell you about what I have been doing for most of the last year, but which you won't be able to have until Moodle 2.1 - unless, that is, you are a student or teacher with the Open University, in which case you will be using it from this December.

What I have done is to rewrite a large chunk of the Moodle quiz system. What chunk is that? Well, first you can split a quiz system into two main parts. There is the quiz part, which says, "This quiz comprises these questions, and will be open to students between these dates". It tracks the student and they attempt the quiz, and stores their total score. Then there is the part that deals with the details of the individual questions within each quiz.

The question part can again be split in two. There is the question bank which lets the teacher create and store questions. For example "This is a multiple choice question where the student must select one of these three options, and it is an 'Elementary maths' question." Then there is the code that controls what happens when a student attempts a question "The student sees three radio buttons and a Submit button, and when they click the button we compute a score as follows and show this feedback." That second bit is what I call the question engine, and that is what I have rewritten.

However, you cannot just change the question engine in isolation. There are knock-on effects. For example, the quiz module still maintains overall control of things, even though it delegates a lot of the details to the question engine. So there are places where the quiz says things like "Dear question engine, please display this question now", or "Dear question engine, the student submitted this data, please process it", or "Dear question engine, the teacher wants to see all students responses to all questions in this quiz, give me the data to display." All those places have to change when the question engine changes.

There were also small changes required to the question bank. mainly because the new question engine has some new features that need some extra options stored with each question. So, the question bank needs to store the new options; let teachers edit them; back them up and restore them; import them and export them; and so on.

Altogether, my year's work added about 52,000 lines of new code and removed about 25,000 lines of old code (or, if you prefer, added 27,000 lines and altered 25,000 lines). At least that is the size of the change that I committed to the OU's CVS server last Friday, just in time to make the feature-freeze for the December update of our VLE. For comparison, the whole of Moodle 2.0 is about 1,600,000 lines of code, although that includes several large third-party libraries.

I am sure that there will be some minor bugs still to be found and fixed, but this new code has already had extensive testing from my colleagues Phil Butcher and Paul Johnson, so I am confident that the remaining bug-fixes will be minor.

There is much more I want to write about the new question engine, but I think this introductory post is already long enough. Therefore, I will split the remainder of what I want to say into separate posts which I hope to publish over the next week or so.

On the other side of the fence

I've been having problems with my teeth recently. Fortunately, none of it has been painful, but cycling home from yet another dentist's appointment this afternoon, I suddenly had a thought that the experience I was having with my dentist might be a bit like the experience of a non-developer who encounters a bug in the Moodle quiz. Allow me to explain.

I think it has been a total of four trips to the dentist over the summer. First I went for a regular check-up. That revealed that one of my fillings was cracked, so there was another appointment to drill it out and redo it. Then a filling in another tooth fell out one Friday evening, so I could not get anything done about it until the following week, and over the weekend a bit of the tooth next to the hole broke off, which was really worrying. (Luckily, as I said above, it was not painful.) So anyway, that required a big filling to fill the resulting hole. Then part of the new filling broke off, so today's appointment was to re-do the missing bit. Hopefully that remedial fix will work. If the new bit breaks off again, I will have to go back to have the whole filling drilled out and redone.

With all this going on, I fear I have been starting to have unkind thoughts like: "Is my dentist competent?" "Should so many things go wrong over a few months?" "How do I know if this is normal?". This is exacerbated by what appears to me to be a slightly casual attitude on her part. I expect that these really are routine problems, and rather boring to her. I, however, am worried, so I would have appreciated a more concerned-seeming bedside manner.

As I say, it was just after I had climbed on my bike to go back to work this afternoon, that I had the epiphany that this is probably how someone feels when they come to the Quiz forum after encountering a problem in one of their Moodle quizzes. To them it seems like some terrible problem that has them really concerned. They describe their symptoms, and I read it and think "oh yes, the problem is probably in that bit of code, let me do a quick fix." And then maybe I screw up and introduce a regression, but when that is pointed out, it too is easily fixed. To me it seems like some routine and minor matter, but I have never really thought how the process of fixing bugs feels to someone who is not a software developer, and who does not really understand what is going on. Will they still trust Moodle?

Now that I have thought about it, will I do anything? I fear I am unlikely to change my 'bedside manner'. My time seems to be more than taken up with actually doing the bug fixes, and other development, to spend too much time being nice to people. Still, I will try to make a bit more effort in future know I now how it feels to be on the other side of the situation.

Before finishing, allow me to point out (particularly to American readers) that all these trips to the dentist have cost me very little or nothing. In particular, the two recent appointments to replace the filling that had (partly) fallen out cost me nothing. Thank you NHS. I hope the new government does not cut you to death.

While I am writing, I will also share the news that I finally bought an apartment of my own over the summer, after years of renting. It is very nice, and worth the hassles of dealing with solicitors. Hopefully now that the stresses of dealing with the move are behind me I can concentrate more on Moodle development, although there are still a few minor things to deal with like getting some more furniture. Over the weekend I ordered some new sofas, including some bright orange cushions. I can't think where I got the idea for such a daring colour ;-)