Tuesday, September 27, 2011

What I want to build next

Earlier this summer I finally finished the new Moodle question engine, which was released as part of Moodle 2.1. As you might expect with such a large change, a number of minor bugs were not spotted until after the release, but I (and others) have fixed quite a lot of them, and we will continue to fix more. I want to say "thank you" to everyone who has taken the time to report the problems they encountered. Pleasingly, some people, including Henning Bostelmann, Tony Levi, Pierre Pichet, Jamie Pratt, Joseph Rézeau and Jean-Michel Vedrine have not only been sending in bug reports, but also submitting bug fixes. I would like to thank them in particular. I don't know whether this means that the new Moodle development processes are working well and encouraging more contributors, or that I released the new question engine full of trivial bugs.

At the moment, apart from fixing bugs, we are about two months away from the end of the OU's one-year project to move from Moodle 1.9 to 2.x and implement a lot of new features at the same time. In the eAssessment area, we had about 30 work-packages to do, of which finishing the question engine was by far the biggest, and we have about 6 left to go. Most of the remaining tasks are at least started, but finishing them is what I, and the developers on my team, will be doing in the near future.

I have, however, been thinking ahead a bit, and I have an idea for what I would like to build, should I be given the opportunity. Honesty compels me to say these are not my ideas. I stole them from other people, and there are proper acknowledgements at the end of this post. I wanted to post about this because: 1. in my experience, if you post about your half-baked ideas, people will be able to suggest ways to make them better; and 2. I am hoping that at least one course-team at the OU will see this and say "we would love to use this in our teaching" because that might persuade the powers that be to let me build this.

Rationale

The Moodle quiz is a highly structured, teacher-controlled tool for building activities where students attempt questions. What I want to create is a more open activity where students can take charge of their learning using a bank of questions to practice some skill where the computer can mark their efforts and give feedback. For the sake of argument, I have been calling this the "Question practice" activity module.

The entry page

When a student goes into a Question practice activity, they see a front screen that lists all the categories in the question bank for this activity.

Next to each category, there are statistics for how the student has performed on that category so far. For example, it might say "recently you scored 19/21 (90%); all time you scored 66/77 (86%).” The categories are nested, and there is a subtotal for each category.

At the bottom of the page is an Attempt some questions… button. This takes the student to the …

Start a session form

… where they set up what practice they would like to do. Students can select which categories they want to attempt questions from. They may also be able to choose how many questions they want. For example "Give me 10 questions", "As many as possible in 20 minutes", or "Keep going until I say stop". The teacher will probably be able to constrain the range of options available here.

Once they are satisfied, the they clicks the "Start session" button. This takes them to the …

Attempt page

… which shows the student the first question, chosen according to the criteria they set. There will probably be a display of running statistics "In this session you have attempted 0 questions so far". The question will contain the controls necessary for attempting the question. There will also probably be a "Please stop, I'm bored" button, so the student can leave at any time.

When they get back to the front page, the statistics will have been updated.

If the student crashes out of a session, then when they go back in, the front page will have a "Continue current session" button.

Overall activity log

One batch of attempting questions will be called a 'practice session'. The system will keep track of all the sessions that the student has done, and what they achieved during each session.

The front page will have a link to a page that lists all of the student's sessions, showing what they achieved in each. This provides more detail than is visible on the front page.

Possible extensions

That is the key idea. Here are some further things that could be added to the basic concept.

Milestones

The system could recognise targets, goal, or achievement (I'm not sure of the best name). That would be something like "Attempt more than 10 questions from the Hard category, and score more than 90%". If the student achieves that target at any time, they system would notice, and the achievement would be recorded on the front page and in the session log in an ego-boosting way (e.g. a medal icon).

The whole point of this activity is to be as student-driven as possible, so should students be able to define their own targets or goals? Should students be able to set goals for each other?

Locks / Conditional access

The activity could also have locks, so that the student cannot access the questions in the Multiplication category until after they have scored more than 80% in the Hard addition category. Of course, unlocking a new category could be an achievement. We appear to be flirting with the gamification buzz-word here, so I will stop.

Performance comparison

Should there by any way for students to compare their performance, or achievements, with their peers? We are definitely getting to features that should be left until version 2.0. Let's get a basic system working first, but make sure it is extensible.

How hard would this be to build

I think this would not require too much work because a lot of the necessary building blocks already exist in Moodle. The question bank already handles questions organised into categories, and we would just use that. Similarly, the attempt page and practice sessions are very easy to manage with the new question engine.

The real work is in two places. First, building the start attempt form, and then writing the code that randomly selects questions based on the options chosen. Second, deciding what statistics to compute, and then writing the code to compute them.

Of course, before we can start writing any code, there are still a lot of details of the design to decide. Also one most not forget things like backup and restore, creating the database, and all the usual Moodle plumbing.

Overall, I think it would take a few months work to get a really useful activity built.

Credit where credit is due

I said earlier that I got most of these ideas from other people. To start with, things like this have been mooted in the Moodle quiz forum over the years. The discussions there usually start from Computerised Adaptive Testing, whereas this idea is about student-driven use of questions. I think the latter is more interesting. (As a mathematician, I think CAT is an interesting concept. I just don't think it would make a useful Moodle activity.)

The real inspiration for this came at a meeting in London at the start of 2011. That meeting was at UCL with Tony Gardiner-Medwin who has already built a system something like this, but stand-alone, not in Moodle; and David Emmett from University of Queensland, Brisbane (who was giving a seminar). David had been hoping to get a grant to build something like this proposal (in Moodle) but that did not pan out. We did, however, have a very interesting discussion, and that is where I got the key idea that this sort of question practice was most interesting if you could give the student control of their own learning as much as possible.

We have also discussed ideas like this on-and-off for a long time at the OU. There has, however, been a lot of other things we needed to deal with first. We had to do a lot of work getting the quiz system working to our satisfaction (a strand of work that eventually lead to the new question engine). We had to sort out the reporting of grades, including working with Moodle HQ on the new gradebook in Moodle 1.9, and integrating Moodle with our student information system. We had to make a new question types that our users wanted. Only now can we start to think seriously about the last piece of the jigsaw: more activities that use all the question infrastructure we have built. I hope this post is a useful starting point for discussing what one of those activities might be.

9 comments:

  1. Excellent! Please use the GIFT extensions we discussed at http://microformats.org/wiki/gift -- the Mediawiki Foundation will be using them (or at least a couple of their volunteers have committed to them.)

    ReplyDelete
  2. Tim,

    I have begun exploring some of the new question types in 2.1. Just last week, I used the "calculated multiple choice" and even tried a common wildcard database. This was a statistics exam, and it went fairly well, but I did encounter some problems with the common database. I will probably be posting a bug on the Moodle tracker as soon as I can confirm the problem.

    I really want to explore these new question types to make sure that I am using them correctly. The documentation still needs improvement, as you probably know. Especially the "synchronize" features.

    Well, I haven't had time to think ahead as you have done. My time will be devoted to understanding, and using, what is already there. I really like what I see, now I just need to use it more effectively.

    ReplyDelete
  3. how about a student centred question system fro both practice, critique and as a source of more questions.... students could: rank questions according to how difficult they thought it was, b. students could add comments i.e. it was difficult because, it was easy because, i didn't understand this aspect etc., c. let students contribute their own questions. that could even be set an an assignment 'give us 5 questions you would like to see in the final exam'. perhaps with the obligation to comment on other students questions too. have the ability to group by contributor, show/hide/make anonymous contributors of questions/comments. links to grade book for question answers if the teacher wanted to use it for formative assessment and also to award marks for contributing a question e.g if the teacher wanted to award marks quality of questions or for best written question by students etc. the teacher to be able to mix and match questions from students and teacher produced questions to make up new quizzes or pools of questions, and to transfer students questions to question banks and visa versa.
    p.s. im also at UQ!

    ReplyDelete
  4. James, of course! you were another of the people I talked to over the years who gave me ideas that fed into this. Sorry for missing you out of the post.

    ReplyDelete
  5. Rick, better to discuss existing Moodle features in the Quiz Forum, I think.

    Matthew, rating and commenting on questions are yet more possible version 2.0 features for this plugin. Question-creation as an activity is already handled by Jamie Pratt's Question creation module (which is currently only available for Moodle 1.9). Question creation can be a valuable learning task, but I am not sure if it should be built into the Question practice module, or kept as a separate activity. Of course, the question bank should make it easy to copy new questions from the Question creation module to the Question practice module (but it doesn't at the moment).

    ReplyDelete
  6. Hi Tim,

    Here are some thoughts on further extensions:

    It might be interesting to link Question practice statistics to Moodle 2 cohorts. You could then open the way to ranking/comparison within a school/college e.g ...most improved maths student in key stage 3 is “Tim... he has achieve this by putting in the time & completing x number of question in x categories”.

    Also you could use the overall activity log to promote particular questions. eg “nobody has got this question correct - ever!” or “nobody has got this question correct on 1st attempt... within 3 minutes” etc

    Further - A grader view of all the questions a particular student has attempted across all the courses in a particular moodle would be generally useful in analysing knowledge, understanding, preferences… http://blog.thelearner.net/2011/05/servingandincludingyourcommunites-slides-11-14/

    Thanks for posting your thoughts online

    Jago Brown

    ReplyDelete
  7. Hi Tim

    I am currently doing some work on Moodle and wish to use a feature where you must complete a course before beginning another one (which i know you have commented on).

    Course prerequisites are a fantastic idea but i think they missed a trick by not allowing you to enable students to complete a whole course before beginning another.

    Did anybody manage to write a module plugin to achieve this or is it just the editing of the original file? If anybody does know of an enrollment module plugin to achieve this then can you please let me know as it will save me having to give out an enrollment key to users once they complete the first course.

    Many thanks,

    Mark

    ReplyDelete
  8. Mark, good question, but this is not the place to ask it. Try asking it at http://moodle.org/mod/forum/view.php?id=55

    ReplyDelete
  9. Hi Tim,
    According to my knowledge what currently you are using to show the results of the students is a table.Instead of that can't we use a graph so that the student can easily see his improvement when comparing with the previous attempts also it is more interesting to look at.
    Another suggestion is to develop a form where students also can add questions to the question base, So other students can try them.

    ReplyDelete