Is learning design like UML?

A couple of weeks ago, I attended the #design4learning conference, which was conveniently on my doorstep at the Open University. Jenny Gray has already written her summary of the conference (and she though she was a bit late writing it up!)

I would like to highlight the point the organisers made with the conference name. Calling learning design "learning design" is a misnomer. You cannot design learning. Learning is something that goes on inside the student's head, perhaps most effectively under the support and guidance of a teacher. Therefore, you can only "design for learning", whatever it is you are designing: a course, a activity, a learning community, …. I think this is more than just semantic pedantry. We should all remember this, particularly when thinking about educational technology. There is no magic bullet that guarantees learning will occur. Just things that are more or less likely to encourage students to learn. (Having said this, I am going to just write "learning design" in the rest of this post, since it is so much easier!)

The main thought I wanted to share here is, however, something else. After two interesting days at a conference all about learning design, I cannot recall a single diagram shown by any speaker where I thought, "that is a graphical representation of the design of a bit of learning." Was I right to expect to see that? I don't know, but I have seen other presentation about tools like CompendiumLD in the past so I know it can be done. Pondering this as I cycled home, I got to thinking about the type of design I do know about: design of software, and though of an interesting comparison.

Software developers have a well-established way to draw the design of their software, called UML (better description on Wikipedia). Let me say immediately that I am not trying to suggest UML as a way to represent learning designs. Rather, I think it is interesting to think about how developers do (or more often don't) use UML to help their work. Can that tell us anything about how and whether teachers might engage with learning design?

There are two different ways to use UML. There is the quick-and-dirty, back-of-the-envelope way, where you draw of a part of the system to help explain or communicate a particular aspect of your design. This is the way I use UML as can be seen, for example, in this documentation I wrote. You include the details that are relevant to making your point, and leave out anything that does not help.

The other way to use UML is much more elaborate. It is called "Model Driven Architecture" which I studied as part of OU module M885. In MDA, you try to draw complete diagrams of the design of your system using a very precise dialect of UML, dotting all the 'i's and crossing all the 't's. Then, using a software tool (that you probably had to buy at great expense) you press the magic button, and it creates all your classes and interfaces for you. Then you just need to fill in all the implementations. At least, that is the promise. As I say, I studied this as part of a postgraduate computing course. It was of some academic interest, but I have never seen anyone write software this way (though a some people do, if the references in the course are to be believed). I expect more people have bought expensive MDA tools than have actually used them. In a previous generation, the same was true of CASE tools that also failed to live up to their promises.

So what, if anything, can this tell us about learning design? Well, I can see exactly the same split happening. There will be hype about magic systems where you input your learning outcomes, and sketch your learning design, press a magic button, and hey, presto, there is your Moodle course. It won't work outside of research labs, but some vendors will try to commercialise it, and a some institutions will fall for it and end up with expensive white elephants.

On the other hand, it would be good to see a common notation emerge to represent learning designs. This would help teachers communicate with each other, and perhaps with students, about how their teaching is supposed to work. A good feature of UML is that it is really very natural. Most developers can understand most of a UML diagram without having to be taught a lot of rules. There are several types of diagram to represent different things, but they are the kinds of things people drew anyway before UML was invented. The creators of UML just picked one particular way of drawing each sort of diagram, and endorsed it, in an attempt to get everyone talking (drawing) a common language. If you want to draw highly detailed UML diagrams, you need to learn a lot of rules, but you can get a long way just by copying what you see other people do, which is a sign of an effective language. It would be nice to see such a language for communicating about learning.

What makes something a horrible hack?

Over in Moodle bug MDL-42974, Derek Chirnside asked "What is it about a hack that makes it 'horrible'??". I had described the code sam wrote to fix that issue in those terms, while at the same time submitting it for integration. It was a fair enough comment. I had helped sam create the code, and it was the kind of code you only write to make things work in Internet Explorer 8.

Although "Horrible hack" is clearly an aesthetic judgement, and therefore rather subjective, I think I can give a definition. However, it is easier to start by defining the opposite term. What is "Good code"? Good code should have properties like this:

1. It works: It does what it is supposed to.
2. It is readable: Another developer can read it and see what it is supposed to do.
3. It is logical: It does what it is supposed to do in a way that makes sense. It is not just that a developer can puzzle out what it does, but it is clear that it does just that and nothing else.
4. It is succinct: This is a companion to point 3).
5. It is maintainable: It is clear that the code will go in working in the future, or if circumstances do change, it is clear how the code could be modified to adapt to it.

Note that property 1) is really just a starting point. It is not enough on its own.

A horrible hack is code that manages little more than property 1. I think sam's patch on MDL-42974 scores a full et.

2. It is not at all obvious what the added code it for. Sam tried to mitigate that by adding a long comment to explain, but that is just a workaround to the hackiness of the code.
3. There is no logical reason why the given change makes things work in IE <= 8. We were just fiddling around in Firebug to try to find out how IE was going wrong. Changing the display property on one div appeared to solve the display problem, so we turned that into code. We still don't really understand why. Another sign of the illogicality is the two setTimeout calls. Why do we need those two delays to make it work? No idea, but they are necessary.
4. The whole chunk of added code should be unnecessary. Without the addition, it works in any other browser. We are adding some code that should be redundant just to make things work in IE.
5. We don't understand why this code works, so we cannot understand if it will go on working. In this case, lack of maintainability is not too serious. The code only executes on IE8 or below. In due course we know we can just delete it.

Normally, you would wish to avoid code like this, but in this case it is OK because:

• The hack is confined in one small area of the code.
• There is a comment to explain what is going on.
• It is clear that we will be able to remove this code in future, once usage of IE8 has dropped to a low enough level.

At least, we hope that the Moodle integration team agree that this code is acceptable for now. Otherwise, we wasted our time writing it.

Load-testing Moodle 2.6.2 at the OU

At the start of June we will upgrade the OU Moodle sites to Moodle 2.6. Before then we need to know that it will still perform well when subjected to our typical peak load of 100,000 page-views per hour. This time, I got 'volunteered' to do the testing.

The testing servers

To do the testing, we have a set of 10 servers that are roughly similar to our live servers. That is six web servers for handling normal requests, one web server that handles 'administrative' requests. That is, any URL starting /admin, /report or /backup. Those pages are often big, long-running processes, rather than quick page views, so it is better to put them on a different server that is tuned differently. There is one 'web' server is just for running the cron batch processes. Finally we have a database server and a file server.

In order to be able to make easy comparisons, we make two copies of our live site onto these servers. That is, we have two different www_root folders, which correspond to different URLs lrn2-perf-cur and lrn2-perf-upg. In due course we will upgrade one of the copies to the new release while leaving the other open running the current version of the code. This make it easy to switch back and forth when comparing the two.

In addition to the servers running Moodle, we have 6 virtual machines to generate the simulated load.

The testing procedure

We test using JMeter. In order to test Moodle, you need to send lots of requests for different pages, many of which include numeric ids in the URLs. Therefore, the JMeter script needs to be written specifically for the site being tested. Fortunately, our former colleague James Brisland made a script that automatically generates the necessary JMeter script. We shared that script with the community, and you can find a copy here. However, we shared it a long time ago, and since then our version has probably diverged from the community version a bit. Oops!

I say this tool automatically generates the necessary JMeter script, but sadly that is an oversimplification. It fails in certain cases like if a forum is set to separate groups mode. So, having generated the JMeter script, you need to run it and check that it actually works. If not, you have to go into the courses and activities being tested and modify the settings. We really ought to automate that, but no one has had the time. Anyway, eventually (and this took ages) you have a working test script.

Tuning the test script

Once the test script works, in that it simulates users performing various actions without error, one at a time, then you have to start running it at high load. That is, simulating lots of users doing lots of things simultaneously. After it has settled down, you let it run for 15 or 20 minutes, and then look at what sort of load you are generating. The goal is to get about the same number of requests per second for each type of page (course view, forum view, post to forum, view resource, ...) in the test run as in real use on the live system. If not, you tweak the time delays, or number of threads, and then run again. It took about four runs to get to a simulated load that was close (actually slightly higher) than the target request rates we had taken from the live server logs.

All that creation and tuning of the tests scripts is done on the lrn2-perf-cur copy of the site. Once that is OK, then you run the same script against lrn2-perf-upg. That should give exactly the same performance, and before proceeding we want to verify that is the case. It turned out at first that it was slightly different. I had to find the few admin settings that were different between the two servers. Once the configuration was the same, the performance was the same, and we were finally in a position to start comparing the old and new systems.

Upgrade to the new version of Moodle

The next step is to upgrade lrn2-perf-upg to the new code. This code is still work-in-progress. Final testing of the code before release happens next month, but we try to keep our code in a releasable state, so it should be OK for load-testing. However, this is the first time we have run the upgrade on a copy of all our data. Unsurprisingly, we found some bugs. Fortunately they were easily fixed, and better to find them now than later.

Also, a new version of Moodle comes with a lot of new configuration options. This is the moment to consider what we should set them to. Luckily, most of the default values were right, so there was not a lot to do. Moodle prompts you for most of the new settings you need to make as part of the upgrade. However, it does not prompt you to configure any new caches, so you have to remember to go and do that.

Compare performance

At long last (about one and a half weeks into the process) you are finally ready to run the test you want. How does 2.6 performance compare to 2.5? Here is a screen-grab of today's testing:

Good news: Moodle 2.6 is mostly a bit faster (5-10%) than Moodle 2.5. Bad news: every 15 minutes, it suddenly goes slow for about 15 seconds. What?!

Problem solving

Actually, there is a logical explanation. We have cron set to run every 15 minutes, so surely the problem is caused by cron, right? No. Wrong! We stopped cron running, and the spikes remained. We tried various things to see what it might be, and could not make any sense of it. One thing we discovered was that the spikes were about as large as the spikes you get by clicking the 'Purge all caches' button. OK, so something is purging caches, but what?

To cut a long story short, you need to remember that our two test sites lrn2-perf-cur and lrn2-perf-upg are sharing the same servers. Therefore they are sharing the same memcache storage. It appears that something in cron in Moodle 2.5 purges at least some of the caches. When we stopped cron on our Moodle 2.5 site the spikes went away on our 2.6 site. I am afraid we did not try to work out why Moodle 2.5 cron was purging caches, but there is probably a bug there. It turns out that purge caches does not cause a measureable slow-down in Moodle 2.5, at least not for us, which is worth knowing.

Why does Purge caches cause a slow-down in 2.6 but not in 2.5? I am pretty sure the reason is MDL-41436. When things slowed down, it was the course page that slowed down the most, and that is the one most dependent on the modinfo cache.

Summary

• Moodle 2.6 is about 5-10% faster than 2.5, at least on our servers, which are RHEL5 + Postgres + memcache cluster store. (MDL-42071 - why has that not been integrated yet?)
• In Moodle 2.5, doing Purge caches when your system is running at high load seems to cause remarkably little slow-down.
• In Moodle 2.6, doing Purge caches does slow things down a lot, but only very briefly. Performance recovered within about 15 seconds in our test, but then the test was only using a few courses.
• In Moodle 2.6, clicking Clear theme caches (at the top of Admin -> Appearance -> Themes -> Theme selector) causes no noticeable slow-down.

The bit about what happens when you clear the caches is important because sometimes, when you patch the system with a bug fix, you need to purge one or more caches to make the fix take effect. In the past, we did not know what effect that had. We were cautious and had people waiting up until after midnight to click the button at a time of low system load. It turns out now that is probably not necessary. We can clear caches during the working day, when staff are in the office to pick up the pieces if anything does go wrong.

The four types of thing a Moodle developer needs to know

In order to write code for Moodle, there is an awful lot you need to know. Quite how much was driven home for me when I taught a Moodle developers' workshop at the German MoodleMaharaMoot in February. When preparing for that workshop, I though you could group that knowledge into three categories, but I have since added a fourth to my thinking.

1. The different types of Moodle plugin

The normal way you add functionality to Moodle is to create a plug-in. There are many different types of plug-in, depending on what you want to add (for example, a report, an activity module or a question type). Therefore, the first thing to learn is what the different types of plug-in are and when you should use them. Then, once you know which type of plug-in to create, you need to know how to make that sort of plug in. For example, what exactly do you need to do to create a new question type?

2. How to make Moodle code do things

Irrespective of what sort of plug-in you are creating, you also need to know how to make your code do certain things. Moodle is written in PHP, so generic PHP skills are a prerequisite, but Moodle has its own libraries for many common tasks, like getting input from the user, loading and saving data from the database, displaying output, and so on. A developer need to know many of these APIs.

3. How to get things done in the Moodle community

If you just want to write Moodle code for your own use, then the two types of know-how above are enough, but if you want to take full advantage of Moodle's open source nature, then you need to learn how to interact with the rest of the Moodle development community. For example how to ask for help, how to report a bug, how to submit the changes for a bug you have fixed or a feature you have implemented, get another developer to review your proposed code change, how to update your customised Moodle site using git, and so on.

4. Something about education

Those three points were what I thought of when trying to work out what I needed to teach during the developer workshop I ran. Since then, while listening to one of the presentations at the UK MoodleMoot as it happens, I realised that there was a fourth category of knowledge required to be a good Moodle developer. It matters that we are making software to help people teach and learn. I am struggling to think of specific concepts here, with URLs for where you can learn about them, as I gave in the previous sections, but there is a whole body of knowledge about what makes for effective learning and teaching and it is useful to have some appreciation of that. You also need some awareness of how educational institutions operate. If you hang around the Moodle community for any length of time you will also discover the educational culture is different in different countries. For example in the southern hemisphere the long summer holiday is also the Christmas holiday, and in America, they expect to award grades of more than 100%.

Summary

Does this subdivision into categories actually help you learn to be a Moodle developer? I am not sure, but it was certainly useful when planning my workshop. The workshop was structured around creating three plugins on the first day, a Filter, a Block and then a Local plug-in. However, those exercises were structured so that while moving through different types of category-one knowledge, we also covered key topics from categories two and three in a sensible order. So it helped me, and I thought it was an interesting enough thought to share.

Reflections on listening to conference presentations in German

I am at the MoodleMaharaMoot in Leipzig listening to people talk about Moodle.

First, the good news is that about half the words in English came from the same roots as German, so there are a fair number of words you can recognise, at least if you have time to read them from the screen. For words that seem really key, there is Google translate. Also, the Germans seems like using English phrases for eLearning-related things, like Learning Analytics, or Multiple Choice.

However, I don’t think I was even understanding 10% of the words. What really makes a difference to intelligibility is what is on the screen. If speaker just had powerpoint slides with textual bullet points, that does not help. If the speaker uses the screen to show you what they are talking about - screen grabs or live demos - that is much better. Of course, this is just: show, don’t tell.

It also makes a big difference whether you already know a little bit about what is being said. I talked to some people from University of Vienna two years ago when they started building their offline quiz activity, so I already knew what it was supposed to do. I followed that presentation (which contained many screen-grabs) better than most. What they have done looks really slick, by the way.

Regarding my presentation, I feel vindicated in my plan to spend almost all of the presentation doing a live demonstration of the question types I was talking about. Of course, I am sure that almost everyone in the audience has better English than I have German. Also, I apologies that I talked for the whole time, and did not leave an opportunity for questions.

Finally, I have been speculating (without reaching any conclusions) about whether the experience of sitting there, failing to understand almost everything that is being said, and just picking some scraps from the slides, is giving me any empathy for people with severe disabilities who need major accessibility support to use software? As I say, these thoughts are inconclusive. What does anyone else think?

By the way, Germans applaud by rapping on the table with their knuckles. Your trivia fact for the day.

Moving the OU Moodle code to Moodle 2.6.1

I spent today upgrading our Moodle codebase from Moodle 2.5.4 to Moodle 2.6.1. This is the start of work towards our June release of the VLE. We have a March release based on Moodle 2.5.4 to get on the live servers first, and testing that will overlap with the development of the 2.6.1-based version.

Doing the merge

The first stage of the process is to merge in the new code. This is non-trivial because even if you just do

git checkout -b temp v2.5.4
git merge v2.6.1

Then you will get a lot of merge conflicts. That is a product of how the Moodle project manages its stable branches. If your own code changes also lead to other merge conflicts, then sorting out the two is a real mess.

Fortunately, there is a better way, because we know how we want to resolve any conflicts between 2.5.4 and 2.6.1. We want to end up with 2.6.1. Using git merge strategies, you can do that:

git checkout -b merge_helper_branch v2.6.1
git merge --strategy=ours v2.5.4

That gives you a commit that is upstream of both v2.5.4 and v2.6.1, and which contains code that is identical to v2.6.1. You can verify that using git diff v2.6.1 merge_helper_branch. That should produce no output.

Having built that helper branch, you can then proceed to upgrade your version of the code. Our version of Moodle lives on a branch called ouvle which we originally branched off Moodle 2.1.2 in October 2011. Since then, we have made lots of changes, including adding many custom plugins, and merging in many Moodle releases. Continuting from the above we do

git checkout ouvle
git merge --strategy-option=patience merge_helper_branch

That gave a lot of merge conflicts, but they were all to do with our changes. Most of them were due to MDL-38189, which sam marshall developed for Moodle 2.6, and which we had back-ported into our 2.5 code. That back-port made a big mess, but fortunately most of the files affected did not have any other ou-specific changes, so I could just overwrite them with the latest versions from v2.6.1.

git checkout --theirs lang/en backup lib/filestorage admin/settings/development.php lib/form/form.js
git add lang/en backup lib/filestorage admin/settings/development.php lib/form/form.js

Simiarly, we had backported MDL-35053 which lead to more conflicts that were easy to resolve. Another case was the Single activity course format which we had used as an add-on to Moodle 2.5. That is now part of the standard Moodle release. The change caused merge conflits, but again there was a simple solution: take the latest from 2.6.1.

After all that, there were only about 5 files that needed more detailed attention. They were mostly where a change had been made to standard Moodle code right next to a place where we had made a change. (Silly rules about full stops at the ends of comments!) They were easily to fix manually. The one tricky file was in lib/moodlelib.php where about 400 lines of code had been moved lib/classes/useragent.php. There were two ou-specific changes in the middle of that, which I had to re-do in the new version of that code.

Verifying the merge

Having resolved all the conflicts, it was then time to try to convince myself that I had not screwed anything up. The main check was to comparing our ouvle code with the standard 2.6.1 code. Just doing git diff v2.6.1 ouvle does not work well because it shows all contents of all the new files we have added. You need to read the git documentation and work out the incantation

git diff --patience --diff-filter=CDMRTUXB v2.6.1 ouvle

That tells git to just show changes to existing files - the ones that are part of standard Moodle 2.6.1. That is a manageable amount of output to review. We have a strict policy that any change to core Moodle code is marked up like this:

// ou-specific begins #2381 MDL-28567
/*
        $select = new single_select(new moodle_url(CALENDAR_URL.'set.php',
                array('return' => base64_encode($returnurl->out(false)),
                        'var' => 'setcourse', 'sesskey'=>sesskey())),
                'id', $courseoptions, $selected, null);
*/
        $select = new single_select(new moodle_url(CALENDAR_URL.'view.php',
                array('return' => $returnurl, 'view' => 'month')),
                'course', $courseoptions, $selected, null);
// ou-specific ends #2381 MDL-28567

That is, the original Moodle code is still there, but commented out, alongside our modified version, and the whole thing is wrapped in paired begin and end markers that refer to a ticket id in our issues database and if applicable a Moodle tracker issue. In this case I can check that MDL-28567 has still not been resolved, so we still need this ou-specific change. What I am doing looking at the diff output is verifying that every change is marked up like that, and that any issues mentioned are things that are still relevant.

The other check is to search the whole codebase for ou-specific and again review all the issue numbers mentioned. These combined checks find a few ou-specific changes that are no longer needed, which is a good thing.

What happens next

Now that I think the code seems right, it is time to test it, so I upgrade my development install. It mostly works, except that our custom memcache session handler no longer works (the session code seems to have changed a lot, including an official memcached session hander in core). For now I just switch back to default Moodle sessions, and make a note to investigate this later.

Apart from that, the upgrade goes smootly, and, apart from thousands of debugging warnings about use of deprecated code, I have a working Moodle site, so I push the code to our git server, and warn the rest of the team that they can upgrade if they feel brave.

The next thing, which will take place over the next few weeks is to check every single one of our custom plugins to verify that it still works properly in Moodle 2.6. To manage that we use a Google Docs spreadsheet that we can all edit that lists all the add-ons, with who is going to be responsible for checking it, and whether they have done so yet. Here is a small section.

The state of OU Moodle customisation

Our regular code-merges are a good moment to take stock of the extend to which we have customised Moodle. Here are some headline numbers:

• 212 custom plug-ins: Of those 10 are ones we have taken from the community, including Questionnaire, Certificate, Code-checker and STACK (we helped create those last two). Of our own plugins, 58 (over a quarter) are shared with the community, though the counting is odd because ForumNG contains 20 sub-plugins.
• 17 ou-specific issues: That is, reasons we made a change to core code that could not be an add-on.
• Due to those 17 reasons, there are 42 pairs of // ou-specific begins/ends comments in the code.

So, we continue to be disciplined about not changing core code unless we really have to, but the number of plugins is getting a bit crazy. A lot of the plugins, are, however, very small. They just do one thing. Also, we run a range of very different sites, including OpenLearn, OpenLearn works, The Open Science Lab and our exams server. A significant number of our plugisn were just designed to be used on one of those sites.

Here are the numbers of custom plugins broken down by type (and ignoring sub-plugins of our custom plugins).

Plugin type	Number
Activity module	25
Admin tools	8
Authentication methods	2
Blocks	30
Course formats	3
Editors	1
Enrolment methods	1
Filters	6
Gradebook reports	1
Local plugins	44
Message outputs	2
Portfolio outputs	1
Question behaviours	4
Question types	14
Quiz reports	6
Quiz access rules	2
Reports	19
Repositories	3
Themes	9
TinyMCE plugins	1