Keeping the discipline of not changing Moodle core

We have said in the past that at the OU we made too many changes to core code in our Moodle 1.9 system, and that as we moved to Moodle 2, we would make far fewer. The release of Moodle 2.1 provides a good opportunity to stop and reflect on how we are doing.

Exactly how many core changes we had made in 1.9 seems to depend on who you ask. It was something of the order of one or two thousand depending on how you count. As a result, every time there is a new Moodle 1.9.x release, someone (Derek) has to do a couple of days painstaking merging to upgrade to the new version.

Moodle 2.1 was released on Friday. On Monday afternoon we decided to try upgrading our development branch to it. The merge (literally git merge MOODLE_21_STABLE) only took a few hours, and that was most mostly a matter of thinking before typing git checkout --theirs to resolve most of the conflicts in favour of the Moodle 2.1 code. Then we had to test test install, upgrade, and basic functionality before pushing the merge to our central repository.

But, how many OU-specific changes do we have in core code right now? Well, the answer appears to be eight. Let me explain that number.

To control the core code changes, we use a simple approval procedure. Each change must be proposed by one of the leading developers. They do this by opening a special sort of ticket in our issue tracking system. That serves as a permanent record of the change, and is also a place to log any discussion. The other leading developers then review the proposal. For the change to be approved, at least one other leading developer must endorse it with a +1 vote, and there must not be any -1 votes. Votes are normally accompanied by an explanation of why that developer is voting that way.

After a suitable time for votes, the issue is declared either accepted or rejected. OU-specific changes can be rejected for two reasons.We may decide that it is not acceptable to change core to implement this feature, so we drop the feature; or we think of some devious way to achieve the feature without changing core code.

If a change is approved, then the code is written. Well, in some cases the code will already have been written, because you can have a much more informed debate about whether a certain change is a good idea if you can see exactly what the proposed change is. Once the code is written and approved, it is committed to out git repository and the issue moves into state 'Code committed'. Finally, we may find a way to get rid of the OU-specific change in future. The most common way that happens is if we contribute the change upstream to moodle.org. For example the new Moodle question engine was an ou-specific change as long as we were using it in Moodle 2.0, but now we have upgraded to Moodle 2.1, it is standard code. Therefore, that issue has now changed status to 'No longer required'.

Overall, our we, have 22 ou-specific change issues in our bug tracker. The break-down is:

Rejected: 2
New (under discussion): 4
Approved (but not yet implemented): 1
Code committed: 8
No longer required: 7

Most of the 'Code committed' changes are pretty boring. For example three of them are bug-fixes to the questionnaire module that we have submitted upstream, but which have not been reviewed and accepted by the questionnaire maintainers yet. Therefore, those three will almost certainly end up as 'No longer required' in due course. Another example is that we want to customise the "Database connection failed / It is possible that the database is overloaded or otherwise not running properly" page that you get when Moodle fails to connect to the database. If Moodle can't connect to the database, then it cannot load the configuration, and so cannot determine which theme to use to display the error. Therefore, the only way customise that page is to edit lib/setup.php.

The one 'serious' ou-specific change we have is some hacking around in course/lib.php to support one of our custom modules called 'subpage' (not released yet, but we hope to share it eventually). Given more time, we might be able to find a more elegant way to handle these changes, but we don't have that sort of time at the moment.

While we have controlled the core code changes, we do have written a lot of custom plugins. Those range from big things like forumng and ouwiki, to small things like a local plugin that just implements a single web-service function. I'm afraid I don't have a complete list, but we must have more than 50 plugins by now. As far as I am aware, the upgrade from 2.0 to 2.1 did not break any of them.

Performance and Scalability

When you set up a web application, often you will start small with everything running on one server. Everything, in this case, typically means the application and the database and data files. That is nice and simple. It has the advantage that everything it fast because it is all one the one server.

The capacity is, however, limited. Suppose the load on your application increases. You can get some way just by upgrading the one server, adding more memory and faster processors, but that will only get you so far.

Eventually, you will have to scale out. You will get a number of separate web servers, with a load-balancer to distribute the incoming requests between them. All the web servers will connect to a shared database sever, or cluster of database servers. The files will probably go on a separate file server.

While this increases the total load that the whole system can support, it means, paradoxically, that processing a single request is slower. For example, if you switch from one server to three servers (application server, database and files) your site will not support three times as many users. The scalability will not be linear. That is because every connection to the database or to get a file now has to travel over the network. Accessing something across a network tends to be an order of magnitude slower than accessing something on the same server.

The above is all standard knowledge about scaling web applications. I have been thinking about about it recently because it explains the way my working life has been evolving. Just over six months ago I was working essentially on my own, re-developing the Moodle question engine. I had been working away like that for a year, and I had got a lot done.

Since then, things have changed, and I am now managing a team including three other developers, and two out-sourced development contracts. It has been particularly 'bad' this last couple of weeks as one development period of the project came to an end and I had to review a lot of code, and then I had to sort out everything we were supposed to be doing for the next three months. I am starting to wonder if I will every get any of my own development work done at all!

That is, however, just some exasperation showing. I know really that this has just been a brief spell with an excessive amount of administration. Overall I am happy that the OU is investing so much in its eAssessment systems (and the other parts of its VLE); as a team we are achieving more than I could on my own; but right now my inner geek would really like to go and hide in a cave for a while and just write code undisturbed.

Herding cats

Just before I left Perth, Martin advertised for a Project Manager for Moodle HQ. I think this is probably a good thing, if a good person applies and is appointed. However, I also think it will be a very difficult job to do.

Martin always has too much to do, and any time he can identify some tasks that can be delegated, it is a good thing. Early in the history of Moodle HQ he found Mike and Darlene who run a lot of the commercial side of the Partners network. More recently he appointed Helen to act as community co-ordinator, and Jordan as system admin. All those were successful appointments, and having watched form the sidelines I have seen that Martin will not recruit someone unless he is sure they are right for the job (important when you are running a small business) and Martin is a good judge of people.

The other reason a project manager would be a good idea is clear to anyone who had been following the Moodle 2.0 development, which still has a long way to go until it is finished. Someone with more time than Martin has to ride herd on the process can only help us get 2.0 out of the door.

However, it is not quite that simple. On the one hand, I have studied the Open University Project Management course from their Postgraduate Computing program. That is an excellent course, and as a result I know a bit about and am interested in what project managers do. On the other hand, I also know the fairly independent way that I and other core developers like to work. That is why I chose the title of this blog post. (Not because of some people's user profile pictures!)

To start with, the Moodle project is not even a project in the sense that Project Managers use the word. A typical definition is "A project is a temporary endeavor, having a defined beginning and end," but Moodle itself will not end (we hope). Still, some of the things we do, for example, the Moodle 2.0 release, are projects in this sense. However, we know there will also be another project called the Moodle 2.1 release later. Therefore, if we contemplate a short-cut that would let us release Moodle 2.0 sooner, we have to balance the short-term benefits with the long-term costs of cleaning things up in a later release. Classic project management tends to think about one project at a time.

That ties in with Moodle's choice of feature-driven, not date-driven releases. We will pick a set of features to be included in a release, and we will release when they are done, tested and bug-fixed, however long that takes. Well that is the general policy, but a certain amount of pragmatism is applied when interpreting it. This is a fairly unusual way to run a project. For example, with the Open University's Moodle-based VLE we operate the other way. We have date-based releases, with the dates announced a long way in advance, but we do not guarantee what features will be an any given release very far in advance.

Then there is the the nature of the work. One area where project management is generally successful is in construction projects. The cliché old maths problem "If it takes 2 men 7 hours to dig a trench 6 metres long ..." has at least some relevance to reality there. There is even the Bluebook cost guide that has established estimates for certain types of construction work.

In contrast, as that classic of software project management "The Mythical Man Month" says, "adding manpower to a late software project makes it later." What software developers do is much closer to an architect drawing the plans than to a builder laying bricks. The compiler (or PHP interpreter) is the builder. This is hardly new, and is why agile software-development methodologies were invented. If Moodle's new project manager is going to adopt a Methodology, I hope the choose an agile one.

Another difficulty is that software developers are not fungible. If there is some tricky job involving security you probably want Petr to do it. If it is to do with the quiz, it would be best if I did it. If it is related to backup and restore, no one wants to do it, and so it ends up with Eloy ;-). If those people do the job, it will be quicker. On the other hand, sometimes you want to give the job to someone else, to increase the bus number of the project. While this sort of think makes the project harder to manage, it is the sort of thing that a good project manager knows how to deal with - once they get to know the people involved.

Another consideration is that developers are highly distractible. That is, while they are working on one thing, they may suddenly notice that something is a bit messy, and decide to rewrite it or refactor it, even if that was not part of the original plan. When considering the long-term health of the moodle, that might be the right thing to do, even if, in the short term it delays the next release. At other times, it might just be self-indulgance and cause bugs. Would a project manager be able to tell the difference? Anyway, as some-time tiddlywinks world champion Andy Purvis once told my, after thrashing me soundly, "Strategy should be a mixture of planning and opportunism."

A final potential problem is that the Moodle core developers are a highly motivated group of people. We care a lot about Moodle, and have demonstrated that over the years. We don't need to be told what to do, and a project manager who tried could piss a lot of people off very rapidly and do a lot of harm to the project. Fortunately, I think the chance of Martin appointing someone like that is tiny. However, I think is an important point to make, that the Moodle Project Manager should be a colleague and peer of the developers (with a particular set of skills). That is, they should manage the project, not the developers.

So, I have said that I think a Moodle Project Manager is a good idea, but I have also said why I think it would be a very difficult job to do. So what do I think the Moodle Project Manager can usefully contribute (that I, as a developer, would not object to ;-)).

A good project manager is always a good communicator, and in multiple directions. The Moodle Project Manager will need to communicate with the developers to track as clearly as possible what development is going on, and what still needs to be completed before the next release. They then need to communicate that picture of where we are to all the Moodle administrators waiting for the next release, including the Moodle Parters. They also need to communicate it back to the developers to help them see what progress is being made and what the priorities. Naturally they will be working closely with Martin. They also need to communicate progress to Translators and third-party plugin authors, to help translators have their translations ready as soon to the main release as possible.

A project manager is also good at extracting clarity from a confusing situation, which is really another part of what I just wrote. They can take reports from a lot of people, and a more-or-less vague list of outstanding tasks, and combine that into a picture of where we are now, and where we are going that other people can understand.

Once you can communicate well, and as a result derive a clear picture of where we are and where we are going, then the actually project planning - getting the right people to work on the right tasks next - is relatively easy, but still requires some knowledge of the people and the tasks. Then there are the people skills to help keep people motivated and focussed on the next release without trying to over-manage them.

Finally, and particular around the release time, a project manager can improve the processes. For example, does Moodle need string and user-interface freezes in the run up to the release, to help translators and theme designers? (The answer needs to be a consensus of all involved, so more communication.)

The project manager will also be one of the people in a position to hear what new features people are most desperate to have next, and so will be able to contribute to planning the scope of future releases.

So, it is a challenging job, but I think someone is needed to do it.