In late April 2012 we kicked off a sprint to upgrade Drupal.org to Drupal 7. Over the next months a huge amount of work was done, resulting in about 300 closed issues (not counting bdd team issues). The Project module suite and many other parts of our site were erased of nearly ten years of technical debt brought on by numerous patchwork fixes over the years. In addition, many parts of the new Drupal.org were rebuilt to take advantage of the features and functionality of Drupal 7. A lot of work went into something completely new for Drupal.org - Behaviour-Driven Development (BDD) testing infrastructure and suite of tests for the most important parts of Drupal.org, including Git & Project functionality, Solr search, and various sections like Marketplace and Case studies. All told, by the end of 2012 the upgrade was close to completion. But, at this point we exhausted our initial budget. The decision was made to suspend all activities until we could reassess the situation, define a solid path forward, and get budget for the finishing part of upgrade project approved.
So what has happened?
In the weeks following that decision we took a step back and reflected on what caused us to miss our deadlines and exhaust our budget. We had the usual discoveries of decisions that could have been made better and identified areas where we could have used more help. In the end we narrowed it down to the following issues:
- Not adjusting project plan
We began the project with the understanding that this would be a partnership with other major users of the Project* suite, with the DA shouldering about 25% of the costs and development responsibility. When that partnership model fell through, we failed to properly re-define and re-scope the project. Rather, we attempted to complete all of the scoped work, without adequately adjusting the expectations for time or budget.
- Too many moving pieces
All the different teams started working on their parts of the project simultaneously, which caused some teams being blocked on others for long periods of time and volunteers getting demotivated. Looking back, better decision would have been to engage the teams at different times, finishing parts of the upgrade one by one. Additionally, we engaged with many developers for very specific and short-term deliverables, with varying rates of pay and timelines.
- Inaccurate estimates
Estimating issues is an art and a science. We failed in the art of initial estimating, and at the science of refining those estimates as more work and data became available. An estimate, by definition, is rarely accurate, but we can do better.
There were troubles in communication between many people working on different parts of the project, which resulted in people often not being on the same page and the board and community receiving unrealistic information as a result.
Unclear decision-making authority
Decisions made by one development team sometimes caused lengthy discussions with another group of developers, which caused delays in the project. As a community built website we have many different stakeholders and owners. The lines of who can make decisions on what sections of the website were not as clear as originally thought when the project began. With a very tight budget, and an already out-of-scope project, every discussion chipped away at our tenuous ability to deliver the upgrade.
Ongoing deployments on Drupal 6 site
During most of the upgrade project we were also developing and deploying new features on the live Drupal 6 site, including updates to Marketplace, new Case studies and Books sections. These deployments required DA and community time that was not then available for the upgrade.
And what are we going to do?
Having all of the above in mind we are re-starting the upgrade project this week. What will be different this time?
- Properly Defining and Scoping the Project
This time, we will recognize our finite resources at the outset, and plan to only address the most critical issues to deliver a minimally viable product. We recognize that this means we won’t produce a 100% feature complete upgrade, but it will be sufficient to meet our needs. Although the work is not yet complete, we are working with outside partners to review the issue queue to produce a smaller, more manageable list of critical issues to address.
A smaller team
To complete the upgrade, we will focus on a small core team, bringing volunteers in primarily for the QA phase only.
- Outside review
We have engaged Metal Toad, Drupal web development shop, to review the issue queue and provide a second set of opinions about the estimates. This will help us reconsider issue estimates from the outset and hopefully provide a more accurate starting point. Those estimates will be available by March 20 and incorporated into the project after that.
To some degree, the smaller project team and internal project management will help address communications issues between team members. We will also have the entire team on twice-weekly “scrums,” rather than meeting in smaller teams. I will be attending all meetings and sharing progress with the community in regular update posts. We will encourage project team members to share real and honest feedback as the project progresses and we’ll act on that feedback.
- Giving Neil Drumm decision-making authority
We would like to begin phase 2 of the project prior to the completion of governance. And, even if we waited for governance to be complete, testing the newly formed governance structure with the emotionally charged issues related to the D7 upgrade would likely kill governance before it begins. For the purpose of finishing this project, we are asking community to support Neil Drumm in making all technical decisions required to complete the project (including Project* related decisions). Neil has commit access to the Project module, significantly increasing our ability to move forward quickly.
- Minimizing updates during upgrade project
We won’t develop and deploy any new features or changes to existing features on live Drupal 6 site, except for a small number of absolutely needed ones, such as new revenue-generating pages.
Last Wednesday at the Board meeting our plan for the part 2 of the upgrade project was approved, so we are now slowly getting up-to-speed again. We should get results of the issue queue review by MetalToad on Wednesday and finalize our MVP - list of issues to work on - by the end of the week. At the same time Neil Drumm and I will start working on some of the issues. The first person to join us as a contractor will be Derek Wright, who will, of course, work on the Project* module suite. We expect couple more people to join the team in the following weeks.
Before we step into the second part of the Drupal 7 upgrade project, we’d like to say a big thanks to those people who already spent a lot of time on the project and helped us during the first phase of the upgrade. We’ve listed them all on a separate page. Thank you!