Hi everyone, if you don't know me I am Rudy Grigar (basic` on Freenode IRC). I have been a volunteer Drupal.org infrastructure maintainer since 2009. This month I started full time at the Drupal Association managing the Drupal.org infrastructure, which I am extremely excited about -- volunteer work has become my day job!

One of my first projects is to improve the development process for people interested in contributing to Drupal.org.

Why?

Back in the Drupal.org redesign days, we began using Jenkins CI to manage our development and staging workflow. Jenkins was configured to handle database sanitization, cron jobs, dev-staging-production deployments, and various other things like clearing caches and provisioning development environments for the Drupal.org redesign.

After the redesign the Jenkins jobs were made more generic to enable developers interested in working on Drupal.org to do so. These jobs facilitated the redesign, Git migration, and the D7 upgrade. Existing development environments provisioned by these jobs have significant limitations. First and foremost, they don't have the proper Git and packaging system we use on Drupal.org, and can't be used for development of those areas of the site. The number of development environments we can provision is very limited, due to memory and disk limitations on the dev servers and big sizes of these sites. Additionally, building one such site takes approximately 2 hours. All of these reasons slow down Drupal.org development. My goal is to make working on Drupal.org issues easier and faster for people interested in contributing.

What's Happening Now?

The Drupal Association has purchased a new server for an OpenStack deployment. The server has been installed and initially configured with OpenStack Havana thanks to help from Jason Ford and the team at Blackmesh. The next steps to finish configuration are setting up a new vlan for the cluster, adding instance images, locking down firewalls and setting up web access.

We will then begin to migrate the existing virtual machines used for development and staging to the Drupal.org OpenStack cluster from the shared Open Source Lab Ganeti cluster. The new hardware is faster and has more resources (CPU, Memory, Disk, etc.) which will directly improve the performance of the existing Jenkins dev/staging deployment workflow.

What to Expect in the Future?

Looking forward I see tight integration with Jenkins, Puppet, Vagrant and OpenStack to let contributors easily boot an instance of the latest sanitized snapshot of Drupal.org (or one of the many sub-sites). This requires better integration with Puppet, and will require refactoring and adding Jenkins jobs and possibly using Vagrant to continuously update snapshots with the latest sanitized copies of Drupal.org. Using Vagrant can make the process more portable in the sense that we can reuse the Puppet and Vagrant integration in other places. This integration could be the foundation for creating "Drupal.org in a box", letting people download a local development environment where they can work on development from their local machines. Meanwhile, the increased performance and capacity of this cluster should let us deploy full copies of Drupal.org with Git and packaging for developers to work on.

The future is promising and I'm excited to see where this project will take us.

Comments

cosmicdreams’s picture

Excellent news!  Especially that part about creating a Vagrant file for common Drupal.org development.  I know a lot of developers have done just that for project work.  Maybe we can provide a baseline vagrantfile for reuse

sheldonkreger’s picture

I'm glad to see progress in the devops of D.O. I know that reducing the onboarding time of developers will allow more people to spend time working on issues.