One of the biggest challenges in working in a large community like the Drupal community is removing bottlenecks. All too often the community can seem to come to a grinding halt on just one issue that can only be managed by one person. On Monday Dries gave a presentation at MIT and talked about how some of the Drupal community’s biggest problems have helped create some of our best solutions. In particular, he cited how our drupal.org server melt down in 2005 lead to the creation of the Drupal association to proactively manage and plan for our infrastructure growth.
Today the community continues to be challenged with re-designing and implementing drupal.org. On the surface, it doesn’t look much has happened, but improvements in projects, and search have made big strides in making Drupal.org easier to use. The Drupal community works well in parallel and through lots of small iterations. Unfortunately, our drupal.org development and staging infrastructure has not supported dozens of developers and themers working in parallel. We’ve now taken the time to build out that parallel development and staging infrastructure.
It is important to understand why a large site like Drupal.org can be hard to do local development for. First, drupal.org serves about 30 million pages a month, and has over 600,000 nodes. It has tens of thousands of attached files including patches, screenshots, videos, and graphics. Setting up a greater than 1GB database and syncing over 5GB’s of files is a tough task for a local development environment.
Much of the new design for Drupal.org is focused on making it easier to find what you are looking for on Drupal.org. For the redesigned development sites we also need to have a working Solr server. For example, a page like the new download and extend landing page has Drupal version search filters and at least seven blocks that are search results themselves. For themers to theme pages, their Drupal.org site needs content, graphic assets, patch files which need to be attached to issue queues, and the site must have working functionality.
Also, Drupal.org content is valuable even though it is licensed under creative commons license. If a copy of the site were to fall into the hands of spammers Drupal.org could suffer abuse. For that reason, we don't casually give away copies of the drupal.org database and code to anyone who asks.
I'll review this new infrastructure and explain how our team plans to work to support at least 10 development sites in parallel.

The Servers Web server


The dedicated virtual machine at stagingvm.drupal.org contains 10 virtualhosts, and is entirely devoted to the redesign project. The webroot of this VM is /var/www.

Database Server


The database server is separate, and lives on stagingdb.drupal.org (which is a cname pointing at civicspace.drupal.org, the solr slave).

The Sites

There are 10 staging sites for the redesign, each of them are password protected with: drupal/drupal to keep out search engines and bots.
Staging 1 (staging1.drupal.org) through Staging 10 (staging10.drupal.org) are virtualhosts on the stagingvm.drupal.org server. Each site lives in it's own directory, /var/www/staging1 through /var/www/staging10, respectively. They are almost all setup with a recent copy of drupal.org.
Each of these virtualhost containers has a separate instance of drupal.org's codebase + sanitized database, and we will give selected volunteers the access to commit theming and administrative changes to an instance for testing purposes.

Backing up a site's database, restoring a site

The redesign administrators have two scripts in their stagingdb.drupal.org home dir, backup-tables and restore tables. The administrators have sudo and can run each script to backup or restore the last backup of a db. For example, if someone is going to test something on staging5 that will likely break it. You can ./backup-tables.sh staging5. Will take about 20 min and then ./restore-tables staging5 will restore the latest backup
Database copies of Drupal.org
Santized DB dumps of Drupal.org database are available on the stagingdb.drupal.org database server. The database server can hold 10 copies max. The problem there right now is that you need to take a dump of the drupal.org database server and then reload it into another database to run an SQL script on it. This continues to be a cumbersome process that can only be done by our two database administrators Narayan Newton and David Strauss.

Granting theme repository access

http://groups.drupal.org/drupalorg-redesign-implementers/guide
* Once the drupal.org re-design developers and themers have signed up for an account on infrastructure.drupal.org we need to review their request and determine if they are a likely candidate to contribute. We get a lot of requests from people who just want to run the drupal.org theme on their own site. Right now over 35 community members have write access to the SVN, and several dozen have read access so they can generate patches. Admittedly, this approval process is a significant bottleneck.
Once the contributors account has been approved they can issue the following commands:
 svn checkout https://svn.drupal.org/drupal/themes/bluecheese/
* They can also log into SVN using their username and password from http://infrastructure.drupal.org.
The documentation tab on the front page of the Drupal.org redesign implementers group for more information about how to get access and use SVN.

Pushing theme changes to development sites

There is an SVN directory called development-themes that will be checked out to each sites /themes directory. Each themer can set the default theme for anyone of the 10 instances to see their changes go live. Since Drupal can support many themes we may be able to support dozens of themers working on a site simultaneously.

Theme Deployment

We have now setup cron jobs to checkout themes from the themers sandbox every 5 minutes.

Syncing Drupal.org production with a master staging site

Code and configuration changes happen on Drupal.org. We need to push these code and configuration changes to staging sites to keep them in sync. The staging sites are managed via an SVN branch HEAD == redesign and the live site is the SVN branch DRUPAL-6--1 == drupal.org. All configuration changes should be in drupalorg.install updates.
Deployment of code, content, and configuration changes continue to be one of the big challenges in Drupal and might be the big feature of Drupal 8. Many of Drupal’s best core innovations come from drupal.org necessities.

Pushing development configuration changes to all the staging sites

When new features are being built on a Drupal.org staging site, we need to push these same configuration changes to all ten of the staging sites, or they might not work as the developers and themers need them to.

Asset management

Since themers work with so many graphic assets we need a way for them to more easily share their working assets and get them accessible to the Drupal.org redesign staging sites.

Infrastructure administrators wanted

Now that our redesign infrastructure is built, we need Drupal administrators and developers to help with the development, staging, and deployment cycles for all 10 of these re-design sites. I know we’ve asked for infrastructure administrators before and there’s been a lot of interest. If you are still interested contact me . A big thank-you to our infrastructure team for making this possible.

Today was a big milestone for the Drupal Association and for the future of European DrupalCons. Over 70 people met in IRC for over two hours to discuss and ratify two documents which lay out the process by which we will select DrupalCon locations going forward, and how the production teams to run DrupalCon will be managed. These are the two documents:

• European Regional DrupalCon Organizers
• Scorecard for DrupalCon Europe Location

If you belong to a local group somewhere in Europe and you want to see DrupalCon happen in your city, region, or country, read both of these documents carefully. The time for nominating locations is now open, you can read the exact details of how it is done via the links provided above. Nominations must be submitted before October 1. That means there are only six full days to work on proposals. Nominations are being taken for both 2010 and 2011. The nominations will be evaluated, as per the plans in the documents, by October 15. They then get submitted to the Drupal Association Board of Directors, who have to meet and ratify the suggestions.

The important point in all of this is the level of openness and transparancy that is being brought to the system. The Drupal Association is working hard to involve the community in decisions, and to have well defined proceses for making important decisions, like where DrupalCon is held. Today we had a great discourse, thoroughly discussed all of the points that were important to people, and came to nearly unanimous agreements about how things will be done. That is what makes Drupal a nice community to live in.

read more

The Drupal Association has used some of the money that it acquired thanks to the Drupal community and its sponsors to buy more computing power for the infrastructure that all services of drupal.org are hosted on.

Due to SUN's Startup essentials programme we were able to get their hardware at discounted prices and helped ourselves to three X4170s and one X2270. The X2270 will our join our pool of webservers, one X4170 will replace the CVS and mail server, the others will become database servers with 32GB each.

The machines have now arrived at our hosting party OSUOSL and will be put into service soon. The most important task is getting CVS migrated to a faster machine.

The other machines have been ordered in order to absorb some of the growth we have been experiencing over the past year. Drupal.org serves now about 30 Mio pages per month whereas it only served 20 Mio pages per month 1 year ago. Another reason was that we expect the redesign to require more resources and we want to be prepared for that.

We would like to thank all the members of the Drupal Association; your money makes it possible for the Drupal Association to facilitate the growth of the Drupal Community. If you are not a member, please become one now!

Below are some images provided by OSUOSL.

AttachmentSize Big machine254.28 KB We eat RAM for breakfast!205.92 KB Smaller machine261.14 KB

I took a few days off from work to spend some time with my family -- I haven't had more than a 3 hour break in the last 4 weeks and was away from home during the weekends. DrupalCon Paris ended about 48 hours ago, so I'm sitting here with post-DrupalCon blues, staring at the waterfront, thinking about how fun and exciting the conference was. As usual, I wasn't able to attend more than a couple of sessions, but nothing beats talking to Drupal users and brainstorming together. And I talked to many Drupalistas ...

Looking back at DrupalCon Paris, it struck me that sometimes we can generate ideas faster when we work alone, and that sometimes creativity happens best in groups. The same is true for solving problems. The time leading up to the conference, the time at the conference, and the weeks after the conference provide a unique blend of individual and collective idea creation and problem solving that is so important for Drupal. Let's not underestimate that. I certainly value the retrospective that seems to be part of the post-DrupalCon blues (and the waterfront Mojitos that help fuel it).

With 850 Drupalistas attending DrupalCon Paris, I wasn't able to talk to all of you -- I feel sad about that because we have such an amazing community. To those people who I have been able to talk to and brainstorm with: thanks for inspiring me!

As always, DrupalCon is largely driven by volunteers, and many of them have a full-time job. It takes a lot to coordinate an event like this and it is not something we should take for granted. Everyone involved -- from the local volunteers to the Drupal Association to the event planning staff -- has my thanks! Great times!

© Jeff Eaton

© Jeff Eaton

An important part of the Drupal.org redesign is the dashboard. Power users have a towers of navigation, contributor, and/or documentation links on every page. That is clutter when you don’t need to jump to another section of the site. And one size does not fit all; there are a few options now, but either you turn on a block of links or you don’t.

The redesigned pages clean up the administrative clutter. You will be able to build your own, or keep the good defaults, dashboard to watch and get to what you care about. However, the content in the prototype is mostly positional. We need to design and build the widgets that are most useful for the Drupal community. We need to know what will help you be more productive on Drupal.org.

I started building out a dashboard module at the previous Drupal.org redesign sprints. But, it needs a lot more work. In addition to the issue queue, I made a wiki page to track priorities. I’ll be at DrupalCon Paris looking for people to help work on this important functionality.

By the way, the API.drupal.org work I wrote about, launched a few days ago. Thanks for all the testing and feedback. There are already a a few comments. There are still a lot of improvements needed, especially rewriting the parser. Find me if you want to help make more improvements.

I'm going to be honest, I'm kind of obsessed with caching. Since becoming involved in web performance, it has pretty much taken over my life. Yes there is the mysql tuning, query profiling, solr deployments....but all in all 90% of my day is looking at hit rates of one form or another. Due to this, HTTP caching for drupal.org has been a mild obsession for me. This week my wish came very close to true.

read more

Just a short time ago, I announced the refresh of Drupal.com. As I announced in my post, Drupal.com has a couple of purposes: one of its key purposes is serving as the current home of the official Drupal trademark policy. As of today, version 1.0 is available and published at http://drupal.com/trademark.

I invite you to read the Drupal Trademark Policy in detail. It's full of illustrative examples, and I hope that we've made it as community-friendly as possible. We can't cover every possible scenario, but I believe it addresses most situations that are likely to occur within our community. It may -- and certainly will -- change over time as we keep in sync with the changing needs of our community and, if necessary, to account for unforeseen situations. That is important to keep in mind.

I've owned the Drupal trademark for a long time. The lack of a Drupal trademark policy doesn't mean the trademark was unprotected -- it was protected by trademark law. The lack of a Drupal trademark policy meant that it was unclear what was allowed and what wasn't allowed, and frankly, that you were bound by trademark law. By creating a trademark policy and a licensing procedure, we've provided us options we did not have before.

The goal of our new policy is to provide guidance and clarity on how the Drupal trademark is allowed to be used. The only community model that really works is one where there is a fair-level playing ground for all people and organizations. Ultimately, that is what this policy seeks to accomplish.

The entire process of developing the policy was a community effort, with help from a variety of legal experts. We worked on the policy over the course of almost two years. A draft version of the policy was posted at http://groups.drupal.org/node/19068, and through the community feedback that developed there, we ironed out many of the wrinkles of my original draft. Larry Garfield, the Drupal Association's current legal representative has provided feedback, and both my own attorney (DLA Piper) and additional attorneys from the Software Freedom Law Center and the Drupal Association were part of the policy's development. To help validate our work, we reviewed other similar policies from sister projects to make sure that we were in-line with the current legal trends in open-source development.

As the owner of the trademark, protection of the trademark falls to me, and is managed by me with the assistance of my attorney, the Drupal Association, and potentially even local Drupal Associations. I personally bear substantial personal costs as part of sustaining the trademark in all its various geographic jurisdictions. To help offset the costs of managing the trademark, the trademark licenses, and to actively pursue those who infringe or inappropriately seek to use our brand, I will sell some advertising space on drupal.com and may also charge a small licensing fee to those that do not qualify for an automatic trademark license (section 1A) and that need to follow the license grant procedure (section 1B). Now the policy is published, I plan to work out the financial details in the next months so stay tuned for an update on that.

Most of you who use Drupal, commercially or otherwise, need not worry about how the new policy may impact you, though I certainly encourage you to study it and to apply for a license if required. For instance, in many cases, you are allowed to use the name 'Drupal' in domain names. Conversely, there are some Drupal domain names in particular that the policy seeks to protect for the good of the community and to create a fair-level playing ground. The introduction of the official policy is only intended to help ensure that the effort of hard-working Drupal contributors is not misappropriated. I think it will make us even stronger, as a community!

Every once in a while, the discussion of Drupal cerification comes up. Is a Drupal certification program a good thing or a bad thing? Are certification programs overrated? Isn't one's track record as a contributor on drupal.org the best measure of someone's past (and possibly future) performance? Should the Drupal Association manage the program, and if not, then who? What do other Open Source projects do?

Lots of unknowns and lots of subjective questions always make for a good discussion.

While I'm often skeptical about certification programs myself, I do see the value in them. Certification isn't for everyone but for some people it is valuable. A masters degree from MIT, an MBA from Harvard or a Ph.D. from Stanford is a form of certification too. But, just as there are good examples, there is no lack of examples where bad certification programs deliver certified engineers that don't know what they are actually doing. That fact, though, doesn't invalidate the valuable certification programs that do exist; for example, the CISCO certification program is very thorough and valued by the market. Similarly, I think that employers do value certificates achieved in higher education. All things considered, I believe there is value in having a well-executed high-quality Drupal certification program, especially as Drupal continues to grow.

My personal view is that the Drupal Association is not the best body to create "the" Drupal certification program. It is my belief that we are best served by allowing many organizations to create their own Drupal certification programs, and have the marketplace set their value -- similar to how universities build reputations. Competition around Drupal certification programs could be a great thing, because it would likely improve quality and allow for specialization. A certification, of course, is ultimately only as valuable as the organization standing behind it. In such a scenario, it is very important that the certification programs are labeled properly; that is, "ACME Drupal certified" rather than just "Drupal certified", where ACME is the name of the company or organization providing the service. It has to be ACME's reputation that is on the line for the quality of their participants. The Drupal trademark policy is designed to help achieve that.

I also don't believe that a certification program is a zero-sum game: that is, holding a certification doesn't imply that your track record on drupal.org (through your user profile) is of either less or more value. Again, the market and the buyer will determine the value of a drupal.org user profile versus a certification program versus a resume versus a portfolio versus degrees obtained through higher education. In any event, it is unlikely that one would hire someone solely on the basis of having a Drupal certification. I certainly wouldn't.

When we launched Acquia two years ago, we announced a Drupal certification program code named 'Yellow Jersey'. That program hasn't materialized yet, and we're not putting resources in it at this point in time. Building a high-quality certification program is a significant undertaking and we're not ready to take on such a program. And last but not least, the marketplace hasn't demanded these types of programs to a sufficient degree. Yet.

Other Open Source projects like MySQL, Zend, Linux, Ubuntu, RedHat, BSD all have certification programs so I think it is a matter of time before we see one or more Drupal certification programs emerge. Personally, I would support a well-executed high-quality Drupal certification program.

oDesk recently provided the data on the growth of Drupal platform and development jobs on oDesk. Results are exciting as Drupal is seeing a steady growth. As shown in the graph below, the number of jobs posted on oDesk have increased by 50% in just last year.


The growth seems to be fueled by the release of Drupal 6 that has made it even easier for individuals or a community of users to publish, manage and organize a wider variety of content on a website. Drupal is gaining wide spread popularity with developers originating from all across the globe. According to oDesk, India, US and Russia have seen the most growth in the adoption of Drupal.

If you are looking for more resources to help on Drupal projects, check out Drupal Group on oDesk. It is a great place to find certified and experienced Drupal talent.

Once in a while I log into google to look at their webmaster tools and what they say about drupal.org.

Yesterday, I did that again, after a hiatus of several weeks. I noticed, that google had sent me three mails which didn't make it to my inbox because I hadn't configured email forwarding.

After fixing that I looked at the mails.

One was abotu their services, they want me to use Adwords and offer some free budget. The other two were almost identical and they probably had sent the second one after I didn't react on the first:

Google thinks we have too many links!

They said that the huge amount of links may lead to googlebot not being able to index all of them and we should consider if maybe we could exclude some through robots.txt.

Helpfully, they also gave a number of example links. While some of them are perfectly valid links, others are indeed not needed. Most of these contain some sort parameter. Strangely enough, some of the sample-URLs are already among the ca 1 Mio local URLs that we block through robots.txt.

I have now added "solrsort" as a forbidden parameter and hope that it helps both google and us. Our search infrastructure has had the occasional hickup and if google searches now less, it sure is a win-win situation.