Drupal.org D7 - improving issue pages

Just like any other community website, Drupal.org has lots of places which could be changed and improved. However upgrading such a website to the next major version of Drupal is already a huge undertaking. So when we originally started the Drupal.org D7 upgrade project our goal was a straight port to Drupal 7. No major regressions, no major new features.

Nearly all sections and pages on D7 Drupal.org will look the same as they do now on Drupal 6. This includes documentation pages, project pages, case studies and marketplace nodes, issue queues, change records etc.

There is however one page which will change - the issue page. Knowing that this is an essential part of the website, where most contributions to Drupal are coordinated and where some people spend many hours each day, we want to explain why we chose to change certain things, show the new UI we are working on, let you try it out on our test site and tell us what is working for you and what isn’t.

Why is the issue page changing?

Redesigning the issue page was not part of our plan. However the Project* suite, and especially the Project Issue Tracking module, suffer from a lot of technical debt. This is some of the oldest contributed code in the entire Drupal project (node/3281 is the url for the Project module).

While porting to Drupal 7, we had to finally pay-off some of this debt:

  1. We rewrote the issue tracker from the ground up using the D7 core Field API and modern best practices.
  2. We tried to remove as much code as possible from Project Issue Tracking and either use existing field modules (core fields, Entity Reference, Field Group, etc) or move our custom code or functionality into general-purpose, reusable field modules (Machine Name, Dereference List, Extended File Field, Node Changes, etc).

At the end it’s safe to say we rewrote close to 90% of the code behind issue pages.

Since it was such a massive rewrite already, we used it as a chance to take a look at how people use the queue and to make changes to the UI. Many of these are ideas that we’ve discussed with Leisa Reichelt through the Prairie Initiative, and that many people on the Drupal.org team have been wanting to address for a long time.

In D6 there are two separate ways to update an issue:

  • to update the issue summary, you have to use the edit tab
  • to update anything else, you have to add a new comment and the comment form includes a bunch of "fields" from the node form.

Not only is this confusing (why are there two separate “modes” for updating an issue?), it means the issue summary is more rarely updated. Not everyone even understands they can do it, and since it’s a different/extra step, it tends to be ignored. Worst of all, it’s actually broken for deep technical reasons (aforementioned technical debt).

How you update an issue in D6 is simply weird. Although it seems natural to many people in the Drupal community that have been using Drupal.org for a while, having to add a comment to try to change something is unnatural and goes against the experience people have with adding comments on basically any other website. Support requests periodically appear in the Project Issue Tracking issue queue from people trying to use this module on their own sites wondering how to update issues once they post them. Newer contributors often find our issue queue strange. This is another Drupalism that many of us have internalized.

Another big problem in the D6 issue page UI is that unless you’ve been participating all along and following an issue closely, it’s often very hard to figure out what patch(es) are current, relevant, worth reviewing or testing, etc. You basically have to read the entire comment history to understand what files are still relevant, and often this means dozens of comments.

So, recreating the exact D6 UI where you have to use the comment form to change certain things about the issue and where most files are attached directly to comments would have taken a huge amount of work and special-case hacks to get working. We thought it was better to just fix the problems and build a new UI using code that will be easier to extend and maintain for the future.

Drupal 7 issue page

There are 2 major changes for the D7 issue page (and a bonus):

1. Single UI for updating an issue

Instead of having two separate modes for updating things about the issue, we're going to move everything into a single mode - updating the issue (by simply editing the node):

  • Key point: if you want to change anything about an issue from now on, you will need to update the issue by editing it. This includes changing the status, changing the summary, attaching additional files, anything.
  • When an issue is updated, the nodechanges module will automatically generate a comment on that issue. It will include a diff of everything that was changed in the new revision. This way comments will still contain the history of all changes to the node.
  • Nodechanges injects a "Reason for your change" field into the node edit form, and if that has a value it's used as the comment body of the auto-generated comment.
  • There are still raw comments, that are *just* discussion about the issue. They are posted via usual comment form without any changes to the issue itself and are just naturally interleaved with the issue-update comments.
  • In addition to the standard node ‘Edit’ link, we added a prominent ‘Update this issue’ button and information when issue was last updated below it.
  • Simultaneous editing of nodes is normally not allowed. So, we wrote and deployed the Conflict module to try to help with this. We’re also considering if it’d be possible to do real-time notification of issue updates.
  • Lastly, commits that reference the issue could trigger an automated comment. This is not in the D7 upgrade MVP, but is in our future plans.

2. The way we handle attachments


Files will always be attached to the issue nodes themselves, never to comments.

  • When you update an issue to upload additional files, the auto-generated comment will reference and display those files, but they wouldn't be directly attached to those comments.
  • There will be a table on the issue page showing the files attached to that issue, including a column linking to the comment that was generated when the file was attached, the user that uploaded the file, etc.
  • Users can choose which files to show in the table (via “Display” checkbox on files), since showing all attached files could be overwhelming on a large issue with lots of patches.
  • Hidden files, without that checkbox set will be located in a separate table, inside a collapsed “N hidden files” fieldset.
  • All changes to the visibility of a file are clearly shown in the comment history.

All of this functionality is either provided by the Extended File Field module (which we wrote to extend the core field file) and/or the Node Changes module.

The basic workflow for attachments is:

  • Upload a patch, defaults to visible.
  • If you re-roll, you update the issue again, attach the new file, and ideally remove the 'Display' checkbox from the now obsolete patch.
  • A comment is auto-generated, showing that you attached the new file, and that the old one is now hidden.
  • When anyone views the issue, they first see the table of relevant files.

Note: Testbot functionality hasn't been ported yet.

3. Bonus: Issue relations


This was not in our MVP however community volunteer (thanks amateescu!) decided to write the code for this new feature while we are busy with our upgrade tasks. So D7 issue pages will have issue relations.

You will be able to specify one parent for any issue. On the issue page we’ll display a parent for this issue and a list of child issues. Additionally there will be a possibility to specify multiple related issues. In the future we are planning to add metadata to this relations, such as ‘druplicate duplicate of’, ‘blocked on’ etc., to make relations even more meaningful.

Your thoughts?

We’ve been working hard building this functionality and think it will be an improvement in many ways. But we are not done yet. Before finishing it completely, we’d like to give you an opportunity to play with this on the test site and tell us what is working for you and what isn't. The D7 upgrade is already late, and we of course don't want to delay it any further, but this piece of the site is too important to rush. So we want to get your feedback while there's still time to make changes if needed to make this work.

Test site url: http://git7site.devdrupal.org/

Use drupal/drupal to access it and your usual username/password to login.
Note that it is being rebuild every day at 08:00-09:00 UTC, during that time it is inaccessible.

Let us know what you think either here in comments or in Drupal.org site upgrade QA queue. While this is not the full Drupal.org D7 QA yet, we encourage you to open new issues there for any problems with the issue pages you might find.

-Drupal.org team

Comments

Love it all. Great work, folks.

Then the time comes, we must keep the wording 'druplicate of'.

Totally missed that :) We should keep the wording indeed.

Good work! Did a few quickt tests at did not experiance any show stoppers, though the relation field did through some PHP strict warnings when I added a related issue.

Thanks for testing! This is work-in-progress still, so errors/warnings might happen.

This is great! now i don't need to look through all the comments to get the correct patch, it's all in centralized place, good job!
However, I never submit a patch before, but I will surely test it soon.
Thanks

Redesign and small tweaks are a good start.  But I think the issue queue needs some deeper, more fundamental changes.  It is after all the heart of our community, but it is also the source of much frustration for many. Michael Keara and Thomas Svenson are doing some research about how we can improve this.

I finally made some time to test out the new issue page, and I have some comments if it is not too late (I didn't file issues):
- Generally: Positive! The philosophy is great; may need some tweaking though...
- On an issue with a lot of comments, it would be helpful if the Files were listed in reverse chronological order. For instance look at:
http://git7site.devdrupal.org/node/731724
It takes me nearly forever to scroll down to the bottom of the list of files.
- The page suffers from the same problem as the current issue page: when there are over 300 comments, the comment links are broken (this is a Core bug)
- I am missing the "most recent comment" and "most recent attachment" and "new comment" jump links. I use those all the time on the current site and would be sad to see them go;
- There should probably not be an "Edit summary" link when you are editing the body of an issue. I think it should just be a field of type Long Text, not Long Text with Summary.
- This issue has a ton of PHP warnings at the top: http://git7site.devdrupal.org/node/731724
- The "Follow" button does not seem to be working right. I started from the Your Issues page, but many issues I looked at still had the "Follow" button active instead of telling me I was already "following" it. Some were working. I am not sure what the difference was between the ones it thought I was following and the ones it didn't. Or maybe it was the Your Issues page that wasn't working right (it should just be showing me issues I have followed, right?)
- Theming issue: when editing an issue, the drag handles for dragging file order or related issue order are nearly invisible.

This is super helpful testing and feedback, thanks!

- Re lots of files: the migration doesn't automate the task of marking files hidden, but the intention is that humans will do so on issues where it still matters. So, for the first few days, there will still be horrible-looking issues like http://git7site.devdrupal.org/node/731724 but after an update to uncheck the "Display" checkbox on most of them, the Files table should be useful again. However, it's an interesting question what the default sort order should be, and if it should be table-sort-able.

- Yeah, 300+ comments are still broken. As you say, it's core. But, we're tracking that in project_issue, too: #268374: "Jump to" links are broken on multi-page issues

- Those "Jump to" links are handy, I use them, too. ;) Just opened an issue about porting them: #2001014: Port "Jump to" links to D7

- Good suggestion about not letting people tweak a summary/teaser on issue nodes! We never display them by teaser, so that's a needless complication in the UI. #2001010: Convert body field on default issue node type to "Long text" (no summary)

- PHP warnings on certain issues are because of files that were deleted. That's being handled at #1961712: Notices and bugs when files are deleted without a new node revision.

- Follow button doesn't yet impact the "Your issues" page or any other views. See #1828756: D7 views don't honor tracker and issue following which is blocked on #949372: Port issue views to Search API so we can have a performant backend . Meanwhile, it should at least be impacting the "Your posts" page (your personal tracker page).

- Re: drag handles: Yup. ;) #2001024: drag handles for tabledrag tables are too small to be usable

Thanks again!
-Derek

Looks like you made sense of all of my notes. :)
I found one more thing: When I change tags on an issue revision, I don't think the tag changes are showing up in the revision diffs. Everything else seems to be there... seems like those should be too.
I also thought it was a bit jarring when viewing a revision diff that my user name showed up in ALLCAPS. I don't care a whole lot, since my user name is all lower-case anyway, but some people do have Intentional Caps And Lower so they might care more.

Took me a second (and some clarification in IRC) to realize you were talking about the output from diff.module, not the issue node itself. The ALLCAPS is thanks to the "Seven" theme I guess.

The lack of taxonomy reference diff'ing is weird. It works locally with the same version of diff.module and same configuration, but is broken on git7site. I just submitted this issue to track it: #2001054: diff.module 7.x-3.2 doesn't display changes to issue tags on git7site but it works locally.

Thanks again,
-Derek

Can I make a request that the red header you've got on the development site carries a link to this node? A colleague sent me the link directly to the dev site, and I just thought "there's no way I'm putting my password in on a non-d.o domain!" Then he sent me a link to here and I relented, but it would be good for that to be clearer.
Otherwise, this looks great: really streamlines everything,a nd I love the right-hand summary for "update this issue". Issue relations should also help too, despite being a bonus. Now if we could only stop people posting related issues inline. We'll have to somehow stop them posting "+1 subscribing" first, mind you....

Here's my review of the new issue page:
- First, the ordering of the files from each comment seems to be reversed. because usually the patch is added, and then the interdiff; but now, the interdiffs are higher in the list.
- The rows for a patch and its interdiff could be merged into a single row. This would make the list more compact, and their relation more visible.
- Also, I see that the project field now contains the project id too. but changing the id to a non-existent one causes some warnings and notices (it happens after focusing out of the field). also, changing either the name or id does not update the other.
Also, the "Your Issues" block in the user dashboard is very hard to read, with the 3-column layout. The issue titles look like this :)
Introduce
Assetic
compatibility
layer for
core's
internal
handling of
assets

 

It's my fault for not looking more closely, but this change in file attachments is a disaster.

 

It's destroyed one of the best features of d.o issues compared to really crappy products like jira.

Oh crap, I see what you mean. Just realized I can no longer upload files along with my comments.

Along with Peter I issue my mea culpa in advance; I missed the review process and was not expecting issue triage to change so dramatically. I maintain a project (Drupal Commerce) where I've closed out about 3,000 issues over the last couple of years, either because they were fixed, by design, duplicate, etc. Oftentimes, issues get recategorized in a way that I disagree with - such as a support request or a feature request being labeled a bug.

Instead of just closing things out (or re-closing them as is the case with countless "zombie" issues), I almost always interact with the comments in the issue as I close it out. I don't want people to think I'm an arbitrary maintainer, randomly closing things at will, and I expect at least some people to interact with me (either in disagreement or for clarification) after updating an issue.
It looks like now I'll first have to read the thread and interact with the commenters and then browse to a second page and issue another form submission to recategorize the issue or change its status. Perhaps this is really only a problem for maintainers, so perhaps at least we ought to have tools in our issue queues to continue to manage the metadata of an issue on the issue page itself?
I get that making something simpler for new users isn't the same as keeping something simpler for existing users, but in this case it seems like the decision has been to decrease the efficiency of the hundreds of power users in the d.o issue queues.

Sorry, the wysiwyg screwed up my formatting. :-/
 
I also just realized that now when I browse to an issue to see the most recent comments if I want to remind myself what the title of the issue is or review any of its metadata, I first have to go from the linked "new" comments section to the top of the page and then scroll back down. Before I could still see this information in the issue update / comment form beneath whatever comment I had just linked to.

Holy crap! Actually, I can't even see the status of the issue from the issue page without editing it. : (

Ahh, wait a minute, it tried to pull a fast one on me. It's the actual title of the table instead of being inside the table of metadata along with the other parameters of the issue.

You can change issue status and leave a comment on the same page, after clicking Update this issue. There is a comment field there.

Yeah, I figured that out, but then I lose the context of comments before that I want to interact with. What I'll probably end up doing is just opening that page in a new tab and flipping between the two.

At this moment, only the issue text itself (which is often unchanged) gets shown after I hit the "Preview" button. That doesn't work for me when typing relatively large comments. A comment preview needs to be there too, preferably at the top.

For now, I'll be doing "first comment, then change status" for anything that needs me to sit back and think about what I'm writing.

(For the rest: yeah, this requires you to open a separate tab for viewing the comments, like Ryan says. I can live with that / was kinda used to working with multiple tabs already when making a longer comment, to prevent constant scrolling between older comments <-> new comment edit box.)

Hey guys! I really like the new look and features of the issue page. One thing that I see as a good adition is to have the issue information block to be sticky so that when you scroll the issue you can always see it. Here's a very quick hack to see how it would work. Go to any issue page and open the Chrome/Firefox JavaScript console. Paste the following two lines:

jQuery('head').append('<script type="text/javascript" src="https://raw.github.com/garand/sticky/master/jquery.sticky.js"></script>');
jQuery('#block-project-issue-issue-metadata').css({width:300}).sticky({topSpacing:10});

The first one loads jquery.sticky.js on the page. The second one turns is on (and also fixes the width of the box to 300px because of an issue with the way sticky ways). Anyway.. I'd be happy to work on this feature if there is interest.

Major UX concern with this idea is that the block will be too distracting, when you try to actually read comments.

"to update anything else, you have to add a new comment and the comment form includes a bunch of "fields" from the node form
...
Not only is this confusing..."

I'm wondering if you have a link to some UX user tests which suggest this is confusing, or an issue where this was discussed more deeploy. I kind of liked the old way of doing things, and would be happy to read more about the debate around changing it.
Thanks!
Albert.