New Page Triage
Engagement Strategy
The Wikimedia Foundation is continuing to explore new, experimental ways to engage the community more effectively when developing software. This spring, the Foundation’s Features Engineering team will experiment with a number of ways to involve community members in the development of New Page Triage, a set of features aimed at improving new page patrol. The following document sets out a plan to accomplish this.
The job of the Wikimedia Foundation’s engineering team is not just to build software; it is to build software that people will use. By necessity, this involves engaging with the community during the feature development process and responding to their suggestions. During the past few years, this process has not always gone as smoothly as either side would like, leading to tensions between sections of the community and the Wikimedia Foundation. For the New Page Triage (NPT) project, we want to try an approach that will support editors in providing feedback on the tools we are developing. This will provide the engineering team with a better idea of the needs of editors, resulting in software that editors should be more comfortable using.
In summary, the plan calls for discussion of the feature to occur on the English-language Wikipedia itself, with a working prototype that editors can comment on rather than static mockups, and with regular and nuanced discussion between the Foundation and the community throughout the design process. This discussion will be facilitated by the Community Liaison, Product Development: a dedicated Foundation contractor, who acts as a bridge between the community and the Foundation.
This new form of collaboration may or may not work – failure is always a possibility – but either way, we hope to continue trying new ways to bring editors into the feature development process.
The Features Engineering's software development process draws on techniques found in many large-scale web projects, while incorporating collaborative elements that make the process unique. The team starts with a problem, researches the problem, develops solutions, and evaluates these solutions from both a user and technical standpoint. Once the best solution is identified, a draft specification is written and mockups are developed. These are usually posted on MediaWiki.org, and users are invited to comment through notices at the Village Pump and various other community noticeboards. In most cases, very little feedback is provided. After this, a prototype is built and deployed on the "test" wiki; again, the developers solicit the community for feedback, and again, little is provided (for reasons explained in this document). The prototype is improved, in line with both the design plan and the limited amount of feedback provided by the community, and turned into a fully developed feature, which is then deployed to the "live" project wikis, such as the English-language Wikipedia. More opinions are solicited from the community while the software is live.
It is at this point – with the software live – that a substantial number of editors participate in the discussion. This is often the only point that many editors learn that the discussion is happening; appeals on noticeboards and mailing lists have simply not proven effective at reaching editors. Whether the feature and/or its implementation is, in their opinion, a good one or a bad one, some editors are unhappy about the software because they feel they were not consulted about its implementation or features. These editors respond sharply to the developers, who in turn feel they are being unfairly criticised, as in their view they repeatedly tried to get editors involved in the process with little success.
In the long term, this type of disconnect undermines the Movement's cohesion. While the Foundation has made attempts to bridge the gap, we need to do a better job at making sure the development teams and editing community are working more closely and effectively together. The Foundation, as an entity within a collaborative and volunteer-based movement, has a responsibility to consider the opinions of our editors when making decisions that directly affect them. Moreover, input from the editing community results in better software, software that our editors will feel comfortable using.
Since the Usability Initiative, the Foundation has tried to approach the development process collaboratively. The results have not been perfect, but we're learning. During the Usability Initiative, we tried an “ambassadors” programme to reach out to the editing community – but without consistency. The engagement process for the Article Feedback Tool (AFT5), with discussion centralised on enwiki, was similarly both novel and an improvement, but may not be feasible for other projects. Like the ambassadors programme, and like the AFT development process, the engagement strategy for New Page Triage is experimental: while it may not work, we are hopeful that it will improve how we work and communicate with the community in the long term.
The trick with improving things is to identify where the current process fails, and why. The full explanation can be found in a dedicated document, but the problems largely fall into 3 areas:
- Communication: Our appeals for feedback are normally in congested venues where they can easily be missed by editors. The feedback process itself has some strengths (permanency, for instance) but also many drawbacks. Talk page and noticeboard discussions lag as each party must wait until the other has a chance to read and respond. Given that developers have hectic schedules, the time between responses can be substantial.
- Location: The Foundation tends to request feedback to be left at MediaWiki.org since the developer community uses this wiki as its primary coordination space. Mediawiki.org, however, is an unfamiliar venue for editors; they are not used to contributing there, nor do they naturally follow discussions, forcing editors to put substantial effort into maintaining a dialogue with the developers.
- Opaqueness: The information available to editors at MediaWiki.org is mostly in the form of text and screenshots, neither of which can communicate in detail what the software is expected to do unless you’re used to software development. This leads to confusion and ambiguity, and limits what editors can actually give feedback on. Editors’ first chance to see an actual working version is on the “prototype” wiki – a site just as obscure and unfamiliar – and they are invariably asked to leave feedback on MediaWiki.org, causing the same communications and location problems.
Whenever possible, we should be relying on processes that not only produce good software, but do so with the involvement of editors; the existing process has led to a weakening of the ties between a tranche of the community and the Foundation, and frustration on both sides. The Foundation has to look for new ways to engage the community. Below is how I hope to run the engagement process for New Page Triage.
Communication
The design process will open with the traditional sources of engagement: notices on the various noticeboards. In addition, messages will be sent directly to each editor who has expressed interest in these tools or in the new pages field generally. A prominent notice will also be placed at the top of Special:NewPages itself, advertising the tool's replacement. The purpose of this notice is to make sure that those who are actually doing New Page Patrol are aware of the project. A regular, weekly newsletter will also be established; users can sign up and receive updates and news about what is being developed so they can more accurately decide when to step in. My hope is that this will deal with the issues relating to how visible our notifications are.
As the Community Liaison, I will be dedicating a substantial portion of my time to acting as a go-between on this feature, improving the response rate to comments, critiques and suggestions. In addition, to solve the problem of talkpages being a high-latency communications method, we will be holding regular office hours sessions (as we have been doing with development of the Article Feedback Tool). This should allow for proper conversations, and hopefully speed up discussion. We may also hold audio chat sessions for more nuanced conversations with the community, assuming there is the demand.
Location
Community engagement for the New Page Triage suite will be centred on the English-language Wikipedia, not a secondary or "meta" wiki such as mediawiki.org. Enwiki was picked because the problems we are experiencing around new page triage are most prominently and commonly identified with that project. En-wiki is a far more familiar environment for the editors who use it than mediawiki.org. It also makes the watchlisting and following of discussions by editors far simpler, allowing for more coherent and constant dialogue.
Transparency
Editors will not simply be asked to comment on a description of the feature or on mockups. As soon as possible, a fully functional prototype will be released and deployed on the English-language Wikipedia, to run in parallel (but not as a replacement to) the existing Special:NewPages interface. It will still be possible to use the old version, and that will be the default until development is complete. The purpose of this parallel deployment is to give editors something to sink their teeth into; they will be able to actually use the tool and discover how it will work in practise before it is deployed as the norm. My hope is that this will permit far more detailed and nuanced feedback to be provided.
This process works very well for New Page Triage, as NPT is primarily focused on the English-language version of Wikipedia and it is technically feasible to run the prototype version alongside Special:NewPages. This will not be the case for all features, or even most features. Similarly, it is perfectly acceptable to centralise discussion on the English-language Wikipedia, because we’re talking about a feature that primarily concerns enwiki. This isn’t true for most of the things we develop. When it comes to engaging with the community on wikis other than enwiki, we encounter problems such as language barriers that make detailed engagement more of a struggle. The result is that, even if this works, we will have to come up with new strategies, just as we come up with new features.
The internal motto at the Foundation is “fail better”. This means constantly experimenting, constantly trying new ideas, in the knowledge that if they fail the failure provides us with information and lets us avoid the same mistakes in the future. That, ultimately, is what this plan of engagement is: an experiment. If bits work well, we’ll incorporate them into future software development; if bits work badly, we won’t. Either way, we hope to learn from this experience and involve our contributors as much as is feasible.