Double-blind review at SSBSE

As PC co-chair for the Symposium for Search-Based Software Engineering in 2014, I participated in the Organizing Committee’s discussion about and decision to implement double-blind submission review for the main research track. On behalf of the Organizing Committee, I composed a post explaining the reasoning behind the experiment. That post was moved around as it became less relevant to the conference (…after reviews were concluded). Since more than two people have since asked for a link to it, it seemed reasonable to reproduce it here.

SSBSE with Double Blind

Why is SSBSE 2014 Implementing Double-blind Submission and Review?

We thought the members of the SSBSE community might be curious about our reasons for implementing double-blind peer review this year. We prepared this web page to provide a brief explanation of our motivations for the change.

Our goal as a PC is to provide as objective and fair an evaluation of the submitted work as possible and, in so doing, to accept high-quality papers that describe high-quality science in search-based software engineering. There is a large and growing body of evidence that subconscious biases influence one’s ability to objectively evaluate quality independent of other identifiable features of a candidate or author, e.g., national origin, gender, or race. A sample of such findings (reported results are statistically significant) include:

• As compared to non-US reviewers, US reviewers in a medical discipline were more likely to recommend acceptance of papers submitted from US-based institutions than those submitted from non-US-based institutions. Removing the institutional information from the submission mitigates the effect. [1]
• Regardless of their gender, faculty at research-focused US universities were less likely to hire a female candidate as a lab manager than a male candidate, and offered the male candidate a ~10% higher starting salary on average. Additional analysis indicated that the faculty viewed the female candidate as less competent than the male candidate. However, the job applications were exactly identical save for the gender associated with the candidate’s name (the authors selected names that are perceived as “equally strong” culturally to mitigate the effects of societal biases related to naming).[2]
• A 2008 study evaluating the effect of blinding on the representation of women in author lists in ecology and evolution journals summarized its results in its abstract as follows: “In 2001, double-blind review was introduced by the journal Behavioral Ecology. Following this policy change, there was a significant increase in female first-authored papers, a pattern not observed in a very similar journal that provides reviewers with author information. No negative effects could be identified, suggesting that double-blind review should be considered by other journals.” [3]
• Studies have found no measurable impact on the quality of the reviews from blinding as judged by both reviewers and reviewees (e.g.,[5]).

We take from these studies and others like them the lesson that humans struggle to make objective judgements in the face of information regarding the race, nationality, of gender of a candidate or submitting author. These biases are pernicious by virtue of being subconscious. We believe that the vast majority of academics are absolutely well-intentioned and that there exist very, very few explicitly racist or sexist reviewers. These biases appear to affect everyone, regardless of the race or gender of the evaluator. However, the evidence suggests that these effects can be mitigated in peer review through the use of double-blind review. We believe this is a constructive step towards providing a review system that objectively evaluates submitted papers based strictly on the quality of the work they describe.

A common counter-argument to double-blind review is that it is possible to infer the identities of authors based on content, especially in a small community, and therefore blinding submissions is merely inconvenient to authors with no obvious benefit. However, studies suggest that blind submissions are “blinder” than we tend to believe. For example, one set of reviewers in the field of public health [4] claimed to be able to identify authors 47% of the time, but were wrong 16% of the time. On the one hand, in ~31% of cases, “blinding” doesn’t work in this field. On the other hand, more than two-thirds of the time, the reviewers couldn’t identify the authors based on the submission; more than half the time, they didn’t even think they could. Moreover, we as a PC hope that we will, as a community, attract papers from new authors and institutions. These submissions will be difficult for reviewers to un-blind.

The blind review process is imperfect. There will always exist papers whose authors are “guessable,” as well as reviewers who strongly desire author information for the purposes of their review and thus expend effort to derive it from context. As a result, we believe double-blind review is more fair to the reviewers than the alternative: Without author info, reviewers who wish to avoid it do so by default, and reviewers who wish to use it still can, by digging into the paper and its references. By contrast, if identifying information is included with papers, reviewers who do not want the information do not have the option to “unread” it.

We regret any inconvenience that authors may experience as a result of this policy and will provide guidelines and latex stylefiles to help support paper submission. We ask our submitting authors to enact a good-faith effort; we will not desk-reject papers that “un-blind” themselves in the prose. We are also happy to answer any questions and hear your feedback. Peer review is a collaborative system and a work-in-progress, and we are interested in the community’s experience to help inform future decisions.

References:

Threats to validity w.r.t references: the members of the organizing committee do not possess research expertise in this area. We attempted to provide well-qualified sources where relevant. Additionally, many of the studies on double-blind review and related issues we reference below come from non-CS scientific disciplines. However, there does not appear to be strong reason to believe that CS or SBSE differ from those disciplines in ways that would result in significantly different findings.

There exists considerably more research on this subject; we selected these references in the interest of brevity. Feel free to contact us if you’d like to know more.

1. Gastroenterology, Bethesda, MD. “US and non-US submissions: an analysis of reviewer bias.” JAMA, Jul 15;280(3):246-7, 1998.
2. Moss-Racusin, C.A, Dovidio, J.F., Brescoll, V.L, Graham, M. J. and Handelsman, J. “Science faculty’s subtle gender biases favor male students.” Proceedings of the National Academy of Sciences, 109, 2012, 16474–16479
3. Budden, T., Aarssen, K, and Leimu, L. “Double-blind review favours increased representation of female authors.” 2008.
4. Yankauer, A. How blind is blind review. Am.J. Public Health 81, 1991, 843-845.
5. van Rooyen, S. et al. (1998) Effect of blinding and unmasking on the quality of peer review-a randomized trial. JAMA 280, 234-237.