A reluctant ICSE submission cap post or, an exploration of primary sources

[ETA: the Townhall will start at 5:45 in the Glass Oaks room, and will include time for discussion and questions, as well as a panel of experts and senior members of the community who will address the issue.]

I’ve been corresponding with Tom Zimmermann of Microsoft Research, who has analyzed ICSE 2016 submission data along the lines I explored in my previous post. Tom received permission to share his analysis, and subsequently gave me the same, as an update to augment some of my previous observations with more recent numbers (one year’s worth of data is not exactly satisfying).  As Tom has more information from 2016 than I did for 2015, he could make somewhat more nuanced observations about the potential impact of the cap (though naturally we don’t know what would actually happen; incentives are weird).

For completeness, note that the official number of submissions to ICSE 2016 is 530; this/Tom’s analysis is based on the data for approximately 500 of them, omitting at least the desk rejects.  That said, in 2016:

• 42 of ~1482 authors (approximately 2.8%) submitted 4+ papers. Those 42 people co-authored 142 submissions in total.
• The 42 "high volume" submitters  co-authored with 364 other people, or 25% of the submitting authors. This totals 406 submitting authors (~27%) potentially affected in some way by the 3-paper policy.

There are several high-level possibilities regarding how many papers would be blocked by the three-submission cap[1]:

• In the most conservative scenario, the cap prevents 16 submissions from ICSE 2016 (142-3*42), saving 0.53 reviews for each of 90 PC members, and 1.14 for each of 28 PB members (assuming each submission gets to final round). This scenario counts each paper once, even if its author list includes multiple high-volume submitters.
• If we account for multiple frequent submitters in the author lists, the policy could save up to 75 submissions.
• The true value (assuming no other changes in behavior, which is dangerous) is likely somewhere in between (unless we all  start submitting up to the cap, which could happen).  If we assume that high volume submitters would have perfect oracular knowledge to only submit those papers that were ultimately accepted, the number of papers saved is around 50 (or a paper and a half per PC member).

Tom also divided submissions into two groups:

• Group A: Submissions where ALL authors have at most three submissions (unaffected by ICSE 3-paper policy)
• Group B: Submissions where at least ONE author has four or more submissions  (affected)

Interestingly, it looks as though the submitters with more papers submitted (submitting papers in group B) are more successful in getting papers accepted (ICSE 2016 had a 19% acceptance rate overall, for context):

IMPORTANT CAVEAT: CORRELATION DOES NOT EQUAL CAUSATION. [ETA and I’m not making any claims about statistical significance, not that one conference’s worth of anecdata should be enough to set any conclusions in stone anyway] That is: I’m not saying that you should submit more papers to increase your acceptance rate. Rather, it seems plausible that high-volume submitters are established, active members of the community with many collaborators, and are thus perhaps more likely to write high-quality papers. Indeed, the overall picture suggests a positive correlation between number of papers submitted and acceptance rate: Authors submitting 4-5 had the highest success rate in 2016.

One final point of interest, at least to me, is that the policy seems to affect submitters from some countries more than others [2]. In particular, submitters from China, Canada, Singapore, and Hong Kong were more likely in 2016 to participate in papers with high-volume submitters.

I’m not drawing firm conclusions, here; I’m just adding more recent data to the previously discussed numbers.  I’m frankly too tired to do anything beyond that at the moment (I blame my very minor jet lag!).  I thought it relevant to post now, however, given the panel scheduled during ICSE dedicated to this issue.  I believe (though am uncertain) that this panel is scheduled during the Town Hall session, Wednesday evening.  I will update this post again if I hear otherwise.

[1] Note that this doesn’t reduce the reviewing load overall; these papers will still be submitted/reviewed by the community, just not by the ICSE PC.^

[2] If I get the time in the next day (keynote tomorrow morning!), I’ll update this post with a table to that effect; I have the data, I just need to make it legible in blog form!^