Claire Le Goues

The ICSE submission cap, 2016 Data Edition.


[ETA: the Townhall will start at 5:45 in the Glass Oaks room, and will include time for discussion and questions, as well as a panel of experts and senior members of the community who will address the issue.]

I’ve been corresponding with Tom Zimmermann of Microsoft Research, who has analyzed ICSE 2016 submission data along the lines I explored in my previous post. Tom received permission to share his analysis, and subsequently gave me the same, as an update to augment some of my previous observations with more recent numbers (one year’s worth of data is not exactly satisfying).  As Tom has more information from 2016 than I did for 2015, he could make somewhat more nuanced observations about the potential impact of the cap (though naturally we don’t know what would actually happen; incentives are weird).

For completeness, note that the official number of submissions to ICSE 2016 is 530; this/Tom’s analysis is based on the data for approximately 500 of them, omitting at least the desk rejects.  That said, in 2016:

There are several high-level possibilities regarding how many papers would be blocked by the three-submission cap[1]:

Tom also divided submissions into two groups:

Interestingly, it looks as though the submitters with more papers submitted (submitting papers in group B) are more successful in getting papers accepted (ICSE 2016 had a 19% acceptance rate overall, for context):

broup accepted submitted rate
A (not affected by policy) 64 358 18%
B (affected by policy) 37 142 26%

IMPORTANT CAVEAT: CORRELATION DOES NOT EQUAL CAUSATION. [ETA and I’m not making any claims about statistical significance, not that one conference’s worth of anecdata should be enough to set any conclusions in stone anyway] That is: I’m not saying that you should submit more papers to increase your acceptance rate. Rather, it seems plausible that high-volume submitters are established, active members of the community with many collaborators, and are thus perhaps more likely to write high-quality papers. Indeed, the overall picture suggests a positive correlation between number of papers submitted and acceptance rate: Authors submitting 4-5 had the highest success rate in 2016.

One final point of interest, at least to me, is that the policy seems to affect submitters from some countries more than others [2]. In particular, submitters from China, Canada, Singapore, and Hong Kong were more likely in 2016 to participate in papers with high-volume submitters.

I’m not drawing firm conclusions, here; I’m just adding more recent data to the previously discussed numbers.  I’m frankly too tired to do anything beyond that at the moment (I blame my very minor jet lag!).  I thought it relevant to post now, however, given the panel scheduled during ICSE dedicated to this issue.  I believe (though am uncertain) that this panel is scheduled during the Town Hall session, Wednesday evening.  I will update this post again if I hear otherwise.

[1] Note that this doesn’t reduce the reviewing load overall; these papers will still be submitted/reviewed by the community, just not by the ICSE PC.^

[2] If I get the time in the next day (keynote tomorrow morning!), I’ll update this post with a table to that effect; I have the data, I just need to make it legible in blog form!^