There must be more than ICSE for academic software engineering research

The International Conference on Software Engineering (ICSE) is the best known conference for academic software engineering research. It gets more and more submissions and the PC chairs are trying different things to deal with it. For this year, Willem Visser and Laurie Williams tried a Program Board approach which worked reasonably well. But it does not fully solve the problem.

Recently, the PC chairs of 2017, Alessandro Orso and Martin Robillard announced the idea to limit the submission to three per author. And they let the social media chairs start a discussion on that. Which I found to be a brilliant idea.

My initial reaction was positiv:

But several other responses were critical to negative. In particular, Andreas Zeller and Christian Kästner took the time to write up their opinion in a blog post each. This led me to the conclusion, maybe I should write up my thinking in something slightly longer than a tweet as well.

First of all, I have to say I can understand the argumentations of both Andreas and Christian, but they still have not changed my mind. Their main argument is that the three paper limit hinders collaboration. As Christian puts it:

I think this specific policy is a terrible move and will hurt one of the facets of software engineering research that I enjoy most: collaboration.

and

Let’s be specific: My submission numbers fluctuate, but in previous years I would have run into this policy multiple times. For example, for ICSE 2014, I submitted 6 papers with disjoint groups of collaborators (21 coauthors total); for ICSE 2016, I submitted 5 papers with different first authors and mostly disjoint collaborators. Only 2 of those 11 papers were resubmissions (significantly revised). Each of these papers was prepared by a collaborator who has worked hard on the submission for many month. There would have been no way for me to scoop in at the last minute and ask my collaborators to not submit.

And I get that. If you work with different people on different projects and publications, you want to send the results to where you think it is most suitable and not where you have not exceeded your paper limit.

Andreas, the thoughtful communicator he is, used the mathematician Paul Erdős, who was famous for his collaborations, as an example in his blog post:

Ever heard of Paul Erdős? The 20th century Hungarian mathematician is not only known for his numerous contributions to Mathematics, but also for his multiple collaborations, engaging more than 500 collaborators. Frequently, he would just show up on their doorstep, work with them for some hours, and then get a joint paper out of that. A low Erdős number indicates academic closeness to Erdős, and is something one can brag about at academic venues. Yet, if today, Paul Erdős knocked on your door, and asked whether you would like to work with him, you should avoid any collaboration with him – if you work in Software Engineering, that is. Why is that?

Initially, this example got me. I thought, yes, we should not prevent software engineering equivalents of Erdős to work. Yet, the example does not really fit. I started to have doubts and looked more closely into his publication records (which are huge, but available). Let us look at his most productive year (1978). This is how often he published in a particular venue:

  • Period. Math. Hungar.: 2x
  • Math. Slovaca: 1x
  • Austral. Math. Soc. Gaz.: 2x
  • Comment. Math. Hell.: 1x
  • Proc. Internat. Conf. Combinatorial Theory: 1x
  • J. Combin. Theory Ser. A: 2x
  • Proc. Amer. Math. Soc.: 1x
  • Real Anal. Exchange: 1x
  • Adv. in Math.: 1x
  • Neder. Akad. Wetens. Indag. Math.: 1x
  • Colloq. Internat. CNRS: 2x
  • Proceedings of the Ninth Southeastern Conference on Combinatorics, Graph Theory, and Computing: 2x
  • Creation Math.: 1x
  • Proceedings of the Seventh Manitoba Conference on Numerical Mathematics and Computing: 1x
  • Discrete Math.: 1x
  • Indian J. Math.: 1x
  • J. Austral. Ath. Soc. Ser. A: 2x
  • Aequationes Math.: 5x (3 of which are short communications)
  • Acta Math. Acad. Sci. Hangar.: 1x
  • General Topolgy Appl.: 1x
  • Acta Sci. Math.: 1x
  • J. Number Theory: 3x
  • J. Graph Theory: 3x
  • Anal. Math: 1x
  • Discrete Math.: 2x
  • Proc. London Math. Soc.: 1x
  • Proc. Fifth Hungarian Colloq.: 1x
  • Ann. Discrete Math: 1x
  • Advances in Math.: 1x
  • Math. Mag: 1x
  • Pacific J. Math.: 1x

So only Aequationes Math. had more than three papers. Furthermore, I would argue that the three short communications could be counted as a different type of paper. So, in essence, it seems that Paul Erdős, even in his most productive year, would not have had a problem with the three paper per author limit. Hence, the example is a bad fit.

So why is that so? And why are many scientists so concerned about such a limit? I believe, it is because we are too focused on specific venues that are supposed to show that you have done work of great quality.

As Andreas puts it:

If you want to publish and present your greatest work, this is where you submit it.

But is (and should) this really be the only place?

Hence, I asked Sven Apel after this tweet:

And I cannot follow this argumentation. Decisions in hiring committees are not god-given. We are the people who make them and can change them.

Mary Shaw already comments very much in that sense in Christians’ blog:

It's much more important in the longer run to deal with the conflict between good science and publication pressure, for example by advocating for the CRA guidance on evaluating scholarship

Let us put more importance on actually reading all these papers than on writing lots of them and sending them all to ICSE. Maybe even, consider using conference for what they are actually supposed to do: discussing current work in progress, and sending our great work to journals where is enough time for reviewing and space for writing.