Business Ethics Journal Article

30 C O M M U N I C AT I O N S O F T H E A C M | O C T O B E R 2 0 2 0 | V O L . 6 3 | N O . 1 0

V viewpoints

I M

A G

E B

Y A

L E

X A

N D

E R

S U

P E

R T

R A

M P

declaration and management of con- flicts, as follows.

1. Uniquely identify all authors in bibliographic data sources, as well as all authors, reviewers, and meta- reviewers in manuscript management systems. (Meta-reviewers are those who manage the review process, such as editors-in-chief and program com- mittee chairs.)

Duplicate names already make it impossible to unambiguously identify by name those involved in the review process, and even make it difficult for conference organizers to ensure they are inviting the right people to join

O VER THE LAST 70 years of com- puter science research, our handling of conflicts of interest has changed very little. Each paper’s corre-

sponding author must still manually declare all their co-authors’ conflicts of interest, even though they probably know little about their most senior co- authors’ recent activities. As top-tier conference program committees in- crease past 500 members, many with common, easily confusable names, PC chairs with thousands of reviews to assign cannot possibly double-check corresponding authors’ manual decla- rations against their paper’s assigned reviewers. Nor can reviewers reliably catch unreported conflicts. Audits at recent top-tier venues across several areas of computer science each uncov- ered more than 100 instances where, at the first venue, a pair of recent co- authors failed to declare their conflict of interest; at the second venue, some- one was assigned to review a recent co-author’s submission; and at the third venue, someone reviewed a sub- mission written by a prior co-author from any year. Even the concept of a conflict deserves closer scrutiny: an au- dit at yet another recent top-tier venue edition found more than 100 cases in which prior co-authors from any year reviewed each other’s submissions.

These are issues of scale. Seven-

ty years of exponential growth have turned our little village into a metropo- lis, and our handling of conflicts of in- terest (conflicts for short) has not kept pace with our community’s growth. But we computer scientists are experts at scaling up! We have already addressed issues of scale in many other aspects of our review processes, including en- hancements such as double-blind re- view, multiple submission deadlines, opportunities for revision and rebut- tal, and online submission and review management systems.

It is time for our venues to leverage existing data sources to improve the

Viewpoint We Need to Automate the Declaration of Conflicts of Interest Leveraging existing data sources to improve the declaration and management of authorship conflicts of interest.

DOI:10.1145/3414556 Richard T. Snodgrass and Marianne Winslett

O C T O B E R 2 0 2 0 | V O L . 6 3 | N O . 1 0 | C O M M U N I C AT I O N S O F T H E A C M 31

viewpoints

V uniquely identified, and the otherg re-quires only reviewers to be uniquely identified.3 In the longer run, we recom-mend that outcalls to a conflict service be directly supported by manuscript management systems, so that the sys- tem can automatically invoke the con- flict service to augment self-reports of conflicts before reviewers are assigned. We also recommend that the authors of reviewer assignment algorithms extend them to avoid additional more subtle bi- ases in the review process, by ensuring diversity of institutions, localities, and countries of origin. Computer science research is now a global enterprise, and we should take advantage of that diver- sity throughout the review process.

Villagers might not need to lock their doors, but metropolis dwellers would be foolish not to. As village life slowly gave way to the anonymity of the big city, our community has had to es- tablish ethics committees and codes of ethics, policies on plagiarism, au- thorship, sexual harassment, and so on. Automated reporting of observable conflicts will greatly reduce the big-city crimes of impersonating others and deliberately underreporting conflicts. Automated audits will offer a further deterrent once the conflict service is in- tegrated into submission systems: the system can automatically recompute the observable conflicts some months after the submission deadline and compare them to those stored in the system. At a minimum, missing self-re- ports should result in a warning letter.

Currently, conflicts are all-or-nothing: today two recent co-authors absolutely cannot review each other’s papers, but maybe tomorrow they absolutely can. Big-city life demands a more nuanced definition that recognizes all the shades of gray, so let us acknowledge that con- flicts differ in their severity, drop the bi- nary definition of conflict, and define a (degree of) conflict as a real number in [0, 1] computed by a formula specified by the publication venue (the aforemen- tioned menu-style specification). Then we can differentiate between the sever- ity of a conflict and a venue’s publicized threshold for automatically disqualify- ing a reviewer, which will legitimately differ between venues (for example, a

g See https://www.ntu.edu.sg/home/assourav/re- search/DARE/closet.html

their program committees. Fortunate- ly, authenticated ORCIDsa exist for ex- actly this purpose, and we should re- quire their use.

2. Disallow changes in the author list after submission. Conflict declara- tions are based on the author list at the time of submission; subsequent chang- es may introduce new conflicts not con- sidered during reviewer assignment.

3. Require automated reporting of all observable conflicts. PC chairs can use a service that identifies all conflicts observable in publicly available in- formation on co-authorships, institu- tional affiliations, and advisor relation- ships, as explained here.

4. Require authors to self-report only non-observable conflicts, such as new employers, new collaborations, family members, and friends.

5. Automatically audit self-reports in retrospect and share the results with the venue’s sponsor or publisher, which should have the power to exam- ine all data they consider relevant and to impose appropriate sanctions for se- rious violations.

6. Use an independent and conflict- of-interest-free committee to select best papers.

7. Consider the use of a more so- phisticated definition of conflict of in- terest, as explained here.

8. Involve the community and our professional societies as needed, as discussed here.

To see how an automated conflict re- porting service for manuscript manage- ment systems can work, consider the traditional definition of a conflict: two people have a conflict if they wrote a pa- per together in the past two years, are at the same institution, are close relatives or friends, were advisor and advisee, or worked together closely on a proj- ect in the past two years. Bibliographic databases such as Google Scholarb and DBLPc implicitly provide a graph of the relevant co-authorship relationships, and can also be mined with high accu- racy to identify advisor-advisee relation- ships.2 DBLP already uses data-driven

a The Open Researcher and Contributor ID (OR- CID) is an international non-profit initiative to uniquely identify scientific and other academic authors; see https://orchid.org

b See https://google.scholar.com c See https://dblp.org

disambiguationd of individuals and associates authors with ORCIDs and employers; see, for example, how DBLP handles its 218 different Wei Wangs.e Authenticated employer information (including unique IDs for institutions) and educational affiliations are also available directly from the ORCID ser- vice, and perhaps authenticated advisor information eventually as well.

The conflict service’s input is: for each paper, the set of (uniquely identi- fied) authors; the set of reviewers and meta-reviewers, also uniquely identi- fied; and a menu-style specification of the venue’s conflict policy. For each paper, the conflict service returns the paper’s conflicts, that is, all review- ers and meta-reviewers who have an observable conflict with an author of the paper, along with an explanation of the source of the conflict. These conflicts must be added to the self- reports in the submission system, af- ter which conference organizers can use any method of assigning papers to reviewers, for example, manually, based on bids, or using the Toronto Paper Matching Service.1 As usual, the assignment algorithm will automati- cally avoid all review assignments that involve a conflict. Note that the con- flict service need not learn anything about a venue’s submissions, beyond the set of all authors.

Two standalone beta versions of conflict services are already available to PC chairs, driven by DBLP data; onef requires authors and reviewers to be

d See https://dblp.uni-trier.de/faq/17334571.html e See https://dblp.uni-trier.de/pers/hd/w/wang:wei f See https://github.com/ebina1/conflict-of-

interest

It is time for our venues to leverage existing data sources to improve the declaration and management of conflicts.

32 C O M M U N I C AT I O N S O F T H E A C M | O C T O B E R 2 0 2 0 | V O L . 6 3 | N O . 1 0

viewpoints

even a computing-wide consortium that includes non-profit societies and for- profit publishers.

2. To expand the definition of a con- flict and devise the infrastructure to support that definition, we may need input from experts on the social issues of privacy and security; the technical issues of data collection, organization, and maintenance; the policy issues in- herent in defining conflict broadly yet specifically; and the administrative is- sues in long-term maintenance and evolution of a conflict service.

3. We should encourage research into relevant topics, including definitions of conflict, scalable algorithms to identify conflicts, and sources and methods for handling suspected false positives.

4. Once they are in place, we should share our community’s metrics, mecha- nisms, and infrastructure with the glob- al research enterprise, including other scientific disciplines and the National Academies of interested countries.

Life in the big city poses new threats and challenges, but we can leverage the metropolis’s great infrastructure to ad- dress those problems. By taking advan- tage of existing datasets, services, and mining algorithms, we can eliminate almost all the tedium of declaring and managing conflicts, with the pleasant side effect of reducing the metropoli- tan crime rate. With those measures in place, we can move on to develop a more nuanced understanding of what constitutes a conflict of interest.

References 1. Charlin, L. and Zemel, R.S. The Toronto paper matching

system: An automated paper-reviewer assignment system. In Proceedings of the International Conference on Machine Learning (ICML) 2013.

2. Wang, C. et al. Mining advisor-advisee relationships from research publication networks. In Proceedings of the 16th ACM Conference on Knowledge Discovery and Data Mining (KDD), 2010.

3. Wu, S. PISTIS: A conflict of interest declaration and detection system for peer review management. In Proceedings of the 2018 ACM SIGMOD/PODS Conference, 2018.

Richard T. Snodgrass ([email protected]) is a Professor and Galileo Scholar at the University of Arizona, Tucson, AZ, USA. He is an ACM Fellow, has served as editor-in-chief of ACM TODS and as chair of ACM SIGMOD and the ACM Publications Board, and was founding co- chair of the ACM History Committee.

Marianne Winslett ([email protected]) is a research professor emerita at the University of Illinois, Urbana IL, USA. She is an ACM Fellow and has served as a coeditor- in-chief of ACM TWEB, as an officer of SIGMOD and SIGART, on the steering committees of ACM CIKM and ACM CCS, and on the editorial boards of ACM TODS, ACM TISSEC, ACM TWEB, IEEE TKDE, and the VLDB Journal.

Copyright held by authors.

workshop versus a top-tier conference). The conflict service described here can easily support such venue-specific cut- off scores and real-valued functions for computing conflicts, making it easy for venues to define and experiment with more sophisticated measures.

We also need to recognize that mul- tiple co-authorships indicate a stronger tie. A dozen papers co-authored five years ago may pose as much of a conflict as does a single paper co-authored last year, because those dozen papers indi- cate a very strong tie. Further, conflicts can have multiple contributing facets, for example, same institution, same city, or a highly overlapping set of prior co-authors. We can weight each type of tie between researchers according to the strength of their tie, model the fading of ties over time as a continuous func- tion, and devise a method to gracefully combine multiple weighted and faded factors into an overall conflict score, corresponding to our best estimate of the chance that two people cannot im- partially review each other’s work.

The prototypes mentioned here show that one can already build useful standalone conflict services that rely on readily available data. But we will need greater community involvement to reach the ultimate solution. Beyond the steps outlined that each venue can take today, we advocate four steps at the community level.

1. To reach a solution suitable for all of computer science, our community will need to provide coordination and funding for infrastructure construc- tion. This could come from the ACM Publications Board, the SIG Governing Board, the IEEE Technical Activities Board, ACM and/or IEEE as a whole, or

The prototypes mentioned here show that one can already build useful standalone conflict services that rely on readily available data.

For further information and to submit your

manuscript, visit telo.acm.org

ACM Transactions on Evolutionary Learning and Optimization (TELO) publishes high-quality, original papers in all areas of evolutionary computation and related areas such as population- based methods, Bayesian optimization, or swarm intelligence. We welcome papers that make solid contributions to theory, method and applications. Relevant domains include continuous, combinatorial or multi- objective optimization.

ACM Transactions on Evolutionary Learning

and Optimization (TELO)

Copyright of Communications of the ACM is the property of Association for Computing Machinery and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.