New checklist for Responsible NLP Research

As announced in the NAACL 2022 call for papers, we have been working hard, in conjunction with several people (Jesse Dodge, Anna Rogers, Margot Mieskes, Amanda Stent, and the ACL Ethics Committee) to incorporate a checklist into the submission process, designed to encourage best research practices in our field, from an ethics and reproducibility perspective. The checklist has now been incorporated into the ACL Rolling Review submission template.

ARR Responsible NLP Research checklist

The ARR Responsible NLP Research checklist is largely based on the NeurIPS 2021 paper checklist, the reproducible data checklist from Rogers, Baldwin, Leins’s paper “Just What do You Think You’re Doing, Dave? A Checklist for Responsible Data Use in NLP”, and the NLP Reproducibility checklist from Dodge et al., 2019, “Show Your Work: Improved Reporting of Experimental Results”.

Authors are required to follow the ACL code of ethics and to fill the checklist to ensure that best practices are put in place (filling the checklist is required for the paper to be eligible to appear at NAACL).

We wrote extensive guidelines to help you fill the checklist, please see:
https://aclrollingreview.org/responsibleNLPresearch/

This section (and subsequent sections) of the NAACL 2021 ethics webpage give specific guidance about what authors need to pay attention to, depending on the type of paper written. We encourage you to take these guidelines into consideration when writing your paper.

Why?

Feedback from previous conference program chairs and ethics chairs suggested a need to improve the consistency and clarity of issues flagged for ethics review, and to do so earlier in the reviewing process (ethics should not come as an afterthought with a review process separate from the technical review). We see this checklist as a first step in educating both authors and reviewers about responsible practices and in making the ethics review process more uniform.

While at first glance, the checklist might appear long, the detailed instructions should make it reasonably quick to fill. You are not expected to write long justifications but simply to provide pointers to sections in the paper where specific types of information are discussed to help with the review process.

We thank you for your cooperation in creating better practices in our field!

Marie, Marine & Ivan
NAACL 2022 program chairs