Background

Language technologies and their applications are an integral and critical part of our daily lives. The development of many of these technologies trace their roots to academic and industrial research laboratories where researchers invented a plethora of algorithms, benchmarked them against shared datasets and perfected the performance of these algorithms to provide plausible solutions to real-world applications. While a controlled laboratory setting is vital for a deeper scientific understanding of the language problem and the impact of algorithmic design choices on the performance of a technology, transitioning the technology to real-world industrial strength applications raises a different, yet challenging, set of technical issues.

Topics

We invite submissions describing innovations and implementations in all areas of speech and natural language processing technologies and systems that are relevant to real-word applications. The primary focus of this track is on papers that advance the understanding of, and demonstrate the effective handling of, practical issues related to the deployment of language processing technologies in non-trivial real-world systems. By “non-trivial real-world system” we mean an application that is deployed for real-world use, i.e. outside controlled environments such as a laboratories, classrooms or experimental crowd-sourced setups, and that uses natural language processing (including speech technology), even if not state of the art in terms of research. There is no requirement that the system be made by a for-profit company, but the users of the system are most likely outside the NLP research community.

This track provides an opportunity to highlight the key insights and new research challenges that arise from real world implementations. Topics include, but are not limited to, the following (in alphabetical order):

  • Best practices and lessons learned
  • Continuous maintenance and improvement of deployed systems
  • Case studies from design to deployment
  • Design of application-relevant datasets
  • Ethics, bias, and fairness in deployed NLP systems
  • Green NLP
  • Handling unexpected user behavior
  • Implementation at speed, scale, and low-cost
  • Negative results related to real-world applications
  • Novel previously unsolved NLP problems
  • Offline and online system evaluation methodologies
  • Online learning for deployed systems
  • System combination and hybridization

In addition, opinion/vision papers related to real-world applications are also welcome.

Submissions must clearly identify one of the following three areas they fall into:

  • Deployed: Must describe a system that solves a non-trivial real-world problem. The focus may include describing the problem related to actual use cases, its significance (against opportunity size, value proposition, and ideal end state), design/formulation of methods, tradeoff design decision for solutions, deployment challenges, and lessons learned.
  • Emerging: Must describe the development of a system that solves a non-trivial real-world problem (it need not be deployed or even close, but there needs to be evidence that this development is intended for real-world deployment). Papers that describe enabling infrastructure for large-scale deployment of natural language processing techniques also fall in this category.
  • Discovery: Must include results obtained from NLP applications in real world scenarios that result in actionable insights. These discoveries should reveal promising directions in their application areas, leading to further system or societal enhancements. For example, an actionable discovery from an analysis of call center transcripts may reveal that certain language choices negatively impact customer experience, leading to better training of service representatives and improved customer experience.

Important Dates

Anonymity period begins: December 15, 2021

Paper submission deadline: January 15, 2022 January 17, 2022

Notification of acceptance: April 7, 2022

Camera-ready version of papers due: May 3, 2022

Note: All deadlines are 11:59PM UTC-12:00 (“anywhere on Earth”).

Evaluation and decision criteria

Submissions will be reviewed in a double-blind manner and assessed based on their novelty, technical quality, potential impact, and clarity. Submissions in the industry track should emphasize real-world implementations of natural language processing systems, the development of such systems, or provide insights based on real-world datasets with obvious industry impact. For papers that rely heavily on empirical evaluations, the experimental methods and results should be clear, well executed, and repeatable (though the data may be proprietary).

Paper Submission

Submission: Authors are invited to submit original, full-length (6 page) industry track papers that are not previously published, accepted to be published, or under consideration for publication in any other forum. Manuscripts should be submitted electronically, in PDF format and formatted using ARR formatting requirements.

Industry Track papers cannot exceed 6 pages in length (excluding ethical considerations and references). The papers can have an optional appendix as described in ARR CFP guidelines. For example, pre-processing decisions, model parameters, feature templates, lengthy proofs or derivations, pseudocode, sample system inputs/outputs, and other details that are necessary for the exact replication of the work described in the paper can be put into appendices. The reviewers are not required to consider the appendix during the review process.

The papers should be submitted through the NAACL-HLT 2022 industry track online submission system in OpenReview.

Double Submission of Papers to NAACL Conferences

  • It is the responsibility of the authors to ensure that they do not submit the same paper to both ARR and the Industry Track.
  • Please note that ARR policy does not allow double submissions. You may not submit the same paper to the Industry Track and the Main Conference. Submissions that violate this policy will be rejected.
  • For papers that do overlap with other papers in the Main Conference, the authors must ensure that there is significant new content and results before submitting their paper to the Industry Track. The authors should also include the papers that their paper overlaps or extends in the references section as follows:
    • Anonymous Authors, “Title of the paper”, Under submission at NAACL 2022

Final version: Accepted papers will be given one additional page of content (up to 7 pages; ethical considerations, acknowledgements and references do not count against this limit) so that reviewers’ comments can be taken into account. Previous presentations of the work (e.g. preprints on arXiv.org) should be indicated in a footnote that should be excluded from the review submission, but included in the final version of papers appearing in the NAACL-HLT 2022 proceedings.

Presentation requirement for accepted papers: Industry track papers will be presented orally or as posters to be determined by the program committee. The decisions as to which papers will be presented orally and which ones as poster presentations will be based on the nature rather than the quality of the work. There will be no distinction in the proceedings between industry papers presented orally and those presented as posters. All accepted papers must be presented at the conference to appear in the proceedings. The 2022 Industry Track will run in parallel with the Research Track. At least one author of each accepted paper must register for NAACL-HLT 2022 by the early registration deadline. The format of the conference (hybrid or in-person) has not yet been decided. More details regarding this will be announced in the future.

Anonymity Period: The anonymity period for the NAACL-HLT 2022 Industry Track is from December 15, 2021 to April 7, 2022. You may not make a non-anonymized version of your paper available online to the general community (for example, via a preprint server) during the anonymity period. You may not update the non-anonymized version during the anonymity period, and we ask you not to advertise it on social media or take other actions that would further compromise double-blind reviewing during the anonymity period.

Ethics Policy

Authors are required to honor the ethical code set out in the ACL Code of Ethics, and comply with the ethics guidelines for ARR submissions.

The NAACL Industry Track submission form will include an ethics checklist similar to the one used for the main conference: authors will be asked that their paper take this list into account; reviewers will be asked to ensure that papers comply with instructions from the checklist.

The NAACL ethics committee will be approached to add ethics review expertise if Area Chairs request it on the basis of issues identified by the reviewers.

Contact Information

Email: naacl2022-industry-track@googlegroups.com

Industry Track Co-Chairs:

  • Rashmi Gangadharaiah (AWS AI, Amazon)
  • Anastassia Loukina (Grammarly)
  • Bonan Min (Raytheon BBN Technologies)

General chair: Dan Roth, University of Pennsylvania & Amazon

Frequently Asked Questions

Is the industry track only for participants from industry? No, the industry track welcomes participants from the entire ACL community. Researchers working on real-world applications that match the industry track call for papers are invited to submit papers. Everyone is welcome to attend industry track sessions.

What do you mean by real-world applications? We are looking for applications that are deployed (or expected to be deployed) for real-world use, i.e. outside controlled environments such as laboratories, classrooms or experimental crowd-sourced setups.

Can students also submit papers to industry track? Yes! If your work matches the industry track call for papers, consider submitting a paper to the industry track.

I work in industry. Can I still submit my paper to the research track? Absolutely! There are no changes to the main conference submissions. The industry track offers a forum to submit papers describing aspects of real-world applications that may differ in focus from the research track reviewing criteria.

Will the papers in the industry track be published in the proceedings? Yes, industry track papers will be published as a separate volume of the proceedings. For example, see the NAACL-HLT 2021 proceedings.

How do I decide whether to submit to the research track or the industry track? Papers describing key lessons learned and challenges pertaining to real-world deployment of NLP and speech technologies are best suited for industry track. Authors are advised to review the call for papers for both tracks and submit to the track that best matches your work. The list of topics and reviewing criteria may be helpful. You can also reach out to the track chairs if you need help deciding.