Operator Comment Taxonomy Developed by NC State, UNC Added to National Security Policy

The National Security Agency recently updated its policy to incorporate an innovative commenting tool created by researchers from UNC-Chapel Hill and the Laboratory for Analytic Sciences at NC State. 

Language analysis professionals at the NSA are tasked with translating, transcribing, and, of course, analyzing foreign communications. They must understand the nuances of a foreign language, like context, cultural overtones, and regional dialects. When preparing intelligence reports for decision-makers, language analysts include comments that enable the reader to consider overtly stated meaning and subtly implied intent. 

The updated NSA policy statement focuses on the quality review process of foreign language analysis. One of the key changes is the direct result of a project conducted at the Laboratory for Analytic Sciences, a collaboration between government, academia, and industry that works to solve problems faced by the intelligence community. The policy now states that language analysts must follow a structured “operator comments” taxonomy to document a variety of considerations in language analysis.

Professors Robert Capra and Jaime Arguello at UNC School of Information and Library Science developed the taxonomy with guidance from LAS. It provides a standardized framework for creating operator comments — written notes that help provide clarity and guidance, like footnotes for the reader. The updated policy also requires operator comments to include a level of confidence.

Patti K., a language analyst at LAS who coordinated the project with UNC, says experienced language analysts know when to write an operator comment and what to include in it, but describing this to others was challenging. 

“Before this project, a software developer at LAS asked me, ‘What kinds of things could an analyst write operator comments about?’” says Patti K. “And my answer was ‘Literally anything.’” 

The developer and Patti K. made a rough prototype and showed it to Capra and Arguello, which interested them in working on the project. The UNC team helped define categories of comments and what should be included in each type of comment. 

Arguello says if he and Capra were discussing their work with LAS in a conversation, they might simply refer to it as “the project.” 

“[Capra] and I know what that means and what it entails,” says Arguello, but others would not. A language analyst with expert knowledge about their conversation could decide to make an operator comment to disambiguate this reference. In this instance, the analyst could label the “the project” with the “entity-organization” category. 

“The comment might explain that this refers to the ‘structured annotation of audio data project between UNC-SILS and LAS,’” Arguello says. “The operator comment might also describe the aim of the project and the people involved. This would enrich the data to support downstream analysis and decision-making.”

Developing the taxonomy categories has made it easier to teach new analysts how to make operator comments, Patti K. says. 

“Most of the analysts who participated in the study said they would like being able to use it to train junior analysts.” 

Translational research and cross-sector collaboration are the core of LAS. Each year, LAS collaborators from government, academia, and industry develop novel advancements in analytic tradecraft that strengthen national security operations, capabilities, and mission. 

“The UNC team is incredibly proud that our work is having a real impact,” Arguello says. “Of course, as researchers, we always aim for our research to have real impact. However, in this case, we have actual proof that our work has added value to an organization.” 

NSA signed the policy in March.