Impact Report

Message from LAS Leadership

This year, NC State proudly celebrated the 10th anniversary of LAS – our collaborative partnership with the National Security Agency that develops technology and tradecraft to improve intelligence analysis. LAS research has evolved since 2013, but our mission remains the same: enabling rapid innovation by bringing together the cutting edge of academia and industry to partner with government analysts and researchers to find solutions to mission challenges. At NC State our faculty and students strive to provide real solutions for real problems, and we are proud of our work with the defense community. This report includes engaging stories from students, faculty, industry, and government collaborators about the impact of our work in 2023. We are excited about the collaboration, and we hope that this report conveys that enthusiasm to you. Enjoy the report.

Alyson G. Wilson, Ph.D.
Principal Investigator, LAS

It is our pleasure to share this 2023 impact report highlighting the many accomplishments of our entire team here at the Laboratory for Analytic Sciences (LAS). Intelligence analysis is a complex problem-solving activity resulting in actionable information that informs strategic decisions and safeguards national security. Though the customer demand for this information is high, the landscape of data to evaluate continues to grow at an exponential rate. Because of this data volume challenge, today’s intel analyst needs a village of support to help them conduct their analysis in a timely and efficient manner. Results from applied research in areas such as data science, machine learning and artificial intelligence can be incorporated into analytic workflows in order to optimize the intelligence analysis process. LAS exists as a premier example highlighting how an effective partnership with academia and private industry can result in innovative solutions in support of the intelligence community. I am thrilled to be a part of this organization and I truly hope you enjoy this publication.

Jacqueline Selig-Gumtow
Director, LAS

LAS By the Numbers

Jump to:

Download:

Mission

The Laboratory for Analytic Sciences is a partnership between the intelligence community and North Carolina State University that develops innovative technology and tradecraft to help solve mission-relevant problems. Founded in 2013 by the National Security Agency and NC State, each year LAS brings together collaborators from three sectors – industry, academia and government – to conduct research that has a direct impact on national security.

Synthetic Data Generation with Fayetteville State University

When computer vision models try to detect objects such as customized weapons or adversaries’ flags in foggy or dark environments, the models tend to perform poorly because of the lack of training data. Leveraging synthetic data generation can be one viable method for training AI computer vision models to detect objects that are rare or uncommon, which means they are not represented in widely used computer vision datasets.

LAS researchers Felecia M. and James S. began their partnership in 2022 with Dr. Sambit Bhattacharya, a computer science professor at Fayetteville State University (FSU) and leading expert in applied computer vision, through the NSA’s Minority Serving Institution Cooperative Research and Development Agreement. With the assistance of his senior design students, Bhattacharya and the LAS developed techniques to augment training data leveraging synthetic data generation and generative adversarial networks to improve the detection of rare objects such as Chinese-manufactured or street cameras (CCTVs).

In 2023, the team conducted research to increase the fidelity and photorealism of the objects and the environments in which they were contained using software graphic tools.

“By placing those objects in various environmental conditions – for example, cities, towns, or different weather conditions – our team increased the variety of synthetic data that correspond to many different conditions that arise in the real world,” Bhattacharya says.

Researchers also used few-shot learning to improve the performance of models trained on synthetic data for the detection of small objects such as CCTVs on real data. The team explored devising a process to semi-automate the generation of both synthetic objects and environments to assist analysts by quickly training AI models for rare object detection.

“Leveraging synthetic data generation to augment training data for the detection of rare objects in videos has many benefits such as minimizing the need for collecting ground truth data in potentially risky settings, minimizing any sensitivities in the data such as personal identifiable information, and automatic annotation which reduces the labeling burden among analysts,” Felecia M. says.

FSU students were able to work on a real-world mission problem that increased their skill sets in critical thinking, AI/ML, applying technological approaches to problem-solving, and developing a capability that addresses an immediate need of the NSA. As a result, the students became highly interested in pursuing careers in the intelligence community, and in 2023, four FSU students received offers for internships and/or full-time positions at the NSA.

“I primarily worked with the LAS for training models to aid in the detection of different objects such as CCTVs and weapon vehicles. Our dataset was created utilizing Unreal Engine to create and test within different simulated environments to run our trained models against actual real-world examples. The environments were made to simulate different locations and weather conditions ideally to aid in our testing. This led to the high confidence in detecting the objects of interest. This project increased my interest in … how the NSA utilizes [machine learning] and raised personal curiosity in other projects the NSA works on dealing with both software and hardware.”

– Melvin B., Student, Fayetteville State University

Knowledge MiNER with Lithios 

“Knowledge MiNER uses AI-infused workflows to streamline manual knowledge curation processes and reduce cognitive load on analysts, while facilitating analyst interaction and correction at all stages of the process.”
– Susie B., LAS government lead, Knowledge MiNER project, 2023

As NSA analysts triage and assess massive amounts of data, they concurrently create a variety of text annotations as part of their workflows to make sense of that data and answer customer needs. But unlike machines, humans tend to use unstructured, narrative formats like comments, notes, transcripts, or report drafts when documenting complex concepts. These contain valuable human-verified knowledge, often with nuances from subject matter experts and inferences based on the evaluation of many pieces of data over time.

There are challenges to organizing this unstructured knowledge and making it discoverable for other analysts. The current process for structuring such knowledge and extracting relationships in the form of linking entities to a corporate repository is done manually. The Knowledge MiNER project at LAS proposed a solution that uses a unique human-in-the-loop interface to apply machine learning and AI technologies across multiple tools and ultimately improve analysts’ ability to share knowledge.

In collaboration with Lithios, a Raleigh-based company chosen for its expertise in user interface development, LAS researchers initially created a prototype for prior work called RedThread, which aimed to help analysts build structured knowledge graphs from unstructured text.

In 2023, LAS continued to work with Lithios to build upon this foundation and create the second-generation prototype: Knowledge MiNER. It uses an iterative design process during which analyst input guided a new interface that further decreases the cognitive burden placed on analysts during the tedious and often repetitive process of structuring unstructured text for knowledge curation and sharing.

A combination of a named entity recognition (NER) model, an underlying knowledge graph with a predefined ontology, and a MiniBert information extraction question and answer model are used under the hood, but this project does not propose to replace existing NSA initiatives in exploring these technologies. Rather, Knowledge MiNER is a human-machine teaming project at its core, which uses AI/ML technology where analysts are already intellectually invested and engaged. It offers a unique interface that can work across already existing tools to help automate the entity linking task while helping analysts populate structured repositories via information extraction more efficiently.

Spotlight on NC State Professor Helen Armstrong

Helen Armstrong, a professor at the NC State College of Design, began collaborating with LAS in 2021 as a part of a studio project in her graduate course. This initiative led to additional studio projects over the next two years, establishing a research partnership between LAS and the College of Design, with Armstrong as principal investigator. The collaboration focused on the relationship between design and knowledge graphs, with the goal being to improve user experience so that analysts could better understand relevant data and forge useful insights.

“Our design team is interested in exploring interfaces that establish trust between humans and machines,” says Armstrong. “The LAS provides [scenarios, data, and sample users] and situates the work in a critical context: the intelligence community.”

Together, LAS and the College of Design students collaborated on projects that addressed critical use cases, improving user interface and experience with applications relevant to both government and industry purposes. The collaboration continues in 2024.

“A colleague once told me that trying to move forward with a large research project can be like trying to steer an awkward, clunky freighter in a fog. The journey with LAS has felt more like a cruiser moving steadily through calm waters.”

“I like to think about human-machine teaming like I’m making dinner and I have a perfectly balanced, well-sharpened knife that … allows me to be the chef that I want to be. A successful human-machine teaming to me is one where the human comes away saying, ‘this is a great tool.’” – Robert Capra, UNC School of Information and Library Science

A Prototype Tool for Annotation in Audio Data with Professors Robert Capra and Jaime Arguello at UNC-Chapel Hill

Dr. Robert Capra, a professor in the UNC School of Information and Library Science, first started collaborating with LAS in 2020. Acting as principal investigator alongside Dr. Jaime Arguello, Capra and his team worked to improve and develop tools that supported the needs of those searching for information.

“We come at information problems from different directions and so, when we look at tools to support us in making sense of information, it’s not a one-size fits all approach,” Capra says. “We’re designing how the machines interact with us. We want to understand what dials and knobs we need so that we can tailor the machines to help us with information tasks.”

In 2023, Capra and Arguello’s work with LAS focused primarily on providing a structure for operators to include comments as they transcribe audio data. Along with government employees and LAS staff, he and his team conducted a study among intelligence analysts, using transcripts from the Nixon White House Tapes. Ultimately, the team developed a prototype system that enables analysts to make operator comments with organization and clarity.

2023 Scholarly Publications 

We’re proud to share the latest research articles published by our collaborators.

1. Zhilan Zhou, Wenyuan Wang, Mengtian Guo, Yue Wang, David Gotz, “A Design Space for Surfacing Content Recommendations in Visual Analytic Platforms,” IEEE Transactions on Visualization and Computer Graphics, Vol. 29, No. 1, 2023, pp. 84-94.
2. Xiaohan Zhang, Saeid Amiri, Jivko Sinapov, Jesse Thomason, Peter Stone, Shiqi Zhang, “Multimodal embodied attribute learning by robots for object-centric action policies,” Autonomous Robots, Vol. 47, 2023, pp. 505-528.
3. Junhua Su, Alexandros Kapravelos, “Automatic Discovery of Emerging Browser Fingerprinting Techniques,” WWW ’23: Proceedings of the ACM Web Conference 2023, Austin, Texas, USA, 2023, pp. 2178–2188.
4. Benjamin Strickson, Cameron Worsley, Stewart Bertram, “Human-centered Assessment of Automated Tools for Improved Cyber Situational Awareness,” 2023 15th International Conference on Cyber Conflict: Meeting Reality (CyCon), Tallinn, Estonia, 2023, pp. 273-286.
5. Jing Ao, Zehui Cheng, Rada Chirkova, Phokion G. Kolaitis, “Theory and Practice of Relationalto-RDF Temporal Data Exchange and Query Answering,” Journal of Data and Information Quality, Vol. 15, Iss. 2, No. 15, 2023, pp. 1–27.
6. Andrew Freeman, Montek Singh, Kenan Mayer-Patel, “An Asynchronous Intensity Representation for Framed and Event Video Sources,” MMSys ’23: Proceedings of the 14th Conference on ACM Multimedia Systems, Vancouver, Canada, 2023, pp. 74-85.
7. Feng Cheng, Xizi Wang, Jie Lei, David Crandall, Mohit Bansal, Gedas Bertasius, “VINDLU: A Recipe for Effective Video-andLanguage Pretraining,” 2023 IEEE/ CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, Canada, 2023, pp. 10739-10750.
8. Zhen Guo, Munindar P. Singh, “Representing and Determining Argumentative Relevance in Online Discussions: A General Approach,” Proceedings of the International AAAI Conference on Web and Social Media, Vol. 17, No. 1, 2023, pp. 292-302.
9. Khuzaima Hameed, Rob Johnston, Brent Younce, Minh Tang, Alyson Wilson, “Motif-Based Exploratory Data Analysis for State-Backed Platform Manipulation on Twitter,” Proceedings of the International AAAI Conference on Web and Social Media, Vol. 17, No. 1, 2023, pp. 315-326.
10. Mengtian Guo, Zhilan Zhou, David Gotz, Yue Wang, “GRAFS: Graphical Faceted Search System to Support Conceptual Understanding in Exploratory Search,” ACM Transactions on Interactive Intelligent Systems, Vol. 13, Iss. 2, No. 9, 2023, pp. 1–36.
11. Abinay R. Naini, Mary A. Kohler, Carlos Busso, “Unsupervised Domain Adaptation for Preference Learning Based Speech Emotion Recognition,” ICASSP 2023 – 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 2023, pp. 1-5.
12. Mohammad Rostani, “I2I: Initializing Adapters with Improvised Knowledge,” 2nd Conference on Lifelong Learning Agents (CoLLAs), Montreal, Canada, 2023, 45139.
13. Jing Ao, Kara Schatz, Rada Chirkova, “Trend Surfing: Effective and Efficient Retrieval of Unusual Temporal Trends,” Computer Science & Information Technology, Vol. 13, No. 16, 2023, pp. 101-123.
14. Saugat Pandey, Oen G. McKinley, R. Jordan Crouser, Alvitta Ottley, “Do You Trust What You See? Toward A Multidimensional Measure of Trust in Visualization,” 2023 IEEE Visualization and Visual Analytics (VIS), Melbourne, Australia, 2023, pp. 26-30.
15. Violet Burbank, John M. Conroy, Sean Lynch, Neil P. Molino, Julia S. Yang, “Fast Extractive Summarization, Abstractive Summarization, and Hybrid Summarization for CrisisFACTS at TREC 2023,” The Thirty-Second Text REtrieval Conference (TREC 2023), Gaithersburg, MD, USA, 2023, 45231.
16. Matthew Peterson, Ashley L. Anderson, Kayla Rondinelli, Helen Armstrong, “The Pictorial Trapzoid: Adapting McCloud’s Big Triangle for Creative Semiotic Precision in Generative Text-to-Image AI,” Visible Language, Vol. 57, No. 3, 2023, pp. 6-51.
17. Mohammad Rostami, Yuliang Cai, Jesse Thomason, “Task-Attentive Transformer Architecture for Continual Learning of Vision-andLanguage Tasks Using Knowledge Distillation,” Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, 2023, pp. 6986–7000.
18. Bogeum Choi, Sarah Casteel, Jaime Arguello, Robert Capra, “Better Understanding Procedural Search Tasks: Perceptions, Behaviors, and Challenges,” ACM Transactions on Information Systems, Vol. 42, Iss. 3, No. 65, 2023, pp. 1–32.
19. A. R. Naini, S. Subramanium, S. -G. Leem, C. Busso, “Combining Relative and Absolute Learning Formulations to Predict Emotional Attributes From Speech,” 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Taipei, Taiwan, 2023, pp. 1-8.
20. Kylie Davidson, Lee Lisle, Kirsten Whitley, Doug A. Bowman, Chris North, “Exploring the Evolution of Sensemaking Strategies in Immersive Space to Think,” IEEE Transactions on Visualization and Computer Graphics, Vol. 29, No. 12, 2023, pp. 5294-5307.

SCADS Research Published in Analytics

In May 2023, the peer-reviewed academic journal Analytics published a special issue featuring research from the 2022 Summer Conference on Applied Data Science (SCADS) held by LAS. The issue highlighted collaborative research between industry, academia, and government on practical applications of data science. Analytics is an international, peer-reviewed, open-access journal on methodologies, technologies, and applications of analytics, published quarterly online by the Multidisciplinary Digital Publishing Institute. The articles published in the special issue include:

1. Grant Forbes, Jordan Crouser, “Metric Ensembles Aid in Explainability: A Case Study with Wikipedia Data,” Analytics 2023, Vol. 2, Iss. 2, 2023, pp. 315-327.
2. Johnathan Stray, “The AI Learns to Lie to Please You: Preventing Biased Feedback Loops in Machine-Assisted Intelligence Analysis,” Analytics 2023, Vol. 2, Iss. 2, 2023, pp. 350-358.
3. Jeremy E. Block, Ilana Bookner, Sharon Lynn Chu, R. Jordan Crouser, Donald R. Honeycutt, Rebecca M. Jonas, Abhishek Kulkarni, Yancy Vance Paredes, Eric D. Ragan, “Preliminary Perspectives on Information Passing in the Intelligence Community,” Analytics 2023, Vol. 2, Iss. 2, 2023, pp. 509-529.
4. Clinton T. White, Neil P. Molino, Julia S. Lang, John M. Conroy, “occams: A Text Summarization Package,” Analytics 2023, Vol. 2, Iss. 3, 2023, pp. 546-559.

2023 Summer Conference on Applied Data Science

Searching Video with EyeReckon 

The EyeReckon project comprises LAS’s video triage efforts. These efforts help government analysts better sort and sift through data as quickly as possible.

In 2023, EyeReckon had several ad hoc project transition wins in the form of rapid development and deployment of Python notebooks to get them in the hands of analysts for immediate mission use.

“Instead of focusing heavily on the machine learning and technology, I connected directly with an analyst who needed to sift through some videos,” says Stephen W., EyeReckon’s project lead.

Stephen W. leveraged the connection with NC State to fast-track a prototype. Where the analyst previously had to manually review each video, he was able to provide samples on a subset of each video.

“Combining that with other information we had about the data allowed me to group the corresponding videos in a nicer format,” Stephen W. says. “As a result of an iterative back-and-forth, we were able to set up an easy DIY solution for the analyst, which streamlined his initial triage of video data and allowed the analyst to focus more time on higher-value data. Ultimately, the speed up has led to more reporting and valuable intelligence to senior leaders.” The technology behind this success wasn’t a fancy LLM or even deep learning. It was a simple sampling strategy that enabled quicker data viewing. With the integration of current LAS collaborator research, there is still room for future refinement.

“The success of this transition was dependent largely on one element – understanding the customer’s needs and putting their workflow first, not technology,” Stephen W. says.

Valuable NC Partnerships: Spotlight on SAS Institute 

SAS Institute, one of the world’s largest privately held software companies, has partnered with LAS since the lab’s launch in 2013. Headquartered in Cary, NC, SAS envisions a world where everyone can make better decisions, grounded in trusted data. “Working with LAS provides [our] team with opportunities for research and creativity,” says Patrick Dougherty, project manager for SAS’s collaborations with LAS since 2017.

In 2023, SAS worked with LAS on a machine learning project that detects and labels objects within video footage, like public security cameras seen in a video recorded by a car dashboard camera. The SAS team prototyped a process to quickly fine-tune and deploy object detection methods on new video data without requiring mass labeling.

LAS’s interest in video analysis workflows is one of many use cases that are generally looking for ways to “automagically” find the important data and place them in user’s work queues.

It’s a collaborative partnership where we focus on common objectives that benefit our nation.

Patrick Dougherty

SAS

“The problem is universal – a new haystack of data arrives daily containing a handful of important needles,” Dougherty says. “Our collaboration with LAS in 2023 brought us a bit closer to that automagic tool by generating prioritized labeling methods to train computer vision models looking for novel objects – in this case CCTVs – in video.”

After personally working on LAS projects for six years, Dougherty has some perspective on what makes them unique and impactful. “It is a true partnership to explore ways to solve analytic challenges. Ideas are freely exchanged and respectfully debated,” says Dougherty. “For us, it is a bit different from our typical [sponsored] projects, which is a pleasant experience for our team.”

A Workforce Pipeline

Outreach and Engagement with 40 Students from 3 of NC’s Minority-Serving Institutions

• Fayetteville State University (2023 LAS Collaborator)
• University of North Carolina at Pembroke (Fall 2023)
• Winston-Salem State University (Spring 2023)

With proximity to 21 minority-serving institutions (MSIs) in North Carolina, LAS staff in Raleigh work to strengthen the National Security Agency’s partnership with underrepresented MSIs. These partnerships attract untapped talent with diverse backgrounds and skill sets to diversify the agency’s workforce. They also enhance research infrastructure and expertise at MSIs and establish a foundation for continued engagement with the NSA. In 2023, LAS developed two new partnerships with the University of North Carolina at Pembroke and Winston-Salem State University. It also continued its partnership with Fayetteville State University.

From LAS to NSA

“[With LAS], I was able to work on and build an AI model that could detect tanks, planes, rocket launchers and even tiny cameras. I got the opportunity to present and show at two LAS symposiums … The team at LAS was always helpful and made sure to guide us in the right direction. It’s a combination of these things that led to my interest in wanting to work at the NSA.”

– Givante L., Student

“LAS taught me the fundamentals of the intelligence communities’ mission needs and provided me with a network of enthusiastic Agency employees who supported my technical development and encouraged me to apply to the Agency as a full-time employee. Since joining the Agency, I have been able to continue to build on the skills I learned at LAS and apply this knowledge toward mission objectives.”

– NSA Employee/Former LAS intern

Student Engagement

Industry Spotlight: Chris Argenta and Rockfish Research

From global companies to small businesses, LAS provides its industry collaborators with access to a diverse group of experts, staff with varied subject matter expertise, and an organized research infrastructure.

“Interesting challenges are often found at the intersection of multiple domains, so it is critical for researchers to pool their insights, experience, and creativity to develop innovative solutions,” says Chris Argenta, founder of Rockfish Research. “LAS has cultivated an environment of multi-disciplinary experts across industry, academia, and government with the shared goal of solving challenges for the [intelligence community]. As a new small business, Rockfish Research benefits by the empowerment to explore new ideas in collaboration with a community of experts that wants to see them succeed.”

Argenta developed a prototype at the 2023 Summer Conference on Applied Data Science to help intelligence analysts thrive in pursuing a longer-term challenge.

“[We] tried new things that proved successful, and started turning our solution into an open framework to help accelerate the community developing a tailored daily report,” he says. “We are pleased to be part of LAS’s collaborative research process that will eventually result in an innovative solution that empowers the intelligence community.”