The article “Scalable Extraction of Training Data from (Production) Language Models” by Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee examines how training data can be extracted from language models. The authors focus on identifying training data stored in large language models (LLMs) and developing methods for extracting this data. They analyze both public and semi-private models and demonstrate that significant amounts of training data can be extracted, raising important privacy and security concerns.
The researchers successfully demonstrated that substantial amounts of training data can be extracted from language models. They found that public, semi-private, and fully closed language models, such as ChatGPT, are susceptible to data extraction. The study was successful with various language models, including public models like GPT-Neo, semi-private models like LLaMA and Falcon, and fully closed models like ChatGPT. Their methods revealed that practical attacks can extract far more data than previously assumed, and that current alignment techniques do not eliminate memorization.
Techniques for Extracting GPT Training Data
The scientists used a combination of manual and automated techniques. They developed new attack methods to circumvent the models’ alignment and induce them to reveal training data.
The researchers employed various techniques for extracting training data from language models: They developed special methods to circumvent the alignment of ChatGPT and get the model to reveal training data. This involved finding a query strategy that caused the model to deviate from its standard dialog style. Additionally, they used automated techniques for open models: They utilized automated procedures to extract training data from open sources such as Wikipedia and use them as input prompts. This method enabled them to extract verbatim memorizations from the models. Verbatim memorizations refer to the ability of language models to reproduce information word-for-word as it was present in the training data. This means that the model can reproduce text passages, facts, or data exactly as they were originally learned, without altering or paraphrasing them. Such memorizations are particularly relevant for discussions about privacy and security, as they show that sensitive or confidential information that is part of the training data can potentially be reproduced in the same way.
Additionally, they used manual techniques and comparisons with public web datasets: For semi-private models, they used a combination of manual techniques and automated comparisons with extensive internet text data to identify potential training data and verify its authenticity. These methods allowed the researchers to gain deep insights into the nature of the stored and revealed data of the examined language models.
The researchers extracted a variety of content, including personal information, copyrighted material, and sensitive data. They found that models can store and reveal training data in various forms, including text, code, and personally identifiable information (PII).
Key Learnings on the Extraction of GPT Training Data
The main insights from the article are the highlighting of security and privacy risks in language models, the need to improve security measures in these systems, and the awareness that larger and more powerful models are more susceptible to data extraction attacks. The research also emphasizes that current alignment and training methods do not completely prevent the memorization of training data.
Further Information:
Article: “Scalable Extraction of Training Data from (Production) Language Models”: https://arxiv.org/abs/2311.17035
Authors’ Blog Post: https://not-just-memorization.github.io/extracting-training-data-from-chatgpt.html