The development of any machine learning systems for high-stakes domains involving data, such as healthcare records, necessitates measures to protect users' data privacy. Some of these measures include policies such as the U.S Health Insurance Portability and Accountability Act (HIPAA) enforce this by mandating that any Protected Health Information (PHI), must be de-identified in electronic health records for secondary use. In addition to extensive redaction to prevent de-identification, sharing sensitive data also typically requires signing data usage agreements. However, this also means that this data remains accessible to a limited number of people and while redaction measures protect user privacy to some extent, they are by no means foolproof. Motivated by these limitations, research has explored privacy-preserving solutions beyond simple redaction, such as training models on synthetic equivalents of the real data. In a separate thread, while preventive measures that focus on limiting the exposure of private information by models are essential, it is imperative to consider how practitioners can empower users to act in accordance with their privacy preferences when interacting with these systems.
Much of the research on enabling the sharing of sensitive data from high-stakes domains focused on text anonymization, a means of replacing or redacting sensitive text spans such as names and addresses from text. These approaches have developed from ones relying on deterministic rule-based systems to more recent differentially private strategies to modify the private information in text. However, this text can still be prone to attacks that re-identify the text [1, 2], and even in the absence of direct identifiers, it has been demonstrated that systems can correctly infer personal attributes from text on the basis of the subtleties in the language cues and the context provided [3].
While de-identification is essential in that it can remove most direct identifiers in text, it stands that this text still remains susceptible to de-identification, which warrants further research that explores how we could minimize the risk of any adversarial attacks compromising the privacy of individuals included in the data.
[1]: Clinical Text Anonymization, its Influence on Downstream NLP Tasks and the Risk of Re-Identification (Ben Cheikh Larbi., 2023)
[2]: A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakage (Xin et al., 2024)
[3]: Beyond Memorization: Violating Privacy Via Inference with Large Language Models (Staab et al., 2024)
Synthetic data generation has emerged as a prominent thread in privacy-preserving research, as a means of reducing the likelihood of an individual's anonymity being compromised. The idea behind synthetic text generation is to create a synthetic distribution that is sufficiently reflective of the original data distribution, in that it could facilitate the development of tools or conducting analyses, while minimizing the risk of a privacy breach by abstracting out identifying information and patterns. There has been extensive research on generating synthetic data in a manner that enforces privacy guarantees by incorporating privacy-preserving elements. While some studies focus on fine-tuning models with differential privacy (DP) [1, 2] over sensitive corpora, others have developed methods that implement privacy-preservation strategies during model inference [3, 4].
[1]: Evaluating Differentially Private Synthetic Data Generation in High-Stakes Domains (Ramesh et al., 2024)
[2]: Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe (Yue et al., 2023)
[3]: Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation (Tang et al., 2023)
[4]: TextMixer: Mixing Multiple Inputs for Privacy-Preserving Inference (Zhou et al., 2023)
[1]research investigates how user self-disclosure tendencies may be influenced by the formality in the language used by the chatbot, in the context of healthcare. The conclusions primarily indicate that users tend to disclose private information regarding their health more often when the chatbot adopts a formal conversational style.
In the context of mental health, [2] examines the impact of the conversational behaviors of chatbots used as mediators between users and mental health professionals. The findings suggest that when chatbots engage in self-disclosure themselves (and in doing so, mimic human-like interactions), users are more inclined to share information that they have disclosed to the conversational agent with the mental health professionals.
[1]: Does Chatbot Language Formality Affect Users’ Self-Disclosure? (Cox et al., 2022)
[2] Designing a Chatbot as a Mediator for Promoting Deep Self-Disclosure to a Real Mental Health Professional (Lee et al., 2020)