Emerging Methods To Protect Privacy In LLM

3 main points
✔️ Propose privacy preserving methods where local models extract high performance from remote models without sharing private data
Evaluates accuracy and privacy with three algorithms (problem description, new unlabeled example generation, and entity replacement)
✔️ Evaluates entity leakage and mapping leakage, with Method 3 being least leaky and Method 2 being optimal under auxiliary information

Can LLMs get help from other LLMs without revealing private information?
written by Florian Hartmann, Duc-Hieu Tran, Peter Kairouz, Victor Cărbune, Blaise Aguera y Arcas
(Submitted on 1 Apr 2024 (v1), last revised 2 Apr 2024 (this version, v2))
Comments: Published on arxiv.
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR); Multiagent Systems (cs.MA)

code：

The images used in this article are from the paper, the introductory slides, or were created based on them.

Summary

Large-scale language models such as Google's Gemini Ultra (2023) and OpenAI's GPT-4 (2023) have shown tremendous performance on many tasks. However, these models not only have high inference costs, but also run in data centers that are not local environments with private data available. On the other hand, models that can run in a private environment, such as the Gemini Nano, run on the user's device, but their performance is limited.

To achieve state-of-the-art performance in a private environment, local models with privacy-preserving mechanisms that can send queries to remote models without sharing sensitive data are needed. In traditional cascade systems, mechanisms for small, less capable models to throw queries to large, more capable models have been studied, but privacy-preserving mechanisms have not yet been explored. In current cascade systems, the decision to use a large model is usually based on whether the query can be handled independently by a smaller model.

If the decision is made to process on a large model, queries containing private data are transferred intact. This creates privacy threats, such as the leakage of sensitive data and the incorporation of the forwarded queries into the training data set of the remote system.

This paper proposes the first privacy-preserving approach to cascading systems. Unlike traditional cascade systems, the proposed approach assumes that the local model always keeps its data private. Therefore, the local model does not share any private information with the remote model. Furthermore, even if the local model does not share private information verbatim, it prevents operators in the remote model from using auxiliary information to reconstruct private data. To address these challenges, we assume that there are no efficiency constraints and that the local model can always seek help from the remote model. Thus, the optimal cascade setting for this approach is to minimize privacy loss while maximizing task performance.

To succeed in this task, the local model, which is usually smaller and less capable, must find a balance that protects private details while exposing enough information about the problem to receive useful signals from the more capable remote model. To learn from the remote model, the local model leverages the gradient-free learning capability through natural language using the in-context learning (ICL) capability of the large-scale language model. It also utilizes the recently introduced social learning paradigm by Mohtashami et al. (2023) and Bandura & Walters (1977) of learning through natural language from other large-scale language models.

Problem-Solving

This paper examines a form of social learning that does not have labeled data. The local model, the "student," has private data that cannot be well labeled by itself, while the larger remote model, the "teacher," can better label its data. These two models form a cascade, as shown in the figure below, where the "student"improves performance through communication with the"teacher.The information that the "student"sends to the"teacher" is called a "query."

This query from "student" to "teacher" has two constraints: thefirst concerns the protection of privacy.The secondconcernsthe single round of communication.Communication between the "student" and the "teacher" takes place in a single round, and neither can maintain state about the other's abilities or update the model.For this reason, all algorithms follow the same structure. That is, if the "student" needs help, the "student" generates a query to the "teacher" using private data. The "teacher" then uses that query to generate an ICL (In Context Learning) example for the "student". Finally, the "student" uses that ICL example to solve the original problem.

It also makes two further simplifying assumptions:first, itassumes that communication with the remote "teacher" is always beneficial;second,it assumes that both the "student" and the "teacher" are aware of the format of the example. This assumptionis useful because the"student" is not simplylearning theformat and chain of thought (CoT), butis learning more complex things about the data from the"teacher".

In such a problem setting, the goal of the "student" is to maximize performance in accurately labeling the data without revealing private information that is part of his or her own data.

Privacy Measures

The"student" data may contain sensitive personal information that should be hidden from the "teacher." For example, in a query describing health symptoms that a user experiences after a particular activity, linking the activity or set of symptoms to a specific user would be a privacy violation.

To address these privacy violations, data anonymization techniques such as differential privacy (DP) can be considered, but DP is useful for computing aggregates across many users. This can result in queries that are not useful for the task.It is also more difficult to make sensitive parts of the "student" data private using the DP-ICL model.

This is because we assume that the "student" will consider many private examples when generating the query. In this paper, we want to be able to generate privacy-preserving queries even when the "student" has only a few private examples. However, DP-ICL will not work in this setting.

Therefore, we utilize data minimization privacy techniques, especially contextual integrity. This describes privacy as the proper flow of information. In this technique, the "student" retains information that is useful for the task and removes personally identifying information that is not relevant to the context of the query. Even with a perfect mask, sensitive information could be compromised if the "teacher" model has access to auxiliary information that can be used to identify specific features. Therefore, an important contribution of the proposed approach is a methodology to measure and evaluate leakage under auxiliary information.

The success of the proposed approach depends on correctly identifying and masking the sensitive parts of the query without tampering with the task description. To this end, we propose various techniques to analyze the information in the query and generate secure queries that can be shared with a "teacher" model. To evaluate the privacy of a query, we consider two specific metrics: an entity leakage metric and a mapping leakage metric that considers a configuration that takes into account auxiliary information.

Contextual integrity defines privacy as the appropriate flow of information. In most production applications, it is difficult to determine what is appropriate to share. As a proxy for this, consider the interpretable metrics of the compromised entities. All entities in the data set, such as names, locations, email addresses, or numbers, are considered private. After masking, measure how many of the entities present in the original example remain in the "student" query.

Even if all entities are removed from the "student's" query, it is still possible for the "teacher" to reconstruct private information through careful analysis of the query. Indeed, the ancillary information to which the "teacher" has access helps to increase its effectiveness. We measure this analysis in the worst case scenario.

Specifically, the "teacher" is presented with the original example and 100 masked queries, assuming that only one of them is generated from the original example.We measure how often the"teacher" can correctly map the original example to this particular (masked) query out of the 100 options.Providing the "teacher" with the complete original examplerepresents the upper limit of auxiliary information the"teacher" may have. To better perform this mapping,allow the "teacher" to throw queries at the "student" model. This is useful because it was used to generate the masked query. To do the mapping, score the original example and the continuation of the 100 generated queries to measure how often the correct query gets the highest score. This (worst-case) access to ancillary information indicates that even if an entity is properly masked, it can lead to significant privacy leaks.

Proposed Method

It introduces three algorithms for helping the "student" learn privately from the "teacher."These methodsare based on a description of the problem the "student" faces and the generation of similar non-private examples that the "teacher" can label. All methods use a hyperparameter called extended size to indicate the number of labeled ICL examples the "student" receives from the "teacher".

The first method (Method 1) is to create an explanation of the problem.In this method, the "student" first analyzes a given problem and then turns it into a high-level explanation. Even if the problem cannot be solved, the "student" can explain the type of problem. This explanation becomes a query to the "teacher.The"teacher"thencreates a small number of examples tohelp the"student" solve the problem he/she is facing.TheTeacheruses the template for example structure togenerate new examples based on theStudent'sdescription.

Thesecond method(Method 2)generates new unlabeled examples.Instead of providing an abstract description of the problem the"student" is facing, a similar new problem is generated. For example, the GSM8k dataset of middle school level math problems could generate similar math problems that differ in detail but have equivalent educational value.

It has been shown that large language models can generate new examples from the original examples seen in context. For many tasks, generating examples is easier than solving them. In this approach, we (1)have the"student's"large-scale languagemodel generate new unlabeled examples, (2) have the "teacher" label these examples, and (3) have the "student" use those labels to solve the original problem.

Thethirdmethod(Method 3)replaces the entities of the original example.Instead of generating a completely new example, the"student" replaces entities such as names, locations, numbers, etc. in the original example. Thus a new unlabeled example is generated that resembles the original example. This method can also generate examples of extended size.

Large languagemodels do a fairly good job of replacing entities. Therefore, we instruct the "student" model to find and replace private entities. The overall flow of this method is the same as the second method, except that entities are replaced in step (1).

Inquiries to "teachers" carry the risk of divulging private information. Therefore, in order to use the "teacher" economically, we introduce a hyperparameter called group size. This indicates the number of private examples that the "student" will group to create ICL examples for the extended size. The "student" combines information from the grouped examples to synthesize new examples. Indicates the budget for "teacher" labeled examples to be created for each original example as Labeling Budget = Extended Size/Group Size. The "student" does not get to choose which examples are grouped.

Experiment

To evaluate the effectiveness of the proposed method, we evaluate it in terms of accuracy and privacy using various data sets and compare it to two baselines.

This paper uses the Gemini 1.0 strain models provided by Google.We use Ultra, the most powerful model inthe family,as the "teacher"and Nano-2 (3.5B parameter model, mobile deployable) as the "student. In some experiments, we use Pro as the "student" because the ability of the "student" model affects the performance of the method. In all experiments, we normalize task success rates by "teacher" performance based on previous reports on Nano performance (Google, 2023).

To demonstrate the versatility of the proposed method, we use the following diverse datasets

• GSM8k mathematical problem (Cobbe et al., 2021)
• Assistant intention detection (Srivastava et al., 2022)
• Subjective/objective classification (Conneau & Kiela, 2018)
• Machine translation with moderate resources (Tiedemann, 2020)

We also compare thebaselineto two baselines: a weak baseline and a strong baseline. In the weak baseline, the "student" does not communicate with the "teacher" at all and has no labeled data himself, resulting in a 0-shot setup. In the strong baseline, we evaluate the case where the "student" has access to 8 arbitrary perfectly labeled examples. These examples are considered a strong baseline because they are perfectly labeled and are identical to the task the "student" is trying to solve. In reality, however, such data does not exist and cannot be easily adapted to the "student's" problem.

The results are shown in the table below. In all datasets, the proposed method outperforms both the weak and strong baselines. However, for GSM8k, we found that a strong "student" model such as Pro.

(Method 3) performs very well on all data sets.This is likely because thequery generated by this methodis closest to the problem the"student" is trying to solve.(Method 1) performed the least well and proved to be the most difficult to make this method work well. For example, for tasks such as intention recognition, this method is less competitive because it can only explicitly account for unlabeled examples if the student model can be labeled.

In addition, a complete grid search was conducted using different techniques to investigate the best use of labeling budgets (extended size/group size). For each labeling budget, we obtain the best achievable performance. As shown in the figure below, these hyperparameter choices allow for budgets less than 1, which would not be possible without grouping.

In addition, to analyze how the proposed method performs in terms of privacy,we compute two metrics:an entity leakage metric and a mapping leakage metric that considers settings that take into account auxiliary information.We let Gemini 1.0 Ultra act as an entity detector, and name, private entities such as names, locations, numbers, etc. To ensure that it reliably detects the desired entities, we performed a manual validation on a subset of the examples. The results of this analysis are shown in the table below.

(Method 1) was observed to leak the most entities. While this method should theoretically generate the most high-level queries, it is difficult to make it work in practice. In a subset of the original example, the "student" may fail to synthesize a high-level description and revert to a detailed description of the problem being faced. On the other hand,the query in(Method 3) is the closest to the original message, but divulges the fewest entities. This is likely because the "student" can find and replace entities without a good understanding of the problem.

However, when analyzing a mapping metric that indicates the ability of an attacker with auxiliary information to identify the original example, the results are different. On this metric,(Method 3)is significantly inferior. Although this method leaks fewer entities, it preserves structure and style, making it particularly easy to map between the original and generated examples. This is especially true for the GSM8k and Subj data sets, where the structure is clear.

We found that thegrouped exampleworks particularly well with(Method 2). With group size = 2, leakage is significantly reduced for both metrics, which can be achieved with GSM8k with no performance degradation.

Finally, we note that the choice of the appropriate method depends on the specific threat model. （ While(Method 1)is not convincing from a quality and privacy perspective,(Method 3) works very well for threat models that do not include ancillary information. On the other hand,(Method 2)is most appropriate whenancillary information is considered.

We will also conduct a detailed analysis of the predictions that the "student" model can make after receiving assistance from the "teacher" in order to better understand where theproposedmethod works and where it falls short. To do this on a large scale, wewill haveGemini 1.0 Ultralook at thegolden labels and"student" predictions and classify the errors into specific classes. Manually check for some cases to ensure that these classifications are valid. The table below shows the results of this analysis on 500 cases of GSM8k based on a strong baseline and the best settings for each method.

Summary

In this paper, weinvestigate whether a large-scale language model can send privacy-preserving queries to anexternallarge-scale languagemodel and improve its performance. The results show that the proposed method significantly outperforms a strong baseline with privacy constraints.

To evaluate the privacy performance of the proposed method, we use two metrics: a simple metric that counts the number of entities compromised and a new metric that measures an upper bound on how much information a "teacher" with auxiliary information can recover from a "student" query. For the first metric, we found that themethod of masking questionswas effective, while the method of generating new questions waseffective when the "teacher" had auxiliary information.

Ultimately, which method is chosen depends on the specific threat model. However, for both threat models, the proposed method showed a low leakage rate and outperformed the strong quality baseline. Furthermore, grouping the samples improves the privacy metric, confirming that the quality of the model improves even under specific labeling budget constraints.

Future research is expected to examine more complex "student" and "teacher" interactions, further improve privacy metrics, and focus on modalities other than text.

• 
• 
• 

Categories related to this article

• Large Language Models
• Machine Learning
• Deep Learning
• Privacy
• Privacy Protection