It Is Clear That Human Memory Characteristics Are Present In LLM!

Large Language Models 27/09/2024

3 main points
✔️ Conducted various experiments to investigate the similarities between human memory characteristics and LLMs
✔️ Confirmed that human-specific phenomena such as the primacy effect, the lethality effect, and memory consolidation through repetition also appear in LLMs
✔️ LLMs to study human biological memory mechanisms LLMs have proven to be a useful tool for studying the mechanisms of human biological memory

Aspect of human memory and Large Language Models
written by Romuald A .janik
(Submitted on 7 Nov 2023 (v1), last revised 8 Apr 2024 (this version, v3))
Comments: Published on arxiv.
Subjects: Computation and Language (cs.CL); Artificial Intelligence(cs.AI); Machine Learning(cs.LG); Neurons and Cognition(q-bio.NC)

code：

The images used in this article are from the paper, the introductory slides, or were created based on them.

Introduction

Large Language Models (LLMs), exemplified by ChatGPT, have made great strides in our ability to model language and produce human-level text output.

By training on a huge text corpus, these models effectively build very complex and accurate probabilistic models of language.

On the other hand, the use of these languages is one of the most important characteristics of Homo sapiens, and understanding the interrelationship between human cognitive abilities and these language characteristics has been considered a very important area of research.

This, plus the fact that LLMs exhibit memory characteristics similar to human characteristics in existing studies, led the author of this paper to assert that "LLMs can be a very useful tool in investigations in this direction."

Against this background, thispaper describesvarious experiments conducted to investigate the similarities between human memory characteristics and LLM, and proves that human-specific phenomena such as the primacy effect, the lethality effect, and memory consolidation through repetition also appear in LLM.

Summary

Although human memory appears to be simple, it actually has very specific properties that have been studied for more than a century by many cognitive psychologists.

Typical examples of such memory propertiesare the primacy effects and recency effects, which are the phenomenon that words at the beginning or end of a list are more easily recalled when memorizing a list of words.

Various other human-specific memory characteristics have also been identified, such as the strengthening of memory through repetition at certain intervals.

In this paper, we argue that LLM can be a very useful tool for studying such human-specific memory characteristics, and then we actually conducted various experiments using LLM.

Experimental Setup

The standard memory testing technique in cognitive psychology involves giving participants a list of words in sequence, having them memorize their positions on the list, and then testing their recall accuracy.

On the other hand, it is difficult to adapt these methods to LLM, so this paper instead devises a textual construct to explore specific memory characteristics.

The procedure followed in this experiment is shown in the figure below.(In this paper, the open source model GPT-J is used throughout all experiments.)

In this experiment, instead of memorizing a list of words, the GPT-J is presented with a list of facts about any person identified by first name.

Then add the following query to GPT-J

The process is then to have GPT-J output the probabilities of all tokens instead of X. If the noun with the highest output probability matches the noun given to a particular person (here Paul) in the list of facts, the answer is considered correct.

In this paper, we conduct various experiments by changing the length, categories, and intervening text of the list of facts to be memorized, and investigate the memory characteristics that emerge.

Experimental Results

Primacy Effects / Recency Effects

To test whether the aforementioned primacy and lethality effects appear in the LLM, we computed recall accuracy as a function of the position of a given X in the list of facts.

Recall accuracy in human and GPT-J memory experiments with a list of 20 facts is shown in the figure below.

The U-shaped curve seen in this graph is a phenomenon specific to the primacy and lethality effects, and the resultsconfirm that the primacy and lethality effects appear inLLMs as well as in humans.

Add Information

Another feature found in tests of human memory is that inserting additional information about a given word improves the likelihood of recalling that word, even if the query does not contain the added information.

In order to test whether a similar phenomenon appears in LLMs in this paper, we inserted the following additional information at certain positions in the list (5th, 10th, and 15th).

A comparison of GPT-J recall accuracy between baseline and after inserting additional information is shown in the figure below.

The figure clearly shows that the insertion of additional information improves recall accuracy.

Reinforcement of Memory through Repetition

It is clear that repetition of memorization of given material enhances memorization, and in this regard, the LLM is expected to exhibit similar behavior.

In the case of human memory, the psychologist Ebbinghaus stated that "memory is most effective when there is a certain interval of time after the initial memorization of the material to be learned (= Ebbinghaus' forgetting curve)," and this paper conducts an experiment to determine whether this property also appears in LLMs. In this paper, we conduct an experiment to see if this property also appears in LLM.

The specific procedure was to insert a repetiton (= list of facts to be memorized) before the sentence of the list of facts to be given to the GPT-J as described above, so that the information to be memorized appears repeatedly in the text.

The results of the experiment compared to the normal baseline are shown in the figure below.

As the figure shows,LLM (=repeated), which is set so that the information to be remembered appears repeatedly in the text,shows a significant improvement in recall accuracy compared to the baseline.

In addition, we observed a trend, consistent with human memory characteristics, that recall accuracy was improved when the text repeating information was located farther away from the text in the list of facts (=separated).

Summary

How was it?In this issue, we haveconducted various experiments to investigate the similarities between human memory characteristics and LLM, and described a paper thatproved that human-specific phenomena such as the primacy effect, the lethality effect, and memory consolidation through repetition also appear in LLM.

The experiments conducted in this paper confirm numerous similarities between the memory characteristics of humans and LLMs, and these results suggest that LLMs are a very useful tool for studying the mechanisms of human biological memory.

With regard to the results of this experiment, the author "argues that the human-like memory properties of LLMs are not automatically derived from the LLM architecture, but rather are learned from the statistics of the training text data," so we look forward to additional research to substantiate this hypothesis in the future! .

The details of the results of the experiments presented here are in this paper, and those interested should refer to it.