This article may be too technical for most readers to understand. Please help improve it to make it understandable to non-experts, without removing the technical details. (March 2022) (Learn how and when to remove this message)

Long short-term memory (LSTM)^[1] is a type of recurrent neural network (RNN) aimed at dealing with the vanishing gradient problem^[2] present in traditional RNNs. Its relative insensitivity to gap length is its advantage over other RNNs, hidden Markov models and other sequence learning methods. It aims to provide a short-term memory for RNN that can last thousands of timesteps, thus "long short-term memory".^[1] It is applicable to classification, processing and predicting data based on time series, such as in handwriting,^[3] speech recognition,^[4][5] machine translation,^[6][7] speech activity detection,^[8] robot control,^[9][10] video games,^[11][12] and healthcare.^[13]

A common LSTM unit is composed of a cell, an input gate, an output gate^[14] and a forget gate.^[15] The cell remembers values over arbitrary time intervals and the three gates regulate the flow of information into and out of the cell. Forget gates decide what information to discard from a previous state by assigning a previous state, compared to a current input, a value between 0 and 1. A (rounded) value of 1 means to keep the information, and a value of 0 means to discard it. Input gates decide which pieces of new information to store in the current state, using the same system as forget gates. Output gates control which pieces of information in the current state to output by assigning a value from 0 to 1 to the information, considering the previous and current states. Selectively outputting relevant information from the current state allows the LSTM network to maintain useful, long-term dependencies to make predictions, both in current and future time-steps.

Motivation

In theory, classic RNNs can keep track of arbitrary long-term dependencies in the input sequences. The problem with classic RNNs is computational (or practical) in nature: when training a classic RNN using back-propagation, the long-term gradients which are back-propagated can "vanish", meaning they can tend to zero due to very small numbers creeping into the computations, causing the model to effectively stop learning. RNNs using LSTM units partially solve the vanishing gradient problem, because LSTM units allow gradients to also flow with little to no attenuation. However, LSTM networks can still suffer from the exploding gradient problem.^[16]

The intuition behind the LSTM architecture is to create an additional module in a neural network that learns when to remember and when to forget pertinent information.^[15] In other words, the network effectively learns which information might be needed later on in a sequence and when that information is no longer needed. For instance, in the context of natural language processing, the network can learn grammatical dependencies.^[17] An LSTM might process the sentence "Dave, as a result of his controversial claims, is now a pariah" by remembering the (statistically likely) grammatical gender and number of the subject Dave, note that this information is pertinent for the pronoun his and note that this information is no longer important after the verb is.

Variants

In the equations below, the lowercase variables represent vectors. Matrices ${\displaystyle W_{q}}$ and ${\displaystyle U_{q}}$ contain, respectively, the weights of the input and recurrent connections, where the subscript ${\displaystyle _{q}}$ can either be the input gate ${\displaystyle i}$, output gate ${\displaystyle o}$, the forget gate ${\displaystyle f}$ or the memory cell ${\displaystyle c}$, depending on the activation being calculated. In this section, we are thus using a "vector notation". So, for example, ${\displaystyle c_{t}\in \mathbb {R} ^{h}}$ is not just one unit of one LSTM cell, but contains ${\displaystyle h}$ LSTM cell's units.

LSTM with a forget gate

The compact forms of the equations for the forward pass of an LSTM cell with a forget gate are:^[1][15]

${\displaystyle {\begin{aligned}f_{t}&=\sigma _{g}(W_{f}x_{t}+U_{f}h_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}x_{t}+U_{i}h_{t-1}+b_{i})\\o_{t}&=\sigma _{g}(W_{o}x_{t}+U_{o}h_{t-1}+b_{o})\\{\tilde {c}}_{t}&=\sigma _{c}(W_{c}x_{t}+U_{c}h_{t-1}+b_{c})\\c_{t}&=f_{t}\odot c_{t-1}+i_{t}\odot {\tilde {c}}_{t}\\h_{t}&=o_{t}\odot \sigma _{h}(c_{t})\end{aligned}}}$

where the initial values are ${\displaystyle c_{0}=0}$ and ${\displaystyle h_{0}=0}$ and the operator ${\displaystyle \odot }$ denotes the Hadamard product (element-wise product). The subscript $$Zdroj:https://en.wikipedia.org?pojem=Long_short-term_memory
Text je dostupný za podmienok Creative Commons Attribution/Share-Alike License 3.0 Unported; prípadne za ďalších podmienok. Podrobnejšie informácie nájdete na stránke Podmienky použitia.