Gated recurrent units (GRUs) are a gating mechanism in recurrent neural networks, introduced in 2014 by Kyunghyun Cho et al.^[1] The GRU is like a long short-term memory (LSTM) with a gating mechanism to input or forget certain features,^[2] but lacks a context vector or output gate, resulting in fewer parameters than LSTM.^[3] GRU's performance on certain tasks of polyphonic music modeling, speech signal modeling and natural language processing was found to be similar to that of LSTM.^[4][5] GRUs showed that gating is indeed helpful in general, and Bengio's team came to no concrete conclusion on which of the two gating units was better.^[6][7]

Architecture

There are several variations on the full gated unit, with gating done using the previous hidden state and the bias in various combinations, and a simplified form called minimal gated unit.^[8]

The operator ${\displaystyle \odot }$ denotes the Hadamard product in the following.

Fully gated unit

Initially, for ${\displaystyle t=0}$, the output vector is ${\displaystyle h_{0}=0}$.

${\displaystyle {\begin{aligned}z_{t}&=\sigma (W_{z}x_{t}+U_{z}h_{t-1}+b_{z})\\r_{t}&=\sigma (W_{r}x_{t}+U_{r}h_{t-1}+b_{r})\\{\hat {h}}_{t}&=\phi (W_{h}x_{t}+U_{h}(r_{t}\odot h_{t-1})+b_{h})\\h_{t}&=(1-z_{t})\odot h_{t-1}+z_{t}\odot {\hat {h}}_{t}\end{aligned}}}$

Variables (${\displaystyle d}$ denotes the number of input features and ${\displaystyle e}$ the number of output features):

• ${\displaystyle x_{t}\in \mathbb {R} ^{d}}$: input vector
• ${\displaystyle h_{t}\in \mathbb {R} ^{e}}$: output vector
• ${\displaystyle {\hat {h}}_{t}\in \mathbb {R} ^{e}}$: candidate activation vector
• ${\displaystyle z_{t}\in (0,1)^{e}}$: update gate vector
• ${\displaystyle r_{t}\in (0,1)^{e}}$: reset gate vector
• ${\displaystyle W\in \mathbb {R} ^{d\times e}}$, ${\displaystyle U\in \mathbb {R} ^{e\times e}}$ and ${\displaystyle b\in \mathbb {R} ^{e}}$: parameter matrices and vector which need to be learned during training

Activation functions

• ${\displaystyle \sigma }$: The original is a logistic function.
• ${\displaystyle \phi }$: The original is a hyperbolic tangent.

Alternative activation functions are possible, provided that ${\displaystyle \sigma (x)\in }$.

Alternate forms can be created by changing ${\displaystyle z_{t}}$ and ${\displaystyle r_{t}}$^[9]

• Type 1, each gate depends only on the previous hidden state and the bias.

  Zdroj:https://en.wikipedia.org?pojem=Gated_recurrent_unit
  Text je dostupný za podmienok Creative Commons Attribution/Share-Alike License 3.0 Unported; prípadne za ďalších podmienok. Podrobnejšie informácie nájdete na stránke Podmienky použitia.

  ---