Sparse dictionary learning (also known as sparse coding or SDL) is a representation learning method which aims at finding a sparse representation of the input data in the form of a linear combination of basic elements as well as those basic elements themselves. These elements are called atoms and they compose a dictionary. Atoms in the dictionary are not required to be orthogonal, and they may be an over-complete spanning set. This problem setup also allows the dimensionality of the signals being represented to be higher than the one of the signals being observed. The above two properties lead to having seemingly redundant atoms that allow multiple representations of the same signal but also provide an improvement in sparsity and flexibility of the representation.

One of the most important applications of sparse dictionary learning is in the field of compressed sensing or signal recovery. In compressed sensing, a high-dimensional signal can be recovered with only a few linear measurements provided that the signal is sparse or nearly sparse. Since not all signals satisfy this sparsity condition, it is of great importance to find a sparse representation of that signal such as the wavelet transform or the directional gradient of a rasterized matrix. Once a matrix or a high dimensional vector is transferred to a sparse space, different recovery algorithms like basis pursuit, CoSaMP^[1] or fast non-iterative algorithms^[2] can be used to recover the signal.

One of the key principles of dictionary learning is that the dictionary has to be inferred from the input data. The emergence of sparse dictionary learning methods was stimulated by the fact that in signal processing one typically wants to represent the input data using as few components as possible. Before this approach the general practice was to use predefined dictionaries (such as Fourier or wavelet transforms). However, in certain cases a dictionary that is trained to fit the input data can significantly improve the sparsity, which has applications in data decomposition, compression and analysis and has been used in the fields of image denoising and classification, video and audio processing. Sparsity and overcomplete dictionaries have immense applications in image compression, image fusion and inpainting.

Problem statement

Given the input dataset ${\displaystyle X=,x_{i}\in \mathbb {R} ^{d}}$ we wish to find a dictionary ${\displaystyle \mathbf {D} \in \mathbb {R} ^{d\times n}:D=}$ and a representation ${\displaystyle R=,r_{i}\in \mathbb {R} ^{n}}$ such that both ${\displaystyle \|X-\mathbf {D} R\|_{F}^{2}}$ is minimized and the representations ${\displaystyle r_{i}}$ are sparse enough. This can be formulated as the following optimization problem:

${\displaystyle {\underset {\mathbf {D} \in {\mathcal {C}},r_{i}\in \mathbb {R} ^{n}}{\text{argmin}}}\sum _{i=1}^{K}\|x_{i}-\mathbf {D} r_{i}\|_{2}^{2}+\lambda \|r_{i}\|_{0}}$, where ${\displaystyle {\mathcal {C}}\equiv \{\mathbf {D} \in \mathbb {R} ^{d\times n}:\|d_{i}\|_{2}\leq 1\,\,\forall i=1,...,n\}}$, ${\displaystyle \lambda >0}$

${\displaystyle {\mathcal {C}}}$ is required to constrain ${\displaystyle \mathbf {D} }$ so that its atoms would not reach arbitrarily high values allowing for arbitrarily low (but non-zero) values of ${\displaystyle r_{i}}$.${\displaystyle \lambda }$ controls the trade off between the sparsity and the minimization error.

The minimization problem above is not convex because of the ℓ₀-"norm" and solving this problem is NP-hard.^[3] In some cases L¹-norm is known to ensure sparsity^[4] and so the above becomes a convex optimization problem with respect to each of the variables ${\displaystyle \mathbf {D} }$ and ${\displaystyle \mathbf {R} }$ when the other one is fixed, but it is not jointly convex in ${\displaystyle (\mathbf {D} ,\mathbf {R} )}$.

Properties of the dictionary

The dictionary ${\displaystyle \mathbf {D} }$ defined above can be "undercomplete" if ${\displaystyle n<d}$ or "overcomplete" in case ${\displaystyle n>d}$ with the latter being a typical assumption for a sparse dictionary learning problem. The case of a complete dictionary does not provide any improvement from a representational point of view and thus isn't considered.

Undercomplete dictionaries represent the setup in which the actual input data lies in a lower-dimensional space. This case is strongly related to dimensionality reduction and techniques like principal component analysis which require atoms ${\displaystyle d_{1},...,d_{n}}$ to be orthogonal. The choice of these subspaces is crucial for efficient dimensionality reduction, but it is not trivial. And dimensionality reduction based on dictionary representation can be extended to address specific tasks such as data analysis or classification. However, their main downside is limiting the choice of atoms.

Overcomplete dictionaries, however, do not require the atoms to be orthogonal (they will never have a basis anyway) thus allowing for more flexible dictionaries and richer data representations.

An overcomplete dictionary which allows for sparse representation of signal can be a famous transform matrix (wavelets transform, fourier transform) or it can be formulated so that its elements are changed in such a way that it sparsely represents the given signal in a best way. Learned dictionaries are capable of giving sparser solutions as compared to predefined transform matrices.

Algorithms

As the optimization problem described above can be solved as a convex problem with respect to either dictionary or sparse coding while the other one of the two is fixed, most of the algorithms are based on the idea of iteratively updating one and then the other.

The problem of finding an optimal sparse coding ${\displaystyle R}$ with a given dictionary ${\displaystyle \mathbf {D} }$ is known as sparse approximation (or sometimes just sparse coding problem). A number of algorithms have been developed to solve it (such as matching pursuit and LASSO) and are incorporated in the algorithms described below.

Method of optimal directions (MOD)

The method of optimal directions (or MOD) was one of the first methods introduced to tackle the sparse dictionary learning problem.^[5] The core idea of it is to solve the minimization problem subject to the limited number of non-zero components of the representation vector:

${\displaystyle \underset{\mathbf{D},R}{min}\{\Vert <!--\; \Vert \; -->X-<!--\; -\; -->\mathbf{D}R{\Vert <!--\; \Vert \; -->}_F^2\}}$Zdroj:https://en.wikipedia.org?pojem=Sparse_dictionary_learning
Text je dostupný za podmienok Creative Commons Attribution/Share-Alike License 3.0 Unported; prípadne za ďalších podmienok. Podrobnejšie informácie nájdete na stránke Podmienky použitia.