Discriminative models, also referred to as conditional models, are a class of models frequently used for classification. They are typically used to assign labels, such as pass/fail, win/lose, alive/dead or healthy/sick, to existing datapoints.

Types of discriminative models include logistic regression (LR), conditional random fields (CRFs), decision trees among many others. Typical generative model approaches include naive Bayes classifiers, Gaussian mixture models, variational autoencoders, generative adversarial networks and others.

Definition

Unlike generative modelling, which studies the joint probability ${\displaystyle P(x,y)}$, discriminative modeling studies the ${\displaystyle P(y|x)}$ or maps the given unobserved variable (target) ${\displaystyle x}$ to a class label ${\displaystyle y}$ dependent on the observed variables (training samples). For example, in object recognition, ${\displaystyle x}$ is likely to be a vector of raw pixels (or features extracted from the raw pixels of the image). Within a probabilistic framework, this is done by modeling the conditional probability distribution ${\displaystyle P(y|x)}$, which can be used for predicting ${\displaystyle y}$ from ${\displaystyle x}$. Note that there is still distinction between the conditional model and the discriminative model, though more often they are simply categorised as discriminative model.

Pure discriminative model vs. conditional model

A conditional model models the conditional probability distribution, while the traditional discriminative model aims to optimize on mapping the input around the most similar trained samples.^[1]

Typical discriminative modelling approaches

The following approach is based on the assumption that it is given the training data-set ${\displaystyle D=\{(x_{i};y_{i})|i\leq N\in \mathbb {Z} \}}$, where ${\displaystyle y_{i}}$is the corresponding output for the input ${\displaystyle x_{i}}$.^[2]

Linear classifier

We intend to use the function ${\displaystyle f(x)}$to simulate the behavior of what we observed from the training data-set by the linear classifier method. Using the joint feature vector ${\displaystyle \phi (x,y)}$, the decision function is defined as:

${\displaystyle f(x;w)=\arg \max _{y}w^{T}\phi (x,y)}$

According to Memisevic's interpretation,^[2] ${\displaystyle w^{T}\phi (x,y)}$, which is also ${\displaystyle c(x,y;w)}$, computes a score which measures the compatibility of the input ${\displaystyle x}$ with the potential output ${\displaystyle y}$. Then the ${\displaystyle \arg \max }$ determines the class with the highest score.

Logistic regression (LR)

Since the 0-1 loss function is a commonly used one in the decision theory, the conditional probability distribution ${\displaystyle P(y|x;w)}$, where ${\displaystyle w}$ is a parameter vector for optimizing the training data, could be reconsidered as following for the logistics regression model:

${\displaystyle P(y|x;w)={\frac {1}{Z(x;w)}}\exp(w^{T}\phi (x,y))}$, with

${\displaystyle Z(x;w)=\textstyle \sum _{y}\displaystyle \exp(w^{T}\phi (x,y))}$

The equation above represents logistic regression. Notice that a major distinction between models is their way of introducing posterior probability. Posterior probability is inferred from the parametric model. We then can maximize the parameter by following equation:

${\displaystyle L(w)=\textstyle \sum _{i}\displaystyle \log p(y^{i}|x^{i};w)}$

It could also be replaced by the log-loss equation below:

${\displaystyle l^{\log }(x^{i},y^{i},c(x^{i};w))=-\log p(y^{i}|x^{i};w)=\log Z(x^{i};w)-w^{T}\phi (x^{i},y^{i})}$

Since the log-loss is differentiable, a gradient-based method can be used to optimize the model. A global optimum is guaranteed because the objective function is convex. The gradient of log likelihood is represented by:

${\displaystyle {\frac {\partial L(w)}{\partial w}}=\textstyle \sum _{i}\displaystyle \phi (x^{i},y^{i})-E_{p(y|x^{i};w)}\phi (x^{i},y)}$

where ${\displaystyle E_{p(y|x^{i};w)}}$is the expectation of $$Zdroj:https://en.wikipedia.org?pojem=Discriminative_model
Text je dostupný za podmienok Creative Commons Attribution/Share-Alike License 3.0 Unported; prípadne za ďalších podmienok. Podrobnejšie informácie nájdete na stránke Podmienky použitia.