softmax

Softmax is a common activation function used in multi-classification problems. It converts a set of arbitrary real numbers into a probability distribution whose output values range from 0 to 1, and the sum of all output values is 1.

The main features are as follows:

• Normalized output: The softmax function normalizes the input to ensure that the output is a valid probability distribution. Even if the input is any real number, the output sum of the softmax function is still 1. It is commonly used at the output layer of multi-classification problems.
• Non-linear: The softmax function is a non-linear function. It can perform non-linear transformation on the input to increase the representation capability of the model, thereby better fitting complex data patterns.
• Translation invariance: The softmax function performs translation invariance on the input. That is, when each element in the input vector adds (or subtracts) the same constant, the softmax output is not changed.

In a neural network, the softmax function is usually used at the output layer to convert the original output of the neural network into a vector representing class probabilities. During training, the difference between the softmax output and the actual label can be used as a loss function. Through backword propagation, network parameters are updated to minimize the loss and improve model performance.

>>> import numpy as np
>>> from libkdnn_ext import softmax
>>> x = np.random.rand(1, 5).astype(np.float32)
>>> softmax(x)
array([[0.19810137, 0.21171768, 0.16419397, 0.24222486, 0.1837621 ]], dtype=float32)