Instructions

Interface Definition

Initialize NormalizationLayerFWD. During construction, the tensor information of the input matrix, scaling matrix, and output matrix needs to be passed. statsinfo describes the mean and variance.

Run the operator. src and dst are the input and output pointers. scale and shift are pointers to the scaling factor and shift value. mean and variance are the pointers to the mean and variance. saveStats indicates whether to save the calculated mean and variance. eps is used to avoid the division-by-zero error.

Run(const void *src, void *dst, const void *scale, const void *shift, float *mean, float *variance, bool saveStats, const float eps)->void

Table 2 Input parameters of Run

Parameter	Data Type	Description	Value Range
src	void*	Input pointer	-
dst	void*	Output pointer	-
scale	void*	Scaling pointer	-
shift	void*	Shift pointer	-
mean	float*	Mean pointer	-
variance	float*	Variance pointer	-
saveStats	bool	Whether to save the calculated variance	Boolean
eps	float	Used to avoid the division-by-zero error	Floating-point number

Assume that the input data shape is {A, B, C, D}. Normalization, scaling, and shifting operations are performed in the last dimension, that is, dimension D.

ValidateInput is used to validate the NormalizationLayerFWD input parameters and is automatically triggered during operator construction.

Supported Data Types

• NormalizationLayerFWD supports the FP16 and FP32 data types. (The Shape, Type, and Layout parameters need to be passed during the initialization of the TensorInfo object. The following lists the data types supported by Type.)

  Table 4 Type supported during TensorInfo object initialization

  srcInfo	statInfo	scaleShiftInfo	dstInfo
  KuDNN::Element::Type::F16(fp16)	KuDNN::Element::Type::F16(fp16)	KuDNN::Element::Type::F16(fp16)	KuDNN::Element::Type::F16(fp16)
  KuDNN::Element::Type::F32(float)	KuDNN::Element::Type::F32(float)	KuDNN::Element::Type::F32(float)	KuDNN::Element::Type::F32(float)

• A maximum of 5 dimensions are supported. The supported sequential data layouts include a, ab, abc, abcd, and abcde.

  They correspond to KuDNN::Layout::A, KuDNN::Layout::AB, KuDNN::Layout::ABC, KuDNN::Layout::ABCD, and KuDNN::Layout::ABCDE, respectively.

  Table 5 Layout supported during TensorInfo object initialization

  Dimension	srcInfo Data Layout	dstInfo Data Layout
  2D	ab	ab
  3D	abc	abc
  4D	abcd	abcd
  5D	abcde	abcde

Examples

Perform layer normalization using Float. Data layout configurations are: KuDNN::Layout::ABCDE for srcInfo data, KuDNN::Layout::ABCD for statInfo data, KuDNN::Layout::A for scaleShiftInfo data, and KuDNN::Layout::ABCDE for dstInfo data.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

using SizeType = KuDNN::SizeType;
using Shape = KuDNN::Shape;
using Type KuDNN::Element::TypeT
Shape shape(5, 5, 5, 5, 5);
// Define tensor information.
TensorInfo srcInfo = {shape, TypeF32, KuDNN::Layout::ABCDE};
TensorInfo statInfo = {{shape[0], shape[1], shape[2], shape[3]}, TypeF32, KuDNN::Layout::ABCD};
TensorInfo scaleShiftInfo = {{shape[4]}, TypeF32, KuDNN::Layout::A};
TensorInfo dstInfo = {shape, TypeF32, KuDNN::Layout::ABCDE};
KuDNN::NormalizationFlags flags = KuDNN::NormalizationFlags::NONE;
// Construct the operator: KuDNN::NormalizationLayerFWD lnormLayer(srcInfo, statInfo, scaleShiftInfo, dstInfo,
flags);
// Initialize matrix data.
SizeType srcSize = srcInfo.GetTotalTensorSize();
SizeType dstSize = dstInfo.GetTotalTensorSize();
SizeType statSize = statInfo.GetTotalTensorSize();
SizeType innerSize = scaleShiftInfo.GetTotalTensorSize();
float *src = (float *)malloc(srcSize * sizeof(float));
float *dst = (float *)malloc(dstSize * sizeof(float));
float *dstRef = (float *)malloc(dstSize * sizeof(float));
float *mean = (float *)malloc(statSize * sizeof(float));
float *variance = (float *)malloc(statSize * sizeof(float));
float *scale = (float *)malloc(innerSize * sizeof(float));
float *shift = (float *)malloc(innerSize * sizeof(float));
float eps = 1e-5;
// Run the operator.
lnormLayer.Run(src, dst, scale, shift, mean, variance, true, eps);