Instructions

Interface Definition

Initialize GroupNormalizationLayerFWD. During construction, the tensor information of the input matrix, scaling matrix, group size, and output matrix, as well as the normalization flag, needs to be passed.

GroupNormalizationLayerFWD(const TensorInfo &srcInfo, const TensorInfo &scaleShiftInfo, SizeType groupSize, const TensorInfo &dstInfo, NormalizationFlags flags)->void

Table 1 Input parameters of GroupNormalizationLayerFWD

Parameter	Data Type	Description	Value Range
srcInfo	KuDNN::TensorInfo	Input matrix information.	{shape{A, ... , D}, type, layout}
scaleShiftInfo	KuDNN::TensorInfo	Scaling and shift information.	{shape{D}, type, layout}
groupSize	KuDNN::SizeType	Group size.	Unsigned integer; number of groups in the second dimension.
dstInfo	KuDNN::TensorInfo	Output matrix information.	{shape{A, ... , D}, type, layout}
flags	KuDNN::NormalizationFlags	Enumeration of normalization modes.	The default value is NONE. The options are as follows: USE_GLOBAL_STATS: uses the input mean and var for calculation. USE_SCALE: scales the normalized result. USE_SHIFT: shifts the normalized result.

Assume that the input data shape is {A, B, C}. Group normalization, scaling, and shifting are performed along the B dimension.

Run the operator. src and dst are the input and output pointers. scale and shift are the scaling factor and shift value. mean and variance are the pointers to the mean and variance. saveStats indicates whether to save the calculated mean and variance. eps is used to avoid the division-by-zero error.

ValidateInput is used to validate the GroupNormalizationLayerFWD input parameters and is automatically triggered during operator construction.

Supported Data Types

The Shape, Type, and Layout parameters need to be passed during the initialization of the TensorInfo object. The following lists the data types supported by Type.

• GroupNormalizationLayerFWD supports the FP16 and FP32 data types. The following table lists the types that can be used during tensor initialization.

  Table 4 Type supported during TensorInfo object initialization

  srcInfo	scaleShiftInfo	dstInfo
  KuDNN::Element::Type::F16(fp16)	KuDNN::Element::Type::F16(fp16)	KuDNN::Element::Type::F16(fp16)
  KuDNN::Element::Type::F32(float)	KuDNN::Element::Type::F32(float)	KuDNN::Element::Type::F32(float)

• Tensors with up to three to five dimensions are supported. The supported data layouts are abc, abcd, and abcde,

  corresponding to KuDNN::Layout::ABC, KuDNN::Layout::ABCD, and KuDNN::Layout::ABCDE.

  Table 5 Layout supported during TensorInfo object initialization

  Dimension	srcInfo Data Layout	dstInfo Data Layout
  3D	abc	abc
  4D	abcd	abcd
  5D	abcde	abcde

Examples

Perform group normalization using FP16. Data layout configurations are: KuDNN::Layout::ABCD for srcInfo data, KuDNN::Layout::A for scaleShiftInfo data, and KuDNN::Layout::ABCD for dstInfo data.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

using SizeType = KuDNN::SizeType;
using Shape = KuDNN::Shape;
using Type KuDNN::Element::TypeT
Shape shape(4, 128, 18, 320);
SizeType groupInfo = 2;
// Define tensor information.
TensorInfo srcInfo = {shape, Type::F16, KuDNN::Layout::ABCD};
TensorInfo scaleShiftInfo = {{shape[1]}, Type::F16, KuDNN::Layout::A};
TensorInfo dstInfo = {shape, Type::F16, KuDNN::Layout::ABCD};
KuDNN::NormalizationFlags flags = KuDNN::NormalizationFlags::NONE;
// Construct the operator: KuDNN::GroupNormalizationLayerFWD gnormLayer(srcInfo, scaleShiftInfo, groupInfo, dstInfo,
flags);
// Initialize matrix data.
SizeType srcSize = srcInfo.GetTotalTensorSize();
SizeType dstSize = dstInfo.GetTotalTensorSize();
SizeType statSize = srcInfo.GetDims()[0] * groupInfo;
SizeType scaleSize = scaleShiftInfo.GetTotalTensorSize();
__fp16 *src = (__fp16*)malloc(srcSize * sizeof(__fp16));
__fp16 *dst = (__fp16*)malloc(srcSize * sizeof(__fp16));
__fp16 *dstRef = (__fp16*)malloc(srcSize * sizeof(__fp16));
float *mean = (float *)malloc(statSize * sizeof(float));
float *variance = (float *)malloc(statSize * sizeof(float));
__fp16 *scale = (__fp16*)malloc(scaleSize * sizeof(__fp16));
__fp16 *shift = (__fp16*)malloc(scaleSize * sizeof(__fp16));
float eps = 1e-5;
// Run the operator.
gnormLayer.Run(src, dst, scale, shift, mean, variance, true, eps);