Instructions

Interface Definition

Construct a Gemm object. During the construction, the tensor information of the input matrix, weight matrix, and output matrix needs to be passed. The bias matrix is optional. If it is not passed, the second constructor will be called for initialization.

• Gemm(const TensorInfo &srcTensor, const TensorInfo &weiTensor, const TensorInfo &dstTensor, int numThreads = 0)->void
• Gemm(const TensorInfo &srcTensor, const TensorInfo &weiTensor, const TensorInfo &dstTensor,const TensorInfo &biasTensor, int numThreads = 0)->void

To perform operator computation, the memory addresses for storing the input and output must be passed. src, wei, dst, and bias are the memory addresses of the input matrix, weight matrix, output matrix, and bias matrix, respectively. Whether to pass the address of the bias matrix depends on whether the bias is passed during construction. The default values of alpha and beta are 1.0f and 0.0f, respectively.

• Run(const void *src, const void *wei, void *dst, float alpha = 1.0f, float beta = 0.0f, int numThreads = 0)->void
• Run(const void *src, const void *wei, void *dst, void *bias, float alpha = 1.0f, float beta = 0.0f, int numThreads = 0)->void

ValidateInput is used to validate the Gemm input parameters and is automatically triggered during operator construction.

• ValidateInput(const TensorInfo &srcTensor, const TensorInfo &weiTensor, const TensorInfo &dstTensor, int numThreads = 0)->KuDNN::Status
• ValidateInput(const TensorInfo &srcTensor, const TensorInfo &weiTensor, const TensorInfo &dstTensor, const TensorInfo &biasTensor, int numThreads = 0)->KuDNN::Status

Supported Data Types

• The following data type combinations are supported for matrix computation. (The Shape, Type, and Layout parameters need to be passed during the initialization of the TensorInfo object. The following lists the data types supported by Type.)

  Table 4 Type supported during TensorInfo object initialization

  srcTensor	weiTensor	dstTensor	biasTensor
  KuDNN::Element::TypeT::F16(fp16)	KuDNN::Element::TypeT::F16(fp16)	KuDNN::Element::TypeT::F16(fp16)	KuDNN::Element::TypeT::F16(fp16)
  KuDNN::Element::TypeT::F16(fp16)	KuDNN::Element::TypeT::F16(fp16)	KuDNN::Element::TypeT::F32(fp32)	KuDNN::Element::TypeT::F32(fp32)
  KuDNN::Element::TypeT::S8(int8)	KuDNN::Element::TypeT::S8(int8)	KuDNN::Element::TypeT::S32(int32)	KuDNN::Element::TypeT::S32(int32)
  KuDNN::Element::TypeT::F16(fp16)	KuDNN::Element::TypeT::S8(int8)	KuDNN::Element::TypeT::F16(fp16)	KuDNN::Element::TypeT::F32(fp32)
  KuDNN::Element::TypeT::F32(fp32)	KuDNN::Element::TypeT::F32(fp32)	KuDNN::Element::TypeT::F32(fp32)	KuDNN::Element::TypeT::F32(fp32)

• A maximum of 5 dimensions are supported. The supported sequential data layouts include a, ab, abc, abcd, and abcde.

  They correspond to KuDNN::Layout::A, KuDNN::Layout::AB, KuDNN::Layout::ABC, KuDNN::Layout::ABCD, and KuDNN::Layout::ABCDE, respectively.

Examples

Matrix multiplication with two-dimensional data of the FP16 type:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

// The layout is AB. M, N, and K, corresponds to 5, 5, and 5, respectively.
using SizeType = KuDNN::SizeType;
using Shape = KuDNN::Shape;
using Type KuDNN::Element::TypeT;
Shape srcShape(5, 5);
Shape weiShape(5, 5);
Shape dstShape(5, 5);
Shape biasShape(5, 5);
// Tensor initialization
const KuDNN::TensorInfo srcTensor = {srcShape, Type::F16, KuDNN::Layout::AB};
const KuDNN::TensorInfo weiTensor = {weiShape, Type::F16, KuDNN::Layout::AB};
const KuDNN::TensorInfo dstTensor = {dstShape, Type::F16, KuDNN::Layout::AB};
const KuDNN::TensorInfo biasTensor = {biasShape, Type::F16, KuDNN::Layout::AB};
SizeType srcSize = 5 * 5;
SizeType dstSize = 5 * 5;
SizeType weiSize = 5 * 5;
SizeType biasSize =5 * 5;
// Allocate memory for storing the input arguments and result.
void * src = malloc(srcSize * Type::F16.GetSize());
void * wei = malloc(weiSize * Type::F16.GetSize());
void * dst = malloc(dstSize * Type::F16.GetSize());
void * ref = malloc(dstSize * Type::F16.GetSize());
// Construct the operator.
KuDNN::Gemm gemmLayer(srcTensor,
weiTensor, dstTensor, numThreads);
// Run the operator.
gemmLayer.Run(src, wei, dst, 1, 0, numThreads);