Compilation and Installation

The backend implementation of the TensorFlow Serving ANNC graph fusion optimization feature has been merged into the official TensorFlow and TF Serving open source repositories. Code related to the frontend graph fusion, XLA graph fusion, and operator optimization are available in the ANNC open source repository hosted on Gitee, which can be cloned and compiled using git.

Obtaining Code

Disable SSL verification for Git.

git config --global http.sslVerify false
git config --global https.sslVerify false

Obtain the TensorFlow and TF Serving code.

git clone --branch v2.15.0-2509 https://gitcode.com/boostkit/tensorflow.git
git clone --branch v2.15.1-2509 https://gitcode.com/boostkit/tensorflow-serving.git

Obtain ANNC code.

git clone --branch v0.0.2 https://gitee.com/openeuler/ANNC.git

Performing compilation and installation

Install GCC 12.3.1.

yum install -y gcc-toolset-12-gcc*
export PATH=/opt/openEuler/gcc-toolset-12/root/usr/bin/:$PATH
export LD_LIBRARY_PATH=/opt/openEuler/gcc-toolset-12/root/usr/lib64/:$LD_LIBRARY_PATH

Compile and install ANNC.

export ANNC=/path/to/ANNC
cd $ANNC
source build.sh
cp bazel-bin/annc/service/cpu/libannc.so /usr/lib64/
cp $ANNC/annc/service/cpu/xla/libs/XNNPACK/build/libXNNPACK.so /usr/lib64
mkdir -p /usr/include/annc 
cp annc/service/cpu/kdnn_rewriter.h /usr/include/annc/ 
cp annc/service/cpu/annc_flags.h /usr/include/annc/
cd python
python3 setup.py bdist_wheel
python3 -m pip install dist/*.whl --force-reinstall

Enable the patch.

export TF_PATH=/path/to/tensorflow
export XLA_PATH=/path/to/tensorflow/third_party/xla
cd $ANNC/install/tfserver/xla
bash xla2.sh

Figure 1 shows an example of successful patching.

Figure 1 Successful patching

Go to the tensorflow-serving directory.
1
cd /path/to/tensorflow-serving/

Create a directory for storing compiling dependencies.

export DISTDIR=$(pwd)/download
mkdir -p $DISTDIR

Set the BAZEL_PATH to the directory that contains the Bazel executable file.
1
export BAZEL_PATH=/path/to/bazel
Run the build script to compile the code.
1
sh compile_serving.sh --tensorflow_dir /path/to/tensorflow --features gcc12,annc
/path/to/tensorflow specifies the TensorFlow path.

The build result is the TF Serving binary file tensorflow_model_server, and the file path is /path/to/tensorflow-serving/bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server.
In the preceding command, gcc12 indicates that GCC 12.3.1 is used for compilation. The compile command in the build script is as follows:
```
bazel --output_user_root=$BAZEL_COMPILE_CACHE build -c opt --distdir=$DISTDIR --override_repository=org_tensorflow=$TENSORFLOW_DIR \
--copt=-march=armv8.3-a+crc --copt=-O3 --copt=-fprefetch-loop-arrays --copt=-Wno-error=maybe-uninitialized  \ 
--copt=-Werror=stringop-overflow=0 --config=fused_embedding  \ 
--define tflite_with_xnnpack=false tensorflow_serving/model_servers:tensorflow_model_server
```
The following describes some parameters:
- --output_user_root: Bazel compilation cache directory. The default value is /path/to/tensorflow-serving/output. You can use the BAZEL_COMPILE_CACHE environment variable to set a custom path. The command is as follows:
  1
  export BAZEL_COMPILE_CACHE=/path/to/your/cache_dir
- --distdir: directory for storing TF Serving compilation dependencies. It ensures reliable access when third-party package download fails.
- --override_repository: specifies the local TensorFlow build and uses the directory specified by tensorflow_dir as the local TensorFlow.
Common build errors and solutions:
- If the message "unable to find valid certification path to requested target" is displayed, the certificate verification fails. For details about the solution, see Failed to Verify the Certificate When Compiling TensorFlow 2.13.0 Source Code in the TensorFlow Porting Guide.
- If the message "Error in download_and_extract" is displayed, rectify the fault by referring to Failed to Download the TF-Serving Source Code Dependency in the TensorFlow Serving Porting Guide. The directory specified by --distdir is /path/to/tensorflow-serving/download.
- If the system displays a message indicating that the org_boost dependency library fails to be pulled, see Failed to Obtain the Dependency of the org_boost Sub-repository in the TensorFlow Serving Porting Guide.
- If the system displays a message indicating that no Golang website certificate is available, rectify the fault by referring to No Golang Website Certificate in the TensorFlow Serving Porting Guide.
- If the golang download fails during ANNC compilation, rectify the fault by referring to Failed to Download the TF-Serving Source Code Dependency in the TensorFlow Serving Porting Guide. The --distdir directory in the solution can be specified in the build.sh script.
- If the system displays a message indicating that an error occurs during upb.c compilation, rectify the fault by referring to Syntax Error Reported During upb.c Compilation in the TensorFlow Serving Porting Guide.

Parent topic: Feature Guide