TEE Device Plugin for Kubernetes

To make Kubernetes sense and manage TEE resources, TrustZone provides a TEE device plugin so that Kubernetes can schedule CA and TA containers to proper nodes.

The plugin can sense and manage only memory resources in the TEE. pods share the TEE memory using the number of virtual devices.
To run the plugin, the TEE must use the confidential OS. For details about how to upgrade the TEE OS to the confidential OS, see (Optional) Upgrading the TEE OS.
You can configure the minimum memory allocation granularity and the health check period. The minimum memory allocation granularity is 512 KB by default, and can be greater than or 512 KB. The default health check period is 60 seconds, and the value 0 indicates no health checks. You can set them by setting the -precision xx and -internal xx parameters when starting the plugin.
You can set a proper memory size for each TA container based on the minimum memory allocation granularity and the heap memory size in the TA resource configuration file configs.xml. If the allocated memory is too large, the memory may be wasted. If it is too small, the TA may exit due to insufficient memory.
When applying for a TA developer certificate for the plugin, you are advised to set the TA to a resident TA, that is, set instanceKeepAlive to true.
TAs must be compiled on an Arm server.
For details about how to set up the Kubernetes environment on openEuler, see "Configuring the Deployment Environment" in Kubernetes 1.18.20 Deployment Guide.

Install the compilation environment. The Go version must be 1.15 or later.

yum install make gcc
wget https://go.dev/dl/go1.15.15.linux-arm64.tar.gz
tar xzvf go1.15.15.linux-arm64.tar.gz
export PATH=$PWD/go/bin/:$PATH
go version

Obtain the device plugin source code and the iTrustee SDK for compiling the TA in the plugin. The itrustee_sdk and kunpengsecl files are in the same directory.
```
git clone -b tee-device-plugin https://gitee.com/openeuler/kunpengsecl.git
git clone https://gitee.com/openeuler/itrustee_sdk.git
```
Configure TA compilation resources.
1. Obtain the TA certificate files (config and private_key.pem) by following instructions in Applying for a TA Developer Certificate in a Debugging Environment and save them to the corresponding directories in kunpengsecl/tee-device-plugin/cmd/TA.
```
mkdir -p kunpengsecl/tee-device-plugin/cmd/TA/TA_cert
mkdir -p kunpengsecl/tee-device-plugin/cmd/TA/signed_config
```
  The following is the directory structure for the two certificate files:
2. Modify the manifest.txt file.
  1. Open manifest.txt.
```
vim kunpengsecl/tee-device-plugin/cmd/TA/manifest.txt
```
  2. Press i to enter the insert mode. Set the parameters values based on the configs.xml file used to apply for the TA certificate. For details, see Table 1.
  3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
3. Modify the CA code.
  1. Open tee_mem_ca.c.
```
vim kunpengsecl/tee-device-plugin/cmd/CA/libteememca/tee_mem_ca.c
```
  2. Press i to enter the insert mode and change the CA UUID to the actual value.
  3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
Compile the device plugin.
```
cd kunpengsecl/tee-device-plugin
make
```
After the compilation is complete, the following files are generated in the pkg directory:
To deploy the plugin directory on the host machine, run the following commands:
```
cp pkg/libteememca.so /usr/lib64/
cp pkg/tee-device-plugin /vendor/bin
cp pkg/*.sec /data/
/vendor/bin/tee-device-plugin
```
Build a Docker image based on the device plugin application.
1. Copy the device plugin application and dependency files to the docker directory. The dependency files can be obtained from a server that supports TrustZone.
  
  You need to copy the files from the system that is consistent as the basic image. Otherwise, the application or library may fail to be used due to system differences.
```
cp pkg/* docker/
cp /usr/bin/tlogcat docker/
cp /usr/lib64/libteec.so docker/
cp /usr/lib64/libboundscheck.so docker/
```
2. Build an image.
```
cd docker
docker build -t tee-device-plugin .
```
Upload the image to the image repository so that compute nodes can pull the image from the repository. You can change the repository name to the name of your private repository.
```
docker tag tee-device-plugin:latest <repository>/tee-device-plugin:latest
docker push <repository>/tee-device-plugin:latest
```
Label the servers that support TrustZone so that the plugin in DaemonSet mode can be automatically deployed onto these nodes.
```
kubectl label nodes <node-name> teetype=trustzone
```
Deploy the plugin in DaemonSet mode.
1. Replace the example image repository name with the actual one.
```
cd ../deploy
vim tee-device-plugin.yml
```
2. Deploy the plugin.
```
kubectl create -f tee-device-plugin.yml
kubectl get pods -A -o wide
```
  Till now, the plugin has been successfully deployed on a compute node that supports TrustZone.
Build a CA and TA image and create a pod configuration file. Specifically, build the image by following instructions in Container Deployment, upload it to the image repository, and edit the YML file based on test-pod.yml.
```
vim test-pod.yml
```
Deploy the pods. After the deployment is successful, you can see that the pods occupy the configured number of resources on the target nodes.
```
kubectl create -f test-pod.yml
kubectl describe node <node-name>
```
```
kubectl get pods -A -o wide
```
View the plugin and TA logs.
1. View the plugin logs.
```
kubectl logs -f k8s-tee-device-plugin-ds-<xxxxx> -n kube-system
```
  In the preceding command, xxxxx is a random character string and can be changed to the actual pod name of the plugin.
2. View the TA logs.
```
kubectl logs -f test-pod-ta
```

Parent topic: Confidential Containers