TEE Device Plugin for Kubernetes

To make Kubernetes sense and manage TEE resources, TrustZone provides a TEE device plugin so that Kubernetes can schedule CA and TA containers to proper nodes.

• The plugin reports only TEE-side memory resources. The plugin enables multiple Pods to share TEE memory by exposing virtual device replicas.
• To run the plugin, the TEE must use the CCOS. For details about how to upgrade the TEE OS to the CCOS, see Upgrading the TEE OS.
• You can configure the minimum memory allocation granularity and the health check period. The minimum memory allocation granularity is 512 KB by default, and can be greater than or equal to 512 KB. The default health check period is 60 seconds, and the value 0 indicates no health checks. You can set them by setting the -precision xx and -interval xx parameters when starting the plugin.
• You can set a proper memory size for each TA container based on the minimum memory allocation granularity and the heap memory size in the TA resource configuration file configs.xml. Setting this value too high leads to memory waste, while setting it too low may cause the TA application to crash or abnormally exit due to insufficient memory.
• When applying for a TA developer certificate for the plugin, you are advised to set the TA to a resident TA, that is, set instanceKeepAlive to true.
• TAs must be compiled on an Arm server.
• For details about how to set up the Kubernetes environment on openEuler, see "Configuring the Deployment Environment" in Kubernetes 1.18.20 Deployment Guide.

1. Install the compilation environment. The Go version must be 1.15 or later.

   1 2 3 4 5

   yum install make gcc wget https://go.dev/dl/go1.15.15.linux-arm64.tar.gz tar xzvf go1.15.15.linux-arm64.tar.gz export PATH=$PWD/go/bin/:$PATH go version

   

2. Obtain the device plugin source code and the iTrustee SDK for compiling the TA in the plugin. itrustee_sdk and kunpengsecl are under the same directory.

   git clone -b tee-device-plugin https://gitcode.com/openeuler/kunpengsecl.git
   git clone https://gitcode.com/openeuler/itrustee_sdk.git

3. Configure TA compilation resources.
   1. Obtain the TA certificate files (config and private_key.pem) by following instructions in Applying for a TA Developer Certificate in a Debugging Environment and save them to the corresponding directories in kunpengsecl/tee-device-plugin/cmd/TA.

      1 2	mkdir -p kunpengsecl/tee-device-plugin/cmd/TA/TA_cert mkdir -p kunpengsecl/tee-device-plugin/cmd/TA/signed_config

      The following is the directory structure for the two certificate files:

      

   2. Modify the manifest.txt file.
      1. Open manifest.txt.

         1	vim kunpengsecl/tee-device-plugin/cmd/TA/manifest.txt

      2. Press i to enter the insert mode. Set the parameters values based on the configs.xml file used to apply for the TA certificate. For details, see Table 1.

         

      3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
   3. Modify the CA code.
      1. Open tee_mem_ca.c.

         1	vim kunpengsecl/tee-device-plugin/cmd/CA/libteememca/tee_mem_ca.c

      2. Press i to enter the insert mode and change the TA UUID to the actual value.

         

      3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
4. Compile the device plugin.

   1 2	cd kunpengsecl/tee-device-plugin make

   After the compilation is complete, the following files are generated in the pkg directory:

   

   To deploy and start the plugin directly on the host, run the following commands

   1 2 3 4

   cp pkg/libteememca.so /usr/lib64/ cp pkg/tee-device-plugin /vendor/bin cp pkg/*.sec /data/ /vendor/bin/tee-device-plugin

5. Build a Docker image based on the device plugin application.
   1. Copy the device plugin application and dependency files to the docker directory. The dependency files can be obtained from a server that supports TrustZone.
      

      To prevent the application or library from becoming unavailable due to system differences, you are advised to copy the application and library from a system that matches the base image.

      1 2 3 4

      cp pkg/* docker/ cp /usr/bin/tlogcat docker/ cp /usr/lib64/libteec.so docker/ cp /usr/lib64/libboundscheck.so docker/

      

   2. Build an image.

      1 2	cd docker docker build -t tee-device-plugin .

      

6. Upload the image to the image repository so that compute nodes can pull the image from the repository. You can change the repository name to the name of your private repository.

   1 2	docker tag tee-device-plugin:latest <repository>/tee-device-plugin:latest docker push <repository>/tee-device-plugin:latest

   

7. Label the servers that support TrustZone so that the plugin in DaemonSet mode can be automatically deployed onto these nodes.

   1	kubectl label nodes <node-name> teetype=trustzone

8. Deploy the plugin in DaemonSet mode.
   1. Replace the image repository name with the actual one.

      1 2	cd ../deploy vim tee-device-plugin.yml

      

   2. Deploy the plugin.

      1 2	kubectl create -f tee-device-plugin.yml kubectl get pods -A -o wide

      

      Till now, the plugin has been successfully deployed on a compute node that supports TrustZone.

9. Build a CA and TA image and create a pod configuration file. Specifically, build the image by following instructions in Container Deployment, upload it to the image repository, and edit the YML file based on test-pod.yml.

   1	vim test-pod.yml

   

10. Deploy the pods. After the deployment is successful, you can see that the pods occupy the configured number of resources on the target nodes.

    1 2	kubectl create -f test-pod.yml kubectl describe node <node-name>

    

    1	kubectl get pods -A -o wide

    

11. View the plugin and TA logs.
    1. View the plugin logs.

       1	kubectl logs -f k8s-tee-device-plugin-ds-<xxxxx> -n kube-system

       

       In the preceding command, xxxxx is a random character string and can be changed to the actual pod name of the plugin.

    2. View the TA logs.

       1	kubectl logs -f test-pod-ta