Confidential Containers

Container Deployment

To enable confidential containers, you need to upgrade the TEE OS to the confidential OS. For upgrade details, see Java and Python Support. After the upgrade is complete, perform the following steps to deploy containers.

Install Docker on the server.
```
yum install docker
```
Prepare a CA and a TA.
You can develop your own CA and TA by following instructions in Setting Up the TA and CA Development Environment. This demo uses helloworld provided in iTrustee SDK as an example.
1. To obtain the source code, download itrustee_sdk and libboundscheck from the Gitee official website and move libboundscheck to thirdparty/open_source/.
```
git clone https://gitee.com/openeuler/itrustee_sdk.git
git clone https://gitee.com/openeuler/libboundscheck.git
mv libboundscheck thirdparty/open_source/
```
2. Prepare TA compilation resources.
  Obtain the config and private_key.pem files by following instructions in Applying for a TA Developer Certificate in a Debugging Environment and move the two files to signed_config/ and TA_cert/, respectively.
```
cd itrutee_sdk/build/signtools/
mkdir signed_config
mkdir TA_cert
cp /path/to/config ./signed_config/    
cp /path/to/private_key.pem ./TA_cert/ 
```
3. Compile the CA of the helloworld demo.
  1. Open ca_demo.c.
```
cd ../../test/CA/helloworld/
vim ca_demo.c
```
  2. Press i to enter the insert mode. Modify the UUID of the TA and add the TA path.
  3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
  4. Generate the demo_hello application (CA).
```
make
```
4. Compile the TA of the helloworld demo.
  1. Open manifest.txt.
```
cd ../../TA/helloworld/
vim manifest.txt
```
  2. Press i to enter the insert mode. Modify the corresponding variables in manifest.txt based on the content of the configs.xml file used to apply for the config file.
  3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
  4. Generate the < UUID >.sec file (for the TA).
```
make
```

Download an openEuler image and import it.

wget https://mirrors.huaweicloud.com/openeuler/openEuler-22.03-LTS-SP1/docker_img/aarch64/openEuler-docker.aarch64.tar.xz
docker load -i openEuler-docker.aarch64.tar.xz

Copy the CA and TA and the dependency files to a directory for building images.
```
mkdir build && cd build
cp ../itrustee_sdk/test/CA/helloworld/demo_hello ./
cp ../itrustee_sdk/test/TA/helloworld/*.sec ./
cp /usr/lib64/libboundscheck.so ./
cp /usr/lib64/libteec.so ./
cp /usr/bin/tlogcat ./
cp /usr/bin/agentd ./
```
The image build directory contains the following files:
You can run the ldd command to view the dependency files. For example, check the dependencies of agentd:
```
ldd agentd
```
The basic image of openEuler contains all the SO libraries except libboundscheck.so and libteec.so. Therefore, in this example, only the two SO libraries are copied to the image. For other OS images, copy all the missing libraries.

Create a Dockerfile.

Create a file.
```
vim Dockerfile
```

Press i to enter the insert mode and add the following content:

FROM openeuler-22.03-lts-sp1

RUN mkdir -p /vendor/bin && mkdir -p /data

COPY *.so /usr/lib64/
COPY tlogcat agentd /usr/bin/
COPY *.sec /data/
COPY demo_hello /vendor/bin/

RUN chmod +x /vendor/bin/demo_hello

CMD /usr/bin/agentd & /vendor/bin/demo_hello

Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.

Run the following command to build the image:
```
docker build -t demo_hello .
```

Start the container based on the image and obtain the run logs of the container.

docker run -it --mount type=bind,source=/var/itrustee/teecd,target=/var/itrustee/teecd,readonly --device /dev/teelog:/dev/teelog:wr --device /dev/tc_ns_cvm:/dev/tc_ns_cvm:wr -d demo_hello
docker logs -f <container_id>

If the container requires TA2TA or trusted storage, start the agentd in the container before executing the CA.
To enable support for high-level languages in a container, copy the tee_teleport tool to the container. The methods of using high-level languages in a container are the same as those of deploying and running high-level languages on the host.
If a high-level language is used, the JVM and Python interpreter cannot be installed in the container. They can be installed and deployed only on the host.

TEE Device Plugin for Kubernetes

To make Kubernetes sense and manage TEE resources, TrustZone provides a TEE device plugin so that Kubernetes can schedule CA and TA containers to proper nodes.

The plugin can sense and manage only memory resources in the TEE. Pods share the TEE memory using the number of virtual devices.
To run the plugin, the TEE must use the confidential OS. For details about how to upgrade the TEE OS to the confidential OS, see Java and Python Support.
You can configure the minimum memory allocation granularity and the health check period. The minimum memory allocation granularity is 512 KB by default, and can be greater than or equal to 512 KB. The default health check period is 60 seconds, and the value 0 indicates no health checks. You can set them by setting the -precision xx and -internal xx parameters when starting the plugin.
You can set a proper memory size for each TA container based on the minimum memory allocation granularity and the heap memory size in the TA resource configuration file configs.xml. If the allocated memory is too large, the memory may be wasted. If it is too small, the TA may exit due to insufficient memory.
When applying for a TA developer certificate for the plugin, you are advised to set the TA to a resident TA, that is, set instanceKeepAlive to true.
TAs must be compiled on an ARM server.
For details about how to set up the Kubernetes environment on openEuler, see "Configuring the Deployment Environment" in Kubernetes 1.18.20 Deployment Guide.

Install the compilation environment. The Go version must be 1.15 or later.

yum install make gcc
wget https://go.dev/dl/go1.15.15.linux-arm64.tar.gz
tar xzvf go1.15.15.linux-arm64.tar.gz
export PATH=$PWD/go/bin/:$PATH
go version

Obtain the device plugin source code and the iTrustee SDK for compiling the TA in the plugin . The itrustee_sdk and kunpengsecl files are in the same directory.
```
git clone -b tee-device-plugin https://gitee.com/openeuler/kunpengsecl.git
git clone https://gitee.com/openeuler/itrustee_sdk.git
```
Configure TA compilation resources.
1. Obtain the TA certificate files (config and private_key.pem) by following instructions in Applying for a TA Developer Certificate in a Debugging Environment and save them to the corresponding directories in kunpengsecl/tee-device-plugin/cmd/TA.
```
mkdir -p kunpengsecl/tee-device-plugin/cmd/TA/TA_cert
mkdir -p kunpengsecl/tee-device-plugin/cmd/TA/signed_config
```
  The following is the directory structure for the two certificate files:
2. Modify the manifest.txt file.
  1. Open manifest.txt.
```
vim kunpengsecl/tee-device-plugin/cmd/TA/manifest.txt
```
  2. Press i to enter the insert mode. Set the parameters values based on the configs.xml file used to apply for the TA certificate. For details, see Configuring the manifest.txt File.
  3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
3. Modify the CA code.
  1. Open tee_mem_ca.c.
```
vim kunpengsecl/tee-device-plugin/cmd/CA/libteememca/tee_mem_ca.c
```
  2. Press i to enter the insert mode and change the CA UUID to the actual value.
  3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
Compile the device plugin.
```
cd kunpengsecl/tee-device-plugin
make
```
After the compilation is complete, the following files are generated in the pkg directory:
To deploy the plugin directory on the host machine, run the following commands:
```
cp pkg/libteememca.so /usr/lib64/
cp pkg/tee-device-plugin /vendor/bin
cp pkg/*.sec /data/
/vendor/bin/tee-device-plugin
```
Build a Docker image based on the device plugin application.
1. Copy the device plugin application and dependency files to the docker directory. The dependency files can be obtained from a server that supports TrustZone. Note that you need to copy the files from the system that is consistent as the basic image. Otherwise, the program or library may fail to be used due to system differences.
```
cp pkg/* docker/
cp /usr/bin/tlogcat docker/
cp /usr/lib64/libteec.so docker/
cp /usr/lib64/libboundscheck.so docker/
```
2. Build an image.
```
cd docker
docker build -t tee-device-plugin .
```
Upload the image to the image repository so that compute nodes can pull the image from the repository. You can change the repository name to the name of your private repository.
```
docker tag tee-device-plugin:latest <repository>/tee-device-plugin:latest
docker push <repository>/tee-device-plugin:latest
```
Label the servers that support TrustZone so that the plugin in DaemonSet mode can be automatically deployed onto these nodes.
```
kubectl label nodes <node-name> teetype=trustzone
```
Deploy the plugin in DaemonSet mode.
1. Replace the example image repository name with the actual one.
```
cd ../deploy
vim tee-device-plugin.yml
```
2. Deploy the plugin.
```
kubectl create -f tee-device-plugin.yml
kubectl get pods -A -o wide
```
  Till now, the plugin has been successfully deployed on a compute node that supports TrustZone.
Build a CA and TA image and create a pod configuration file. Specifically, build the image by following instructions in Container Deployment, upload it to the image repository, and edit the YML file based on test-pod.yml.
```
vim test-pod.yml
```
Deploy the pods. After the deployment is successful, you can see that the pods occupy the configured number of resources on the target nodes.
```
kubectl create -f test-pod.yml
kubectl describe node <node-name>
```
```
kubectl get pods -A -o wide
```
View the plugin and TA logs.
1. View the plugin logs.
```
kubectl logs -f k8s-tee-device-plugin-ds-<xxxxx> -n kube-system
```
  In the preceding command, xxxxx is a random character string and can be changed to the actual pod name of the plugin.
2. View the TA logs.
```
kubectl logs -f test-pod-ta
```

If the container fails to be started, perform the following steps:

Ensure that teecd and tzdriver have been loaded on the host.
Ensure that the /var/itrustee/teecd/teecd.sock, /dev/teelog, and /dev/tc_ns_cvm files exist on the host.
Ensure that libteec.so and tlogcat used to build the image are the same as those on the host.
Ensure that the mounted files used to start the container are the same as those in 7.
If the container requires TA2TA or trusted storage, start the agentd daemon in the container.

Parent topic: Feature Usage