---
title: 使能机密容器
description: "1. 使用virtCCA_sdk的一键部署脚本完成containerd和k8s部署。"
url: https://www.hikunpeng.com/document/detail/zh/kunpengcctrustzone/ccakit/fg-tz/kunpengtee_16_0076.html
sourcePath: /source/zh/kunpengcctrustzone/ccakit/fg-tz/kunpengtee_16_0076.html
indexId: c52bc32f9d21741a4fa38ced41a853caadc355aa91c94a5b128893dbd56099b266
---
# 使能机密容器

1. 使用virtCCA_sdk的一键部署脚本完成containerd和k8s部署。

```
git clone https://gitcode.com/openeuler/virtCCA_sdk.git
cd virtCCA_sdk/kata-v3.15.0/tools && chmod 755 build.sh
./build.sh containerd && ./build.sh k8s
```


  使用kubectl get pod -A命令确保集群pod状态正常。

2. 去掉master污点并添加worker标签，允许在当前节点调度。

```
kubectl taint nodes <node-name> node-role.kubernetes.io/control-plane:NoSchedule-
kubectl label node <node-name> node.kubernetes.io/worker=
```


3. 下载并安装k8s包管理工具helm。

```
wget https://get.helm.sh/helm-v4.1.4-linux-arm64.tar.gz
tar -zxvf helm-v4.1.4-linux-arm64.tar.gz
cp linux-arm64/helm /usr/local/bin/
```


  helm version回显有版本打印即说明安装成功。

4. 安装社区kata-deploy的helm chart。

```
export CHART="oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy"
export VERSION=3.30.0
helm install kata-deploy --set shims.disableAll=true --set shims.qemu.enabled=true --set shims.qemu-cca.enabled=true  --set debug=true "${CHART}" --version "${VERSION}"
```


  验证kata-deploy安装成功：

  查看社区kata-deploy安装指导，请参见此篇(https://kata-containers.github.io/kata-containers/installation/)。 卸载kata-deploy命令：helm uninstall kata-deploy

5. containerd配置文件中新增runtime kata-qemu-cca相关配置。

  a. runtime kata-qemu前新增runtime kata-qemu-cca相关配置。

```
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-cca]
runtime_type = "io.containerd.kata-qemu.v2"
runtime_path = "/opt/kata/bin/containerd-shim-kata-v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
container_annotations = ["io.kubernetes.container.terminationMessage*"]
snapshotter = "nydus-for-kata-tee"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-cca.options]
ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-qemu-cca.toml"
```


  b. containerd默认snapshotter修改为nydus-for-kata-tee。
  c. 重启containerd使配置生效：systemctl restart containerd
6. 修改runtime kata-qemu-cca的配置文件，firmware相关配置不涉及，因此需要置空，打开guest pull镜像拉取模式。

```
sudo sed -i 's#experimental_force_guest_pull = false#experimental_force_guest_pull = true#; s#firmware = "/opt/kata/share/aavmf/AAVMF_CODE.fd"#firmware = ""#; s#firmware_volume = "/opt/kata/share/aavmf/AAVMF_VARS.fd"#firmware_volume = ""#' /opt/kata/share/defaults/kata-containers/configuration-qemu-cca.toml
```


7. 编译并部署Kunpeng CCA使用的kata shim。

```
git clone https://github.com/kata-containers/kata-containers.git -b 3.30.0
cd kata-containers
```


  - 若OpenEuler/kernel宏定义与linaro/kernel不一致，执行以下命令。

```
sed -i 's/KVM_ARM_CAP_RME_ID = 240/KVM_ARM_CAP_RME_ID = 300/' src/runtime/virtcontainers/hypervisor_linux_arm64.go
```


  - 容器化编译参考以下命令。

```
docker exec -it coco-build-env /bin/bash
make -C src/runtime
```


  - 拷贝到目标路径。

```
cp src/runtime/containerd-shim-kata-v2 /opt/kata/bin/containerd-shim-kata-v2
```


    参考社区文档部署远程证明组件(https://gitcode.com/openeuler/CCA/blob/master/sdk/coco/docs/部署远程证明组件.md)步骤1完成容器编译环境构建。


8. 编译Kunpeng CCA使用的guest kernel Image。

  a. 固定tag：6.6.0-147.0.0

```
git clone https://gitcode.com/openeuler/kernel.git -b 6.6.0-147.0.0 --depth=1
```


  b. 准备启动kata的编译选项。

```
cd ./kernel
make openeuler_defconfig
opts="VSOCKETS VSOCKETS_LOOPBACK VIRTIO_VSOCKETS VIRTIO_VSOCKETS_COMMON NET_9P NET_9P_FD NET_9P_VIRTIO FAILOVER VIRTIO_BLK SCSI_VIRTIO VIRTIO_NET NET_FAILOVER VIRTIO_CONSOLE VIRTIO VIRTIO_PCI_LIB VIRTIO_PCI_LIB_LEGACY VIRTIO_PCI VIRTIO_BALLOON VIRTIO_MMIO EXT4_FS JBD2 FS_MBCACHE FUSE_FS VIRTIO_FS NETFS_SUPPORT 9P_FS CONFIG_TSM_REPORTS CONFIG_ARM_CCA_GUEST CONFIG_OVERLAY_FS"
for opt in $opts; do ./scripts/config -e $opt; done
```


  c. 编译kernel。

```
make -j `nproc`
```


  d. 软链到当前编译生成的Image。

```
sudo ln -s ./arch/arm64/boot/Image /opt/kata/share/kata-containers/vmlinuz-confidential.container
```


9. 使用Kunpeng CCA的qemu二进制，可先参考Yum源安装qemu完成Kunpeng CCA的qemu安装。

```
# kata-qemu-cca配置文件中指定的qemu路径软链到实际的CCA qemu二进制路径
ln -s /usr/bin/qemu-kvm /opt/kata/bin/qemu-system-aarch64-cca-experimental
```


10. 文件系统配置本地镜像仓根证书和域名解析以支持guest pull。

```
losetup -fP --show /opt/kata/share/kata-containers/kata-containers-confidential.img
mount /dev/loop0p1 /mnt  //loop0p1为参考值，具体loop号以实际部署为准。
cat rootCA.crt >> /mnt/etc/ssl/certs/ca-certificates.crt
echo "<IP> registry.hw.com" >> /mnt/etc/hosts
umount /mnt
losetup -d /dev/loop0
```


  rootCA.crt请替换成实际本地镜像仓根证书文件路径；IP和registry.hw.com请分别替换成本地镜像仓IP和域名。

11. 启动Kunpeng CCA机密容器，容器配置文件需要在kernel_params注解中添加rodata=full配置。
  test-cca.yaml示例配置如下：

```
apiVersion: v1
kind: Pod
metadata:
name: test-kata-qemu-cca
annotations:
io.containerd.cri.runtime-handler: "kata-qemu-cca"
io.katacontainers.config.hypervisor.kernel_params: "agent.debug_console agent.log=debug rodata=full"
spec:
runtimeClassName: kata-qemu-cca
terminationGracePeriodSeconds: 5
containers:
- name: box-1
image: registry.hw.com:5000/busybox:latest
imagePullPolicy: Always
command:
- sh
tty: true
```


  启动容器：

```
kubectl apply -f test-cca.yaml
```


12. 配置使用远程证明（可选）。
  参考社区文档部署远程证明组件(https://gitcode.com/openeuler/CCA/blob/master/sdk/coco/docs/部署远程证明组件.md)完成远程证明组件编译部署。

13. 启动如下使能远程证明的机密容器（启动命令同上），远程证明验证通过后容器启动成功（可选）。
  test-remote-attestation.yaml示例配置如下：

```
apiVersion: v1
kind: Pod
metadata:
name: cca
annotations:
io.containerd.cri.runtime-handler: "kata-qemu-cca"
io.katacontainers.config.hypervisor.kernel_params: "agent.debug_console agent.log=debug rodata=full agent.image_policy_file=kbs:///default/security-policy/test agent.enable_signature_verification=true agent.guest_components_rest_api=all agent.aa_kbc_params=cc_kbc::http:90.91.185.34:8080"
spec:
runtimeClassName: kata-qemu-cca
terminationGracePeriodSeconds: 5
containers:
- name: box-1
image: registry.hw.com:5000/busybox:latest
imagePullPolicy: Always
command:
- sh
tty: true
```