Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

(Optional) Building a Docker Image

Building a Docker image ensures environment consistency, facilitates quick deployment, provides container isolation, and simplifies dependency management. It also offers good portability and flexibility, allowing applications to run efficiently in any environment where Docker is installed.

This section supplements the Milvus installation operations. Based on the openeuler-22.03-lts-sp4:latest image, the Milvus installation method is written into the Dockerfile to build a Milvus Docker image.

  1. Prepare the Docker environment.
    1
    yum install docker
  2. Prepare the base image.

    Download a base Docker image from the corresponding link in Table 2 and import the image.

    • Arm
      1
      docker load -i openEuler-docker.aarch64.tar.xz
    • x86
      1
      docker load -i openEuler-docker.x86_64.tar.xz
  3. Prepare a Dockerfile.
    1. Create and open the Dockerfile.
      1
      vim Dockerfile
    2. Press i to enter the insert mode and copy the following content to the file:
      FROM openeuler-22.03-lts-sp4:latest

ARG TARGETARCH
ARG CACHE_BUST
RUN echo "Cache busting with $CACHE_BUST"
# Obtain the architecture according to the system type.
RUN echo "target arch ${TARGETARCH}"

# Install dependencies, including Python and OpenBLAS.
RUN yum install -y wget g++ gcc gdb libatomic libstdc++-static ninja-build git make zip unzip tar which \
    autoconf automake python3 python3-pip texinfo gfortran numa* libstdc* curl vim \
    pkg-config libuuid-devel libaio perl-IPC-Cmd libasan openblas-devel libomp hdf5 hdf5-devel gtest-devel && \
    yum clean all

# Configure Miniconda3.
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-py312_24.7.1-0-Linux-aarch64.sh -O miniconda.sh
RUN sh miniconda.sh -b && rm miniconda.sh
ENV PATH=/root/miniconda3/bin:$PATH
RUN conda config --set auto_activate_base false

# Configure a pip proxy.
RUN mkdir /root/.pip && \
    echo -e "[global]\nindex-url=https://repo.huaweicloud.com/repository/pypi/simple\n[install]\ntrusted-host=repo.huaweicloud.com\n" >> /root/.pip/pip.conf

# Create a Python virtual environment.
RUN conda create -y --name milvus python=3.11

# Download the Python dependency Conan.
RUN conda init bash && source /root/.bashrc && conda activate milvus && pip install conan==1.61.0 && \
    conan remote add conancenter https://center.conan.io False -f && \
    conan remote add default-conan-local https://milvus01.jfrog.io/artifactory/api/conan/default-conan-local False -f && \
    sed -i '32s/self._verify_ssl = verify/self._verify_ssl = False/' /root/miniconda3/envs/milvus/lib/python3.11/site-packages/conans/client/downloaders/file_downloader.py

# Download and install CMake.
RUN wget -qO- "https://cmake.org/files/v3.27/cmake-3.27.5-linux-`uname -m`.tar.gz" | tar --strip-components=1 -xz -C /usr/local

# Download and install Go.
RUN mkdir /root/go && mkdir /usr/local/go && \
    wget -qO- "https://repo.huaweicloud.com/go/go1.21.10/go1.21.10.linux-${TARGETARCH}.tar.gz" | tar --strip-components=1 -xz -C /usr/local/go
ENV GOROOT=/usr/local/go
ENV GOPATH=/root/go
ENV PATH=$GOROOT/bin:$GOPATH/bin:$PATH
ENV GO111MODULE=on
ENV GOPROXY=https://repo.huaweicloud.com/repository/goproxy/
ENV GOSUMDB=off

# Download and install Rust.
RUN curl https://sh.rustup.rs -sSf | \
    sh -s -- --default-toolchain=1.73 -y
ENV PATH=/root/.cargo/bin:$PATH

# Download and install OpenBLAS.
RUN mkdir OpenBLAS-0.3.10 && \
    wget -qO- https://github.com/xianyi/OpenBLAS/archive/v0.3.10.tar.gz | tar --strip-components=1 -xz -C OpenBLAS-0.3.10
RUN cd OpenBLAS-0.3.10 && make FC=gfortran USE_OPENMP=1 -j && make install && \
    cd .. && rm -rf OpenBLAS-0.3.10
ENV LD_LIBRARY_PATH=/opt/OpenBLAS/lib:$LD_LIBRARY_PATH

# Download and install etcd.
RUN mkdir /usr/local/etcd && \
    wget -qO- "https://github.com/etcd-io/etcd/releases/download/v3.5.0/etcd-v3.5.0-linux-${TARGETARCH}.tar.gz" | tar --strip-components=1 -xz -C /usr/local/etcd
ENV PATH=/usr/local/etcd:$PATH

# Download and install MinIO.
RUN wget -q "https://dl.min.io/server/minio/release/linux-${TARGETARCH}/minio" && chmod +x ./minio && mv ./minio /usr/bin/

# Download Milvus.
RUN git clone https://github.com/milvus-io/milvus.git && \
    cd /root/milvus && git checkout v2.4.5

      1. If the network environment of the physical machine cannot directly access the Internet, add the proxy configuration to the Dockerfile. Specifically, add ENV proxy settings after FROM and clear the proxy information at the end of the file. The following is an example:

      FROM openeuler-22.03-lts-sp4:latest

# Add a proxy to connect to the Internet.
ENV http_proxy="http://proxy service user name:proxy service password@proxy service IP address:proxy service port"
ENV https_proxy=$http_proxy

...

ENV http_proxy=""
ENV https_proxy=""

      2. If an external certificate is required to prevent connection rejection, place the certificate in the same directory as the Dockerfile and copy the certificate to the correct path. The following is an example:

      ...

RUN echo "Cache busting with $CACHE_BUST"
# Obtain the architecture according to the system type.
RUN echo "target arch ${TARGETARCH}"

COPY certificate.crt /etc/pki/ca-trust/source/anchors/
RUN update-ca-trust

...

      3. Cloning Milvus from GitHub can take a significant amount of time, and a failed build would necessitate a complete rebuild. To mitigate this, you can prepare Milvus beforehand, place it in the same directory as the Dockerfile, and then copy it into the image. The following is an example:

      ...

# Download and install MinIO.
RUN wget -q "https://dl.min.io/server/minio/release/linux-${TARGETARCH}/minio" && chmod +x ./minio && mv ./minio /usr/bin/

COPY milvus/ /root/milvus
RUN cd /root/milvus && git checkout v2.4.5

...
  4. Build the image.

    Run the following command to build the image using the Dockerfile.

    • Arm
      1
      docker build --no-cache --build-arg TARGETARCH=arm64 --build-arg CACHE_BUST=$(date +%s) -t openeuler-22.03-lts-sp4-milvus:latest .
    • x86
      1
      docker build --no-cache --build-arg TARGETARCH=amd64 --build-arg CACHE_BUST=$(date +%s) -t openeuler-22.03-lts-sp4-milvus:latest .
  5. Export the image.

    Package the built image into a TAR package on a local drive.

    1
    docker save -o openeuler-22.03-lts-sp4-milvus.tar openeuler-22.03-lts-sp4-milvus:latest
  6. Import the image.
    To use this image, upload it to a server and import the TAR package to a local image repository.
    1
    docker load -i openeuler-22.03-lts-sp4-milvus.tar
  7. Create a container.

    Use the image to run a container instance in the background. Table 1 describes the parameters in the command.

    1
    docker run -d -it --cpus=16 --cpuset-cpus=0-15 --cpuset-mems=0 -m 64g --name milvus-16u64g-01 --network host --privileged=true -v /data/milvus:/data/milvus openeuler-22.03-lts-sp4-milvus /bin/bash
    Table 1 Parameters in the command

    Parameter

    Description

    -d

    Runs the container in the background.

    -it

    Allocates an interactive terminal (-i) and a pseudo-TTY (-t) for terminal login.

    --cpus=16

    Allocates 16 CPU cores to the container.

    --cpuset-cpus=0-15

    Binds specific CPU cores.

    --cpuset-mems=0

    Configures memory binding, which allows the container to use memory exclusively from NUMA node 0.

    -m

    Limits the maximum amount of memory the container can use.

    --name

    Specifies the container name.

    --network host

    Enables the container and host to share the network.

    --privileged=true

    Grants the root permission.

    NOTICE:

    This option performs the following operations: adds all capabilities to the container and allows the container to access all settings of the host. With privileged = true configured, more operations are allowed, such as loading kernel modules and directly operating hardware devices. However, this option introduces security risks, potentially causing container escape and arbitrary operations on the host.

    -v /data/milvus:/data/milvus

    Mounts a directory, which is shared by Docker and the host.

    openeuler-22.03-lts-sp4

    Image name.

    /bin/bash

    Command to be run inside the container when it starts.
  8. Access the container.

    After accessing the container, you can operate within it just as you would on the host.

    1
    docker exec -it milvus-16u64g-01 /bin/bash

Some possible faults can be modified only after Milvus is compiled and the configuration file is generated, for example, Failed to Find GCC Version During Milvus Compilation. The built image does not involve Milvus compilation. After a container is created, you need to enter the container and run the following commands to compile Milvus. If any problem occurs, refer to Troubleshooting.

conda activate milvus
cd /root/milvus
make milvus