Installing Flink

Deploy and configure Flink on physical machines to run in multiple Docker containers, allowing Flink jobs to be submitted and executed.

1. Go to the /usr/local directory on the physical machine and download the Flink software package.

   1 2	cd /usr/local wget --no-check-certificate https://archive.apache.org/dist/flink/flink-1.16.3/flink-1.16.3-bin-scala_2.12.tgz

   

   flink-1.16.3-bin-scala_2.12.tgz is an example of the Flink software package name. If you are using a different Flink version, replace the example with the actual package name.

2. Extract the flink-1.16.3-bin-scala_2.12.tgz package in the /usr/local path and create a soft link for later version replacement.

   1 2 3

   tar -zxvf flink-1.16.3-bin-scala_2.12.tgz chown -R root:root flink-1.16.3 ln -s flink-1.16.3 flink

   

   flink-1.16.3 is an example of the Flink software directory. If you are using a different Flink version, replace the example with the actual directory.

3. Configure the masters and workers files of Flink.
   1. Open the /usr/local/flink/conf/masters file.

      1	vi /usr/local/flink/conf/masters

   2. Press i to enter the insert mode and change the content of masters to the flink_jm_8c32g container ID 8081. For example:

      1	4a376b30106b:8081

   3. Press Esc, type :wq!, and press Enter to save the file and exit.
   4. Open the /usr/local/flink/conf/workers file.

      1	vi /usr/local/flink/conf/workers

   5. Press i to enter the insert mode, and change the content of workers to the IDs of the flink_tm1_8c32g and flink_tm2_8c32g containers. For example, configure four TaskManagers for each of two Docker containers.

      1 2 3 4 5 6 7 8

      c3ddf10d0353 c3ddf10d0353 c3ddf10d0353 c3ddf10d0353 c3bbdbcc1ae1 c3bbdbcc1ae1 c3bbdbcc1ae1 c3bbdbcc1ae1

   6. Press Esc, type :wq!, and press Enter to save the file and exit.
   7. Open the /usr/local/flink/conf/flink-conf.yaml file.

      1	vi /usr/local/flink/conf/flink-conf.yaml

   8. Press i to enter the insert mode, replace the configuration with the following, and change the value of jobmanager.rpc.address to the ID of the flink_jm_8c32g container ID. It is recommended that the total number of slots be greater than the degree of parallelism. For example:

      taskmanager.memory.process.size: 8G
      jobmanager.rpc.address: 4a376b30106b
      jobmanager.rpc.port: 6123
      jobmanager.memory.process.size: 8G
      taskmanager.memory.process.size: 8G
      taskmanager.numberOfTaskSlots: 2
      parallelism.default: 16
      rest.port: 8081
      rest.bind-port: 8081
      io.tmp.dirs: /tmp
      #pipeline.operator-chaining: true
      #execution:runtime-mode: STREAMING
      #jobmanager.execution.failover-strategy: region
      #table.optimizer.agg-phase-strategy: ONE_PHASE
      #pipeline.object-reuse: true
      #==============================================================================
      # JVM
      #==============================================================================
      # JVM options for GC
      #env.java.opts: -verbose:gc -XX:NewRatio=3 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:ParallelGCThreads=4

   9. Press Esc, type :wq!, and press Enter to save the file and exit.