Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

AI Tuning

Command Function

Leverages Huawei-developed high-performance AI tuning solution to tune applications in database and big data scenarios. After the analysis, tuning suggestions on parameter configurations in complex scenarios are provided.

  • AI tuning is available on CentOS 7.6, openEuler 20.03, and openEuler 22.03 LTS.
  • For details about how to configure password-free authentication for the MySQL database, see Configuring Password-Free Authentication for MySQL.
  • The drive where the data directory is located must have a sufficient space for AI tuning to back up data.

Syntax

devkit tuner aituner [-h] [-l {0,1,2,3}] [-r n] [-f <file>] [-o <dir>] [-a]

When running the command, enter either of the -f and -a options or both. If you enter the -f option, the default parameters (not customized) in the application configuration file are used for the tuning. If you enter the -a option, you need to edit the advanced parameters of the application.

Parameter Description

Table 1 Parameter description

Parameter

Option

Description

-h/--help

-

Obtains help information.

-l/--log-level

0/1/2/3
Log level, which defaults to 2.
  • 0: DEBUG
  • 1: INFO
  • 2: WARNING
  • 3: ERROR

-r/--round

-

Number of tuning iterations, which defaults to 50 and must be greater than 0. A larger number of iterations requires a longer total time.

-f/--config-file-path

-

Path of the application configuration file. For an example parameter configuration, see /usr/local/devkit/tuner/tools/aituner/config_file.

-o/--output

-

Report folder name. By default, reports are saved in the current directory. The default folder name is in the format of aituner-YMD-HMS. The report folder contains tuning logs, historical tuning data, and optimal parameter sets.

-a/--advanced-param

-

Advanced parameters of the application.

NOTE:
  • The tool provides the default parameter configuration in /usr/local/devkit/tuner/tools/aituner/advanced_param/advanced_param.json (using an RPM installation as an example). Generally, the -a parameter is not required.
  • If you need to customize parameters, use the -a parameter in the command to set advanced parameters. After you manually modify the configuration file, the configuration file is automatically loaded when you run the command.

Application Configuration Description

By default, the application configuration file is stored in the /usr/local/devkit/tuner/tools/aituner/config_file directory when installing the RPM package. Supported databases are MySQL, openGauss, Vastbase, and RocksDB. Supported big data software components are Flink, Hive, and Spark.

  • If you decompress the tool package, the file path is /home/DevKit-CLI-x.x.x-Linux-Kunpeng/tuner/tools/aituner/config_file, where /home is the decompression directory.
  • The JSON file consists of four parts: hsots_list, app, load, and metric.
  • The hosts_list parameter may contain information about multiple nodes. Each pair of braces ({}) indicates a node and they are separated by commas (,).
  • The hosts_list parameter of openGauss and Vastbase requires you to log in as the root user.
Table 2 Common parameters in the JSON file

Level-1 Parameter

Level-2 Parameter

Example Value

Description

hosts_list

name

local

Node name. You can customize it.

ip

127.0.0.1

Node IP address.

username

root

Node OS user.

port

22

Node port.

app

host_name

["local"]

Application node name, which corresponds to the name parameter in hosts_list, indicating that the application is deployed on this node.
Table 3 mysql_sysbench_config.json

Level-1 Parameter

Level-2 Parameter

Level-3 Parameter

Example Value

Description

app

name

-

mysql

Application name.

app_version

-

8.0.28

Application version. Supported MySQL versions: 5.7.22 to 5.7.43, 8.0.11 to 8.0.28, and 8.0.30~8.0.34.

app_configs

mysql_username

root

MySQL user name.

NOTE:

For security purposes, enable password-free authentication for the MySQL user. For details, see Configuring Password-Free Authentication for MySQL.

mysql_port

3306

Port for connecting to the MySQL database.

mysql_exe_path

/usr/bin

Directory of the mysql execution file.

mysqld_exe_path

/usr/sbin

Directory of the mysqld execution file.

sock_path

/data/mysql/run/mysql.sock

Directory of the mysql.sock file.

NOTE:

You can also enter "", and the system automatically obtains the value from the my.cnf file.

config_file_path

/etc/my.cnf

Directory of the MySQL configuration file.

load

name

-

mysql_sysbench

Indicates that MySQL uses Sysbench as the pressure test tool. Only Sysbench is supported.

host_name

-

["local"]

Name of the node on which the pressure test tool is installed.

test_tool_path

-

/home/sysbench-0.5/sysbench/

Pressure test tool directory.

test_tool_version

-

0.5

Pressure test tool version.

test_case_name

-

oltp_mix

Test case name, which is oltp_distinct, oltp_index, oltp_nonindex, oltp_order, oltp_point, oltp_simple, oltp_sum, oltp_delete, or oltp_mix.

test_case_path

-

/home/sysbench-0.5/sysbench/tests/db/

Test case directory.

thread_number

-

50

Number of Sysbench threads used in the pressure test. The value ranges from 1 to 8,192.

database_name

-

sysbench

Name of the database used for the pressure test.

oltp_tables_count

-

10

Number of OLTP tables.

oltp_table_size

-

1000

OLTP table size.

exec_duration

-

600

Pressure test duration.

metric

tune_metric

-

tps

Tuning performance metric. The options are tps and latency.
Table 4 opengauss_benchmarksql_config.json

Level-1 Parameter

Level-2 Parameter

Level-3 Parameter

Example Value

Description

app

name

-

opengauss

Application name.

app_version

-

5.0.1

Application version. Supported openGauss versions are 3.0.0, 3.0.3, 3.0.5, 5.0.0, and 5.0.1.

app_configs

system_username

omm

OS user who uses openGauss.

data_node_path

/data/openGauss/install/data/dn

Data node directory of openGauss.

load

name

-

opengauss_benchmarksql

Indicates that openGauss uses BenchMarkSQL as the pressure test tool. This parameter cannot be modified.

test_tool_version

-

5.0

Pressure test tool version.

host_name

-

["local"]

Name of the node on which the pressure test tool is installed.

config_path

-

/home/benchmarksql-5.0/run/props.openGauss.1000w

Path of the props configuration file.

script_path

-

/home/benchmarksql-5.0/run

Press test script directory.

numa_bind

-

""

Cores bound to the pressure test client, for example, 1,2 or 2-5.

metric

tune_metric

-

tpmC

Tuning performance metric.
Table 5 vastbase_benchmarksql_config.json

Level-1 Parameter

Level-2 Parameter

Level-3 Parameter

Example Value

Description

app

name

-

vastbase

Application name.

app_version

-

G100

Application version. Supported Vastbase version: G100.

app_configs

system_username

vastbase

OS user who uses Vastbase.

data_node_path

/home/vastbase/data/vastbase

Data node directory of Vastbase.

load

name

-

vastbase_benchmarksql

Indicates that Vastbase uses BenchMarkSQL as the pressure test tool. This parameter cannot be modified.

test_tool_version

-

0.5

Pressure test tool version.

host_name

-

["local"]

Name of the node on which the pressure test tool is installed.

config_path

-

/opt/benchmarksql/benchmarksql-5.0/run/props.vastbase.1000w

Path of the props configuration file.

script_path

-

/home/benchmarksql-5.0/run

Press test script directory.

numa_bind

-

""

Cores bound to the pressure test client, for example, 1,2 or 2-5.

metric

tune_metric

-

tpmC

Tuning performance metric.
Table 6 rocksdb_benchmarksql_config.json

Level-1 Parameter

Level-2 Parameter

Level-3 Parameter

Example Value

Description

app

name

-

rocksdb

Application name.

app_version

-

7.9.2

Application version. Supported RocksDB versions are 7.0.4, 7.1.1, 7.1.2, 7.2.0, 7.2.2, 7.3.1, 7.4.3-7.4.5, 7.5.3, 7.6.0, 7.7.2, 7.7.3, 7.7.8, 7.8.3, and 7.9.2.

app_configs

database_dir

/home/database/rocksdb/db1

RocksDB database directory.

load

name

-

rocksdb_dbbench

Indicates that RocksDB uses db_bench as the pressure test tool. This parameter cannot be modified.

test_tool_version

-

7.9.2

Pressure test tool version.

host_name

-

["local"]

Name of the node on which the pressure test tool is installed.

test_tool_path

-

/home/database/rocksdb/tools

Pressure test tool directory.

benchmarks_list

-

[ "fillseq"]

Test case, which is fillseq, fillsync, fillrandom, overwrite, readrandom, newiterator, newiteratorwhilewriting, seekrandom, seekrandomwhilewriting, readseq, readreverse, multireadrandom, mixgraph, readwhilewriting, readwhilescanning, readrandomwriterandom, updaterandom, xorupdaterandom, randomwithverify, fill 100K, crc32c, xxhash, compress, uncompress, acquireload, fillseekseq, randomtransaction, randomreplacekeys, timeseries, or readtocache.

key_size

-

50

db_bench parameter. The values ranges from 1 to 2,147,483,647 (bytes).

value_size

-

830

db_bench parameter. The values ranges from 1 to 2,147,483,647 (bytes).

num

-

100

db_bench parameter. The values ranges from 1 to 9,223,372,036,854,775,807.

thread_number

-

16

db_bench parameter, which specifies the number of threads used for the pressure test. The value ranges from 0 to 2,147,483,647.

cache_size

-

16777216

db_bench parameter. The values ranges from 1 to 9,223,372,036,854,775,807 (bytes).

exec_duration

-

0

Pressure test duration. The value ranges from 0 to 2,147,483,647 (seconds).

metric

tune_metric

-

IOPS

Tuning performance metric.
Table 7 flink_hibench_config.json

Level-1 Parameter

Level-2 Parameter

Level-3 Parameter

Example Value

Description

app

name

-

flink

Application name.

app_version

-

1.12

Application version. Supported Flink versions: 1.12 to 1.15.

app_configs

app_exe_path

/home/hibench/flink/bin

Application execution file directory.

master_type

standalone

Deployment mode, which is yarn or standalone.

startup_params

{}

Application startup parameter.

load

name

-

flink_hibench

Indicates that Flink uses HiBench as the pressure test tool. This parameter cannot be modified.

NOTE:

HiBench does not support pressure tests for Flink 1.15. Use Huawei Cloud HiBench instead.

host_name

-

["local"]

Name of the node on which the pressure test tool is installed.

test_case_name

-

identity

Test case name, which is identity, repartition, or wordcount.

test_tool_version

-

7.0

Pressure test tool version.

exec_duration

-

180

Pressure test duration, in seconds.

test_tool_path

-

/home/hibench/HiBench-7.0

Pressure test tool directory.

hibench_datagen_throughput

-

600K

Throughput of the test case, which is 20K, 40K, 60K, 80K, 100K, 200K, 300K, 400K, 500K, 600K, 700K, 800K, 900K, 1000K, 2000K, 4000K, 6000K, 8000K, or 10000K.

metric

tune_metric

-

latency

Tuning performance metric, which is throughput, latency, or throughput/latency.
Table 8 hive_tpcds_config.json

Level-1 Parameter

Level-2 Parameter

Level-3 Parameter

Example Value

Description

app

name

-

hive

Application name.

app_version

-

3.1.3

Application version. Supported Hive versions: 3.0.0 and 3.1.0 to 3.1.3.

app_configs

app_exe_path

/usr/local/hive/bin

Application execution file directory.

load

name

-

hive_tpcds

Indicates that Hive uses TPC-DS as the pressure test tool. This parameter cannot be modified.

test_tool_version

-

3.0

Pressure test tool version.

host_name

-

["local"]

Name of the node on which the pressure test tool is installed.

tpcds_sql_index_list

-

[1,2]

Test case ID, which ranges from 1 to 99.

database_name

-

tpcds_bin_partitioned_orc_3

Name of the database used for the pressure test.

exec_timeout

-

1000

Pressure test timeout interval, in seconds.

sql_file_tmpdir

-

/tmp

Temporary directory for storing test cases, which will be automatically deleted after the pressure test is complete.

metric

tune_metric

-

latency

Tuning performance metric.
Table 9 spark_tpcds_config.json

Level-1 Parameter

Level-2 Parameter

Level-3 Parameter

Example Value

Description

app

name

-

spark

Application name.

app_version

-

3.3.1

Application version. The supported Spark versions are 2.3.0-2.3.2, 2.4.1-2.4.7, 3.0.0-3.0.3, 3.1.0-3.1.2, 3.2.1, 3.2.2, 3.3.0, and 3.3.1.

app_configs

app_exe_path

/home/bigdata/spark/bin

Application execution file directory.

omni_home_dir

""

OmniOperator directory.

NOTE:

If the value is "", OmniOperator is disabled.

master_type

yarn

Application deployment mode, which is yarn or standalone.

load

name

-

spark_tpcds

Indicates that Spark uses TPC-DS as the pressure test tool. This parameter cannot be modified.

test_tool_version

-

3.0

Pressure test tool version.

host_name

-

["local"]

Name of the node on which the pressure test tool is installed.

tpcds_sql_index_list

-

[1]

Test case ID, which ranges from 1 to 99. Test cases 14, 23, 24, and 39 are of types a and b, for example, 14a and 14b.

database_name

-

tpcds_bin_partitioned_orc_300

Name of the database used for the pressure test.

exec_timeout

-

1000

Pressure test timeout interval, in seconds.

sql_file_tmpdir

-

/tmp

Temporary directory for storing test cases, which will be automatically deleted after the pressure test is complete.

metric

tune_metric

-

latency

Tuning performance metric.

Example

devkit tuner aituner -r 10 -f /usr/local/devkit/tuner/tools/aituner/config_file/mysql_sysbench_config.json -o /home/mysql_20_tuner
  • -r 10 indicates 10 rounds of tuning, -f specifies /usr/local/devkit/tuner/tools/aituner/config_file/mysql_sysbench_config.json as the configuration file, and -o /home/mysql_20_tuner specifies the directory for storing the generated report.
  • The command output contains too much application information, most of which has been omitted for easy reading. For the detailed information, see the actual command output.

Command output:

...
...
...
[MESSAGE]The baseline test is successful. The baseline performance result is 187463.0.
[MESSAGE]-- Tuning rounds: 1. The result of this round: 210.0, the current best tuning result is: 210.0
[MESSAGE]-- Tuning rounds: 2. The result of this round: 214422.0, the current best tuning result is: 214422.0
[MESSAGE]-- Tuning rounds: 3. The result of this round: 230803.0, the current best tuning result is: 230803.0
[MESSAGE]-- Tuning rounds: 4. The result of this round: 177219.0, the current best tuning result is: 230803.0
[MESSAGE]-- Tuning rounds: 5. The result of this round: 208.0, the current best tuning result is: 230803.0
[MESSAGE]-- Tuning rounds: 6. The result of this round: 210.0, the current best tuning result is: 230803.0
[MESSAGE]-- Tuning rounds: 7. The result of this round: 198.0, the current best tuning result is: 230803.0
[MESSAGE]-- Tuning rounds: 8. The result of this round: 207.0, the current best tuning result is: 230803.0
[MESSAGE]-- Tuning rounds: 9. The result of this round: 230353.0, the current best tuning result is: 230803.0
[MESSAGE]-- Tuning rounds: 10. The result of this round: 198912.0, the current best tuning result is: 230803.0
[MESSAGE]After 10 rounds of performance optimization, the optimal performance result is: 230803.0,Performance improved by 23.1%.
...
...
...
Parent topic: System Profiler