Verifying Compatibility

This section describes how to verify the compatibility of the open source Faiss with the Kunpeng platform. This section uses the sift-128-euclidean.hdf5 dataset, Faiss-supported algorithm (HNSW), and 32 threads as an example.

Obtaining the Dataset and Test Program

Obtain a dataset.

1	wget http://ann-benchmarks.com/sift-128-euclidean.hdf5 --no-check-certificate

Obtain a test program. The branch is v1.4.0. Assume that the program runs at the directory /path/to/sra_test. The full directory structure is as follows:

├── configs                                                   // Stores configuration files for the algorithm and dataset.
      └── hnsw
            └── hnsw_sift-128-euclidean.config 
├── include                                                   // Stores the header file of the test framework.
├── src                                                       // Stores source files of the test framework.
├── Makefile                                                  // Script for compiling the program.
└── test.sh                                                 // Script for running the test.
├── data                                                      // Stores the dataset.
      └── sift-128-euclidean.hdf5
├── indexes
      └── hnsw                                                // Stores manually generated index.
            └── sift.faiss                                   // Index generated by running the hnsw_test executable (with save_or_load set to save in the dataset configuration file).
└── hnsw_test                                                // Executable file generated after compilation.

Open Source Faiss Compatibility Verification

Download the Faiss source code from the GitHub repository using the v1.8.0 tag. Save the file to a path accessible to a compiler, such as /path/to/faiss-1.8.0.
```
git clone --branch v1.8.0 --single-branch https://github.com/facebookresearch/faiss.git
```
Assume that the program runs at the /path/to/sra_test directory. Save the dataset to the data folder in this directory.
If this is the first execution, check that save_or_load in the hnsw_sift-128-euclidean.config file is set to save. For subsequent executions, change the value to load to use the prebuilt graph index or retriever for querying.

Install the dependencies.

yum install hdf5 hdf5-devel numactl numactl-devel

Obtain libfaiss.so by following the instructions provided in Compiling Faiss. Modify FAISSROOT in the Makefile file based on the actual Faiss installation path.
Build the executable file.
1 2
export LD_LIBRARY_PATH=/opt/OpenBLAS/lib/:/path/to/faiss/install/lib64:$LD_LIBRARY_PATH make hnsw_test
During the test, select the appropriate compilation instruction for each algorithm.
- HNSW: make hnsw_test
- PQFS: make pqfs_test
- IVFPQ: make ivfpq_test
- IVFPQFS: make ivfpqfs_test
- IVFFLAT: make ivfflat_test
Run the executable file. Add the OpenBLAS and Faiss dynamic library paths to the environment variable.
1
numactl -C 0-31 -m 0 ./hnsw_test hnsw sift-128-euclidean
The following figure shows the execution result.

Parent topic: Faiss Description