Examples

This section uses the sift-128-euclidean.hdf5 dataset, Faiss-supported algorithm (HNSW), and 32 threads as an example.

Obtaining Datasets and Test Programs

Obtain a dataset.

1	wget http://ann-benchmarks.com/sift-128-euclidean.hdf5 --no-check-certificate

Obtain a test program. The branch is v1.3.0. Assume that the directory where the program runs is /path/to/krl-faiss-test. The full directory structure is as follows:

├── configs                                                   // Stores configuration files for the algorithm and dataset.
      └── hnsw
            └── hnsw_sift-128-euclidean.config 
├── include                                                   // Stores the header file of the test framework.
├── src                                                       // Stores source files of the test framework.
├── Makefile                                                  // Script for compiling the program
└── test.sh                                                 // Script for running the test
├── data                                                      // Stores the dataset
      └── sift-128-euclidean.hdf5
├── indexes
      └── hnsw                                                // Stores manually generated index.
└── sift.faiss // Index generated by running the hnsw_test executable (with the dataset configuration set to save_or_load = save).
└── hnsw_test // Executable file generated after compilation

Example

Assume that the program runs in the /path/to/krl-faiss-test directory. Save the dataset to the data folder in this directory.
If this is the first execution, check that save_or_load in the hnsw_sift-128-euclidean.config file is set to save. For subsequent executions, change the value to load to use the prebuilt graph index or retriever for querying.

Install the dependencies.

yum install hdf5 hdf5-devel numactl numactl-devel

Check that KRL has been installed. Modify FAISSROOT in the Makefile file based on the actual Faiss installation path in Enabling KRL in Faiss.
Build the executable file.
1 2
export KRL_PATH=/usr/local/sra_krl/lib make hnsw_test
KRL_PATH can be set based on the actual location of the KRL dynamic library file. After the RPM is installed, the library is stored in /usr/local/sra_krl/lib by default.

KRL operators can be enabled to accelerate the Faiss-supported algorithms such as HNSW, PQFS, IVFPQ, and IVFPQFS. When testing, be sure to select the appropriate compilation instruction for each algorithm.
- HNSW: make hnsw
- PQFS: make pqfs
- IVFPQ: make ivfpq
- IVFPQFS: make ivfpqfs

Run the executable file. Add the OpenBLAS and Faiss dynamic library paths to the environment variable.

export LD_LIBRARY_PATH=/opt/OpenBLAS/lib/:/path/to/faiss/install/lib64:$LD_LIBRARY_PATH
numactl -C 0-31 -m 0 ./hnsw_test hnsw sift-128-euclidean

The command output is as follows:

Parent topic: KRL Interfaces