Running and Verifying OpenFOAM

Use PuTTY to log in to the server as the root user.
Run the following commands on PuTTY to load environment variables:
```
export PATH=/path/to/OPENMPI/bin:$PATH
export LD_LIBRARY_PATH=/path/to/OPENMPI/lib:$LD_LIBRARY_PATH
```
This step declares the MPI environment variables. If the MPI environment variables have been declared, skip this step.
Run the following command to create the hostfile file and add node information:
```
echo -e 'node1\nnode2\n…\nnodex' > /path/to/HOSTFILE
```
- In the preceding command, node1, node2, ..., and nodex indicate the host names of the nodes. You can run the hostname command to query the host names. \n indicates a newline character.
- This step is mandatory when multiple nodes are running and can be skipped when only one node is running.

Run the following commands to load the OpenFOAM application:

Node_list=$(cat /path/to/HOSTFILE | xargs | tr ' ' ',')
clush -w ${Node_list} "cp /root/.bashrc /root/.bashrc_bak;echo source /path/to/OPENFOAM/OpenFOAM-v1906/etc/bashrc >> /root/.bashrc"

Run the following command to copy the case file to any location in the computing environment:

cp -r /path/to/OPENFOAM/OpenFOAM-v1906/tutorials/incompressible/pisoFoam/LES/motorBike/motorBike ./

Run the following command to modify the RunFunctions file:
1. Open RunFunctions.
  vi /path/to/OPENFOAM/OpenFOAM-v1906/bin/tools/RunFunctions
2. Press i to go to the insert mode.
  - Multi-node:
    Add --allow-run-as-root -x PATH -x LD_LIBRARY_PATH --hostfile /path/to/HOSTFILE to the end of the mpirun command, save the modification, and exit.
    
    Before the modification:
```
$mpirun -n $nProcs $appRun $appArgs "$@" </dev/null >> $logFile 2>&1
```
    After the modification:
```
$mpirun --allow-run-as-root -x PATH -x LD_LIBRARY_PATH --hostfile /path/to/HOSTFILE -n $nProcs $appRun $appArgs "$@" </dev/null >> $logFile 2>&1
```
  - Single-node:
    Add --allow-run-as-root to the end of the mpirun command.
    
    Before the modification:
```
$mpirun -n $nProcs $appRun $appArgs "$@" </dev/null >> $logFile 2>&1
```
    After the modification:
```
$mpirun  --allow-run-as-root -n $nProcs $appRun $appArgs "$@" </dev/null >> $logFile 2>&1
```
3. Press Esc, type :wq!, and press Enter to save the changes and exit.

Run the following command to modify the configuration file:

sed -ri "s/\(20 8 8\)/\(64 12 8\)/" motorBike/system/blockMeshDict

Two-node scenario:

sed -ri "s/8/256/" motorBike/ system/{decomposeParDict,decomposeParDict.hierarchical,decomposeParDict.ptscotch}
sed -ri "s/\(4 2 1\)/\(32 4 2\)/"  motorBike/system/{decomposeParDict,decomposeParDict.hierarchical,decomposeParDict.ptscotch}

Single-node scenario:

sed -ri "s/8/128/" motorBike/ system/{decomposeParDict,decomposeParDict.hierarchical,decomposeParDict.ptscotch}
sed -ri "s/\(4 2 1\)/\(16 4 2\)/"  motorBike/system/{decomposeParDict,decomposeParDict.hierarchical,decomposeParDict.ptscotch}

If there are two nodes and 128 tasks are concurrently executed on each node, then 128 x 2 = 256, 32 x 4 x 2 = 256.
If there is only one node and 128 tasks are concurrently executed on the node, then 128 x 1 = 128, 16 x 4 x 2 = 128.
You can run the lscpu |grep ^CPU$s$ command to view the number of concurrent tasks (128) on the single node. You can also change the number based on the actual configuration of the node.

Run the following commands to go to the test case folder and execute the calculation script:

cd motorBike
./Allrun

During the computing, the following information is displayed:

Running blockMesh on /hpcstore/ysf/openfoam-test/1020_motorBike
Running decomposePar on /hpcstore/ysf/openfoam-test/1020_motorBike
Running snappyHexMesh (256 processes) on /hpcstore/ysf/openfoam-test/1020_motorBike
Restore 0/ from 0.orig/ for processor directories
Running renumberMesh (256 processes) on /hpcstore/ysf/openfoam-test/1020_motorBike
Running potentialFoam (256 processes) on /hpcstore/ysf/openfoam-test/1020_motorBike
Running checkMesh (256 processes) on /hpcstore/ysf/openfoam-test/1020_motorBike
Running simpleFoam (256 processes) on /hpcstore/ysf/openfoam-test/1020_motorBike

After the calculation is complete and the system exits properly, a calculation log file is generated. You can run the following command to view the log file:

vi log.simpleFoam

forceCoeffs forceCoeffs1 execute: 
    Coefficients 
        Cd       : 0.413353     (pressure: 0.398578     viscous: 0.0147753) 
        Cs       : 0.0100002    (pressure: 0.00999416   viscous: 6.01357e-06) 
        Cl       : 0.0683827    (pressure: 0.0683764    viscous: 6.2544e-06) 
        CmRoll       : -0.0101679       (pressure: -0.0102318   viscous: 6.39234e-05) 
        CmPitch       : 0.15315 (pressure: 0.146948     viscous: 0.00620151) 
        CmYaw       : 0.0118663 (pressure: 0.0120074    viscous: -0.000141087) 
        Cd(f)    : 0.196509 
        Cd(r)    : 0.216844 
        Cs(f)    : 0.0168664 
        Cs(r)    : -0.00686619 
        Cl(f)    : 0.187341 
        Cl(r)    : -0.118959 
ensightWrite ensightWrite write: ( k omega p U ) 
End 
  
Finalising parallel run

Parent topic: OpenFOAM 1906 Porting Guide (CentOS 7.6)