Related Concepts
- In-memory shuffle: During shuffle, data is cached in the memory instead of being directly written to drives. In this way, the drive I/O overhead is reduced to improve the data processing efficiency.
- OCKD process: After OmniShuffle is installed, you can use the OCKD process to start or stop OmniShuffle.
- Remote Shuffle Service (RSS): The shuffle service is deployed on a node outside the Spark cluster and shuffles data remotely.
Parent topic: Feature Overview