JUNE 18–22, 2017

Presentation Details

Name: (RP13) Enhanced Adaptive Compression in Lustre
Time: Tuesday, June 20, 2017
08:35 am - 09:45 am
Room:   Substanz 1+2  
Breaks:07:30 am - 10:00 am Welcome Coffee
Presenter:   Anna Fuchs, University of Hamburg
The poster presents the design of transparent compression in the Lustre filesystem. Compression can increase the throughput and save costs for storage systems. In Lustre it will be based on stripes which are divided into chunks corresponding to the ZFS records. The most important challenges are potential performance losses related to bad read-ahead performance and read-modify-write behavior. ZFS offers solutions for the read-ahead problem when using the tree structure and saving the compressed data without gaps. RMW requires complex strategies, however, its impact can be reduced significantly by using chunks instead of complete stripes. First benchmarks showed that lz4 and zstd are well suited for the approach due to their very high compression speed, although it requires parallel compression to satisfy fast interconnects.

Anna Fuchs, Universität Hamburg
Michael Kuhn, Universität Hamburg
Julian Kunkel, DKRZ
Thomas Ludwig, Universität Hamburg / DKRZ


RP13_Fuchs.pdf (425 KB)