JUNE 18–22, 2017
FRANKFURT AM MAIN, GERMANY

Presentation Details
 
Name: (RP13) Enhanced Adaptive Compression in Lustre
 
Time: Tuesday, June 20, 2017
08:35 am - 09:45 am
 
Room:   Substanz 1+2  
 
Breaks:07:30 am - 10:00 am Welcome Coffee
 
Presenter:   Anna Fuchs, University of Hamburg
 
Abstract:  
The poster presents the design of transparent compression in the Lustre filesystem. Compression can increase the throughput and save costs for storage systems. In Lustre it will be based on stripes which are divided into chunks corresponding to the ZFS records. The most important challenges are potential performance losses related to bad read-ahead performance and read-modify-write behavior. ZFS offers solutions for the read-ahead problem when using the tree structure and saving the compressed data without gaps. RMW requires complex strategies, however, its impact can be reduced significantly by using chunks instead of complete stripes. First benchmarks showed that lz4 and zstd are well suited for the approach due to their very high compression speed, although it requires parallel compression to satisfy fast interconnects.

Authors:
Anna Fuchs, Universität Hamburg
Michael Kuhn, Universität Hamburg
Julian Kunkel, DKRZ
Thomas Ludwig, Universität Hamburg / DKRZ

  
 
Download

RP13_Fuchs.pdf (425 KB)