Accessibility navigation

Data compression for climate data

Kuhn, M., Kunkel, J. M. and Ludwig, T. (2016) Data compression for climate data. Supercomputing Frontiers and Innovations, 3 (1). pp. 75-94. ISSN 2313-8734

Text (Open access) - Published Version
· Available under License Creative Commons Attribution Non-commercial.
· Please see our End User Agreement before downloading.


It is advisable to refer to the publisher's version if you intend to cite from this work. See Guidance on citing.

To link to this item DOI: 10.14529/jsfi160105


The different rates of increase for computational power and storage capabilities of supercomputers turn data storage into a technical and economical problem. Because storage capabilities are lagging behind, investments and operational costs for storage systems have increased to keep up with the supercomputers' I/O requirements. One promising approach is to reduce the amount of data that is stored. In this paper, we take a look at the impact of compression on performance and costs of high performance systems. To this end, we analyze the applicability of compression on all layers of the I/O stack, that is, main memory, network and storage. Based on the Mistral system of the German Climate Computing Center (Deutsches Klimarechenzentrum, DKRZ), we illustrate potential performance improvements and cost savings. Making use of compression on a large scale can decrease investments and operational costs by 50% without negatively impacting performance. Additionally, we present ongoing work for supporting enhanced adaptive compression in the parallel distributed file system Lustre and application-specific compression.

Item Type:Article
Divisions:Science > School of Mathematical, Physical and Computational Sciences > Department of Computer Science
ID Code:77684
Publisher:South Urals University


Downloads per month over past year

University Staff: Request a correction | Centaur Editors: Update this record

Page navigation