Accessibility navigation

Investigating read performance of Python and NetCDF when using HPC parallel filesystems

Jones, M., Blower, J., Lawrence, B. and Osprey, A. (2016) Investigating read performance of Python and NetCDF when using HPC parallel filesystems. In: Taufer, M., Mohr, B. and Kunkel, J. M. (eds.) High Performance Computing. Lecture Notes in Computer Science, 9945. Springer International Publishing, Cham, Switzerland, pp. 153-168. ISBN 9783319460789

Full text not archived in this repository.

It is advisable to refer to the publisher's version if you intend to cite from this work. See Guidance on citing.

To link to this item DOI: 10.1007/978-3-319-46079-6_12


New methods need to be developed to handle the increasing size of data sets in atmospheric science - traditional analysis scripts often inefficiently read and process the data. NetCDF4 is a common file format used in atmospheric and ocean sciences, and Python is widely used in atmospheric and ocean science data analysis. The aim of this work is to provide insight into which read patterns and sizes are most effective when using the netCDF4-python library. Quantitative information on these would be useful information for scientists, library developers, and data managers. Three different read patterns were compared to simulate different types of reads: sequential, strided, and random, with each tested across three file systems - Panasas, Lustre, and GPFS. Read rate and standard deviation were measured using Python and C, reading from plain binary files and NetCDF4 files. Read performance for netCDF4-python was compared with the performance of native Python, the C NetCDF library, and the C Posix library. As expected, comparison between the different read modes shows that access pattern and read size significantly affect achieved performance. The results also show read performance profiles that are similar for the C, C NetCDF, and Python tests, however netCDF4-python performs less efficiently.

Item Type:Book or Report Section
Divisions:Science > School of Mathematical, Physical and Computational Sciences > NCAS
Science > School of Mathematical, Physical and Computational Sciences > Department of Meteorology
ID Code:67988
Publisher:Springer International Publishing

University Staff: Request a correction | Centaur Editors: Update this record

Page navigation