Accessibility navigation


Predicting performance of non-contiguous I/O with machine learning

Kunkel, J., Zimmer, M. and Betke, E. (2015) Predicting performance of non-contiguous I/O with machine learning. In: Kunkel, J. M. and Ludwig, T. (eds.) High Performance Computing. Lecture Notes in Computer Science (9137). Springer, pp. 257-273. ISBN 9783319201184

[img]
Preview
Text - Accepted Version
· Please see our End User Agreement before downloading.

2MB

It is advisable to refer to the publisher's version if you intend to cite from this work. See Guidance on citing.

To link to this item DOI: 10.1007/978-3-319-20119-1_19

Abstract/Summary

Data sieving in ROMIO promises to optimize individual non-contiguous I/O. However, making the right choice and parameterizing its buffer size accordingly are non-trivial tasks, since predicting the resulting performance is difficult. Since many performance factors are not taken into account by data sieving, extracting the optimal performance for a given access pattern and system is often not possible. Additionally, in Lustre, settings such as the stripe size and number of servers are tunable, yet again, identifying rules for the data-centre proves challenging indeed. In this paper, we (1) discuss limitations of data sieving, (2) apply machine learning techniques to build a performance predictor, and (3) learn and extract best practices for the settings from the data. We used decision trees as these models can capture non-linear behavior, are easy to understand and allow for extraction of the rules used. Even though this initial research is based on decision trees, with sparse training data, the algorithm can predict many cases sufficiently. Compared to a standard setting, the decision trees created are able to improve performance significantly and we can derive expert knowledge by extracting rules from the learned tree. Applying the scheme to a set of experimental data improved the average throughput by 25–50 % of the best parametrization’s gain. Additionally, we demonstrate the versatility of this approach by applying it to the porting system of DKRZ’s next generation supercomputer and discuss achievable performance gains.

Item Type:Book or Report Section
Refereed:Yes
Divisions:Faculty of Science > School of Mathematical, Physical and Computational Sciences > Department of Computer Science
ID Code:77678
Publisher:Springer

Downloads

Downloads per month over past year

University Staff: Request a correction | Centaur Editors: Update this record

Page navigation