Accessibility navigation


Understanding metadata latency with MDWorkbench

Kunkel, J. M. and Markomanolis, G. S. (2018) Understanding metadata latency with MDWorkbench. In: Workshop on Performance and Scalability of Storage Systems, 24-28 June 2018, Frankfurt, Germany, pp. 75-88. (ISBN 9783030024642)

[img]
Preview
Text (Updated) - Accepted Version
· Please see our End User Agreement before downloading.

2MB
[img] Text - Accepted Version
· Restricted to Repository staff only

2MB

It is advisable to refer to the publisher's version if you intend to cite from this work. See Guidance on citing.

Official URL: https://doi.org/10.1007/978-3-030-02465-9_5

Abstract/Summary

While parallel file systems often satisfy the need of applica- tions with bulk synchronous I/O, they lack capabilities of dealing with metadata intense workloads. Typically, in procurements, the focus lies on the aggregated metadata throughput using the MDTest benchmark. However, metadata performance is crucial for interactive use. Metadata benchmarks involve even more parameters compared to I/O benchmarks. There are several aspects that are currently uncovered and, therefore, not in the focus of vendors to investigate. Particularly, response latency and interactive workloads operating on a working set of data. The lack of ca- pabilities from file systems can be observed when looking at the IO-500 list, where metadata performance between best and worst system does not differ significantly. In this paper, we introduce a new benchmark called MDWorkbench which generates a reproducible workload emulating many concurrent users or – in an alternative view – queuing systems. This benchmark pro- vides a detailed latency profile, overcomes caching issues, and provides a method to assess the quality of the observed throughput. We evaluate the benchmark on state-of-the-art parallel file systems with GPFS (IBM Spectrum Scale), Lustre, Cray’s Datawarp, and DDN IME, and conclude that we can reveal characteristics that could not be identified before.

Item Type:Conference or Workshop Item (Paper)
Refereed:Yes
Divisions:Faculty of Science > School of Mathematical, Physical and Computational Sciences > Department of Computer Science
ID Code:79590
Additional Information:Part of the Lecture Notes in Computer Science book series (LNCS, volume 11203).
Publisher:Springer

Downloads

Downloads per month over past year

University Staff: Request a correction | Centaur Editors: Update this record

Page navigation