IOPS

A generic benchmark orchestration framework for automated parametric experiments.

IOPS automates the generation, execution, and analysis of benchmark experiments. Instead of writing custom scripts for each study, you define a YAML configuration describing what to vary, what to run, and what to measure; IOPS handles the rest.

Without IOPS: Write bash scripts → Parse outputs → Aggregate data → Generate plots → Repeat for each parameter change

With IOPS: Write one YAML config → Run iops run config.yaml → Get interactive HTML reports

Originally designed for I/O performance studies (see our 2022 paper), IOPS has evolved into a generic framework for any parametric benchmark workflow.

Key Features

  • Parameter Sweeping: Generate and execute tests for all parameter combinations
  • Search Strategies: Exhaustive, Bayesian optimization, or random sampling
  • Execution Backends: Run locally or submit to SLURM clusters
  • Execution Exploration: Find and filter execution folders by parameter values
  • Caching: Skip redundant tests with parameter-aware result caching
  • Budget Control: Set core-hour limits to avoid exceeding compute allocations
  • Reports: Interactive HTML reports with plots and statistical analysis
  • Output: Export results to CSV, Parquet, or SQLite

Blog

Tutorials and use cases on the blog: