User Guide on IOPS

IOPS YAML Format Reference

Mon, 01 Jan 0001 00:00:00 +0000

Complete guide to the IOPS benchmark configuration format

File Structure Overview
benchmark
vars
constraints (optional)
command
scripts
output
reporting (optional)
hooks (optional)
machines (optional)

See also: Templating and Context Reference for Jinja2 syntax, dynamic values, conditionals, and context variables.

File Structure Overview

benchmark: # Global configuration (name, workdir, executor, search method)
vars: # Variable definitions (swept and derived)
constraints: # (Optional) Parameter validation rules
command: # Command template and labels
scripts: # Execution scripts and parsers
output: # Output configuration (CSV, Parquet, SQLite)
reporting: # (Optional) Report generation settings
hooks: # (Optional) Commands fired on run events
machines: # (Optional) Per-machine config overrides

Required sections: benchmark, vars, command, scripts, output

Templating and Context Reference

Mon, 01 Jan 0001 00:00:00 +0000

IOPS uses Jinja2 templating throughout the configuration to enable dynamic values, conditional logic, and computed expressions. This guide covers both the Jinja2 syntax and the context variables available in each configuration section.

Overview
Jinja2 Syntax Reference
Context by Configuration Section
Complete Example

Overview

Template Support by Field

Field	Jinja2 Support	File Path Support	Inline Support
`command.template`	Yes	No	Yes
`command.env` values	Yes	No	Yes
`scripts[].script_template`	Yes	Yes	Yes
`scripts[].post.script`	Yes	Yes	Yes
`scripts[].parser.file`	Yes	No	Yes
`scripts[].parser.parser_script`	No	Yes	Yes
`vars[].expr`	Yes	No	Yes
`output.sink.path`	Yes	No	Yes
`hooks[].command`	Yes (own context, see Hooks)	No	Yes

Standard Context Variables

These variables are available in all Jinja2 templates:

Writing Scripts

Mon, 01 Jan 0001 00:00:00 +0000

The scripts[] section of your YAML tells IOPS how to launch your benchmark. IOPS owns the parameter sweep and the result database; you own the shell script that actually runs the workload. This page is the practical guide to writing scripts; for the schema (every field, every option), see YAML Schema Reference / scripts.

Why You Need a Script Template

IOPS does not assume anything about how your benchmark is launched: some benchmarks are a single executable, others need MPI launchers, module loads, environment setup, or pre-flight checks. The script_template is the bridge: a Jinja2-rendered shell script that IOPS materializes to disk and hands to the configured executor. Per test execution, IOPS writes it to <execution_dir>/run_<name>.sh so you can inspect (or rerun) exactly what was executed.

Writing Parsers

Mon, 01 Jan 0001 00:00:00 +0000

IOPS does not know the output format of your benchmark. Instead, you provide a small Python script (the parser) that reads whatever your benchmark produces and returns the metrics you care about. This page explains how parsers work, how to write one, and where the extracted data ends up.

How It Works

Every benchmark writes its results differently: JSON files, plain text logs, CSV tables, binary formats. The parser is the bridge between raw benchmark output and structured data that IOPS can store, plot, and analyze. After each execution completes, IOPS runs it through this sequence:

Execution Matrix Generation

Mon, 01 Jan 0001 00:00:00 +0000

How IOPS builds the parameter space from your variable definitions

Overview

IOPS generates an execution matrix from your variable definitions; each row is a unique parameter combination to test.

vars: Execution Matrix:
 nodes: [1, 2] → | nodes | threads |
 threads: [4, 8] | 1 | 4 |
 | 1 | 8 |
 | 2 | 4 |
 | 2 | 8 |

Variable Types

Swept Variables

Variables with sweep define the parameter space. IOPS creates a Cartesian product of all swept variables.

Search Methods

Mon, 01 Jan 0001 00:00:00 +0000

IOPS supports multiple search strategies for exploring your parameter space, each with different trade-offs between thoroughness and efficiency.

Exhaustive Search

Tests all possible parameter combinations. Thorough but expensive for large parameter spaces.

benchmark:
 search_method: "exhaustive"

Bayesian Optimization

Uses a surrogate model to explore the parameter space, focusing on promising regions. Default parameters are empirically tuned: with 20 iterations (~7% of search space), Bayesian optimization achieves ~90% of optimal vs ~79% for random search.

Bayesian Optimization

Mon, 01 Jan 0001 00:00:00 +0000

Bayesian optimization lets IOPS explore a parameter space to find configurations that maximize or minimize a target metric, using far fewer evaluations than exhaustive search. Instead of testing every combination, it builds a surrogate model from observed results and focuses on the most promising regions.

Overview
Basic Configuration
How IOPS Implements Bayesian Optimization
Choosing a Surrogate Model
Acquisition Functions
Convergence and Early Stopping
- Default behavior
- Early stopping with xi boost
Combining with Exhaustive Variables
Fallback to Exhaustive
Configuration Reference

Overview

Bayesian optimization (BO) is a sequential, model-based strategy for optimizing expensive black-box functions. In benchmarking, each “function evaluation” is a full benchmark run that may take minutes or hours; BO reduces the number of runs needed by learning from previous results. Use it when runs are expensive and you want the best configuration without testing every combination. Prefer exhaustive or random search when the space is small enough to cover fully, when you need complete coverage for statistical analysis, or when there is no clear optimization target.

Adaptive Variables

Mon, 01 Jan 0001 00:00:00 +0000

Adaptive variables let IOPS probe for threshold values by starting at an initial value and progressing until a stop condition is met. Use them to find limits such as maximum problem sizes, memory capacities, or performance cliffs without manually defining the search space.

Overview
Basic Configuration
Step Methods
Stop Conditions
Direction
Safety Limits
Probe Results
Multiple Repetitions
Combining with Swept Variables
- How probes advance
- Multiple swept variables
Staircase Search with an Escalating Variable
Complete Example
Constraints
- Why only one adaptive variable?
- Adaptive vs Bayesian optimization
Configuration Reference

Overview

Unlike swept variables (which test a fixed list of values) or derived variables (which are computed from other variables), an adaptive variable is driven by a feedback loop:

Caching

Mon, 01 Jan 0001 00:00:00 +0000

Overview
Configuration
Inspecting the Cache
Rebuilding the Cache

Overview

IOPS caches execution results in SQLite so that tests with identical parameters can be skipped on re-runs. This speeds up iterative development, lets you safely retry failed runs without re-executing successful tests, and handles repetitions correctly (each repetition is cached separately).

Machine Overrides

Mon, 01 Jan 0001 00:00:00 +0000

Use a single configuration file across multiple systems with per-machine overrides.

Overview
Merge Rules
Configuration
Example: Local Development + HPC Cluster
Use Cases
Tips

Overview

The machines section lets you define per-machine overrides inside a single YAML file. Instead of maintaining separate configs for your laptop, HPC clusters, or cloud instances, write one base config and override only what differs per environment.

Select the target machine at runtime:

Execution Backends

Mon, 01 Jan 0001 00:00:00 +0000

IOPS supports two execution backends for running benchmarks: local execution and SLURM cluster submission.

Local Executor
Parallel Execution
SLURM Executor

Local Executor

Runs benchmarks directly on your local machine using subprocesses. Suited for development, small experiments, single-node benchmarks, and quick iterations.

benchmark:
 name: "Local Test"
 workdir: "./workdir"
 executor: "local"

scripts:
 - name: "benchmark"
 script_template: |
 #!/bin/bash
 set -euo pipefail
 {{ command.template }}

Parallel Execution

By default, IOPS runs tests sequentially (one at a time). Run multiple tests concurrently by setting the parallel option:

Running IO500 with IOPS

Mon, 01 Jan 0001 00:00:00 +0000

Running the IO500 Benchmark with IOPS

The configuration generated by:

iops generate --io500

[!NOTE] This is a ready-to-use template, but it should not be used as-is. Before running any benchmark, make sure to review and adjust the configuration to match your HPC system, storage setup, and testing goals.

The generated YAML file includes comments that explain each parameter and how to customize it. It is strongly recommended to read them before starting the benchmark.

Single-Allocation Mode

Mon, 01 Jan 0001 00:00:00 +0000

Experimental Feature

Single-allocation mode is experimental. It may contain bugs or undergo breaking changes in future releases. Please report any issues you encounter.

Single-allocation mode runs all tests within a single SLURM allocation instead of submitting separate jobs for each test.

Overview
When to Use
Basic Configuration
How It Works
Running MPI Programs

Overview

By default, IOPS submits a separate SLURM job for each test (per-test mode). Single-allocation mode changes this behavior:

Budget Control

Mon, 01 Jan 0001 00:00:00 +0000

IOPS can track core-hours consumption and automatically stop execution when a budget limit is reached, helping prevent exceeding compute allocations on HPC systems.

Configuration
How Core-Hours Are Calculated
Duration Source and Accuracy
Budget Enforcement Behavior
Dry-Run Estimation
Cache Interaction

Configuration

Set a budget limit in your YAML config:

benchmark:
 max_core_hours: 1000 # Stop after 1000 core-hours
 cores_expr: "{{ nodes * processes_per_node }}"

Or from command line (overrides config value):

iops run config.yaml --max-core-hours 1000

Option	Description
`max_core_hours`	Maximum core-hours budget. Execution stops when reached.
`cores_expr`	Jinja2 expression to compute cores per test. Defaults to `1` if not set.

How Core-Hours Are Calculated

core_hours = cores × (duration_seconds / 3600)

cores: Evaluated from cores_expr using test variables
duration_seconds: Actual script execution time (see Duration Source)

Setting cores_expr Correctly

The cores_expr should reflect the cores you’re actually charged for on your HPC system:

Hooks

Mon, 01 Jan 0001 00:00:00 +0000

Hooks run a command of your choice when something happens during a run: the run starts, a share of the tests has concluded, too many tests are failing, or the run ends. The typical use is monitoring a long run on a cluster, for example sending a mail when the run is half done and another when it finishes. They are equally useful for housekeeping, such as copying the results somewhere once the run is over.

Resource Sampling

Mon, 01 Jan 0001 00:00:00 +0000

IOPS can optionally sample CPU, memory, GPU, and I/O activity during benchmark execution, so you can correlate parameter configurations with resource footprint (e.g., heatmap analysis of how parameters affect resource usage).

Warning: Performance Impact

Resource sampling runs background processes that periodically collect system metrics and write to CSV files. Although the samplers run at low priority (via renice), they may still affect benchmark results: a small amount of CPU time for reading proc files, an nvidia-smi invocation per GPU sample, one CSV append per sample interval, and ~1-2 MB of memory per process. For performance-critical measurements, run your benchmark without sampling first to establish a baseline, then enable sampling in a separate run to collect resource data.

Metadata Files

Mon, 01 Jan 0001 00:00:00 +0000

IOPS generates metadata files with the __iops_ prefix to track executions, enable fast lookups, and support report generation. This page documents each file, when it is written, and how to control its generation.

I/O Overhead Considerations
Metadata Files Overview
Controlling Metadata Generation
File Reference
Best Practices

I/O Overhead Considerations

Metadata files are written to the workdir during benchmark execution. They are small (typically a few KB each), but consider disabling them when:

Exploring Executions

Mon, 01 Jan 0001 00:00:00 +0000

IOPS provides tools to explore, search, and inspect execution folders after running benchmarks, useful for large parameter sweeps where you need to find specific configurations or investigate failures. For the full flag reference, see the CLI guide.

Finding Executions
Filtering Executions
Advanced Options
Watch Mode
IOPS Metadata Files
Inspecting Archives
Archiving Workdirs

Finding Executions

Use the find command to locate and display execution folders with their parameters:

iops find /path/to/workdir

The find command works with four types of paths:

Command Line Interface

Mon, 01 Jan 0001 00:00:00 +0000

Complete reference for the IOPS command-line interface.

Overview
Basic Usage
Subcommand Reference

Overview

IOPS uses a subcommand-based CLI similar to git and docker. Each subcommand has its own options and help.

Basic Usage

iops <subcommand> [arguments] [options]

Command	Purpose
`iops run <config.yaml>`	Execute benchmark
`iops check <config.yaml>`	Validate configuration
`iops generate [path]`	Create config template
`iops report <path>`	Generate HTML report
`iops find <path> [filters...]`	Explore executions
`iops archive <create\|extract>`	Archive and extract workdirs
`iops cache <create\|rebuild\|list\|show\|stats>`	Cache management
`iops convert <file.xml>`	Convert JUBE XML to IOPS YAML

Subcommand Reference

run - Execute Benchmark

iops run <config.yaml> [options]

Options:

Custom Reports & Visualization

Mon, 01 Jan 0001 00:00:00 +0000

IOPS includes a reporting system that generates interactive HTML reports with custom plots and statistical analysis from your benchmark results.

Introduction
Basic Usage
Report Sections
Controlling Sections
Custom Plots
- Logarithmic Axis Scales
- Explicit Axis Ranges
Default Plots
Themes and Styling
Plot Sizing and Defaults
Output Configuration
Per-Test Image Gallery
Software Version Capture Probe
Examples

Introduction

Reports are generated as self-contained HTML files with embedded interactive Plotly visualizations. You can auto-generate reports after execution, define custom plots and themes per metric, control which sections appear, regenerate reports with different configurations without re-running benchmarks, and optionally export all plots as image files (requires kaleido).

JUBE Conversion

Mon, 01 Jan 0001 00:00:00 +0000

Experimental Feature

JUBE conversion is experimental. It may contain bugs or undergo breaking changes in future releases. Please report any issues you encounter.

Convert existing JUBE benchmark configurations to IOPS format using iops convert.

Overview

JUBE (Juelich Benchmarking Environment) is an HPC benchmarking framework that uses XML configurations. iops convert provides a best-effort translation of JUBE benchmarks to IOPS YAML, using JUBE’s own Python library to parse the XML for accurate interpretation of JUBE-specific syntax. The output includes TODO markers for features that require manual adjustment.

User Guide on IOPS

IOPS YAML Format Reference

Table of Contents

File Structure Overview

Templating and Context Reference

Table of Contents

Overview

Template Support by Field

Standard Context Variables

Writing Scripts

Why You Need a Script Template

Writing Parsers

How It Works

Execution Matrix Generation

Overview

Variable Types

Swept Variables

Search Methods

Exhaustive Search

Bayesian Optimization

Bayesian Optimization

Table of Contents

Overview

Adaptive Variables

Table of Contents

Overview

Caching

Table of Contents

Overview

Machine Overrides

Table of Contents

Overview

Execution Backends

Table of Contents

Local Executor

Parallel Execution

Running IO500 with IOPS

Running the IO500 Benchmark with IOPS

Single-Allocation Mode

Table of Contents

Overview

Budget Control

Table of Contents

Configuration

How Core-Hours Are Calculated

Setting cores_expr Correctly

Hooks

Resource Sampling

Metadata Files

Table of Contents

I/O Overhead Considerations

Exploring Executions

Table of Contents

Finding Executions

Command Line Interface

Table of Contents

Overview

Basic Usage

Subcommand Reference

run - Execute Benchmark

Custom Reports & Visualization

Table of Contents

Introduction

JUBE Conversion

Overview