IOPS

Design Philosophy

Mon, 01 Jan 0001 00:00:00 +0000

Five design principles guide IOPS development. They explain why IOPS looks the way it does and what trade-offs were chosen.

1. One YAML to Rule Them All

A single, self-contained specification captures the complete benchmark lifecycle: workload definition, parameter space, run rules, environment, and validation criteria. You should never need a second file, a sidecar script, or a hidden environment variable to understand what a study does. The YAML is the source of truth.

Execution Loop

Mon, 01 Jan 0001 00:00:00 +0000

This page documents the IOPS execution loop architecture: how the Runner, Planner, and Executor components orchestrate benchmark execution.

Overview

The execution loop follows a pull-based design where the Runner requests tests from the Planner and delegates execution to the Executor:

┌─────────────────────────────────────────────────────────────────┐
│ IOPSRunner │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Planner │─────►│ Runner │─────►│ Executor │ │
│ │ │ │ Loop │ │ │ │
│ │ next_test() │ │ │ │ submit() │ │
│ │ │◄─────│ │◄─────│ wait_and_ │ │
│ │ record_ │ │ │ │ collect() │ │
│ │ completed() │ │ │ │ │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Matrix │ │ Cache │ │ Parser │ │
│ │ Builder │ │ │ │ │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────────┘

Component Roles

Component	Responsibility
Runner	Orchestrates the execution loop, manages cache, tracks budget, writes results
Planner	Generates test configurations, manages execution order, prepares artifacts
Executor	Submits jobs, waits for completion, collects output and metrics
Matrix Builder	Generates parameter combinations from config, applies constraints
Cache	Stores/retrieves results to skip redundant executions
Parser	Extracts metrics from benchmark output files

The Execution Loop

Main Loop (`IOPSRunner.run()`)

def run(self):
 while True:
 # 1. Check budget
 if budget_exceeded:
 break

 # 2. Get next test from planner
 test = self.planner.next_test()
 if test is None:
 break # No more tests

 # 3. Check cache
 if self.cache and self.use_cache_reads:
 cached_result = self.cache.get_cached_result(test.vars, test.repetition)
 if cached_result:
 # Use cached result, skip execution
 test.metadata.update(cached_result)
 test.metadata['__cached'] = True

 # 4. Execute if not cached
 if not used_cache:
 self._write_status_file(test, status="RUNNING")
 self.executor.submit(test)
 self.executor.wait_and_collect(test)

 # Store in cache if succeeded
 if succeeded:
 self.cache.store_result(...)

 # 5. Write status and results
 self._write_status_file(test)
 save_test_execution(test)

 # 6. Notify planner of completion
 self.planner.record_completed_test(test)

Sequence Diagram

Runner Planner Executor Cache
 │ │ │ │
 │ next_test() │ │ │
 │───────────────────►│ │ │
 │ │ │ │
 │ ExecutionInstance │ │ │
 │◄───────────────────│ │ │
 │ │ │ │
 │ get_cached_result() │ │
 │──────────────────────────────────────────────────────────►│
 │ │ │ │
 │ (cache miss) │ │ │
 │◄──────────────────────────────────────────────────────────│
 │ │ │ │
 │ submit(test) │ │ │
 │────────────────────────────────────────►│ │
 │ │ │ │
 │ wait_and_collect(test) │ │
 │────────────────────────────────────────►│ │
 │ │ │ │
 │ test.metadata updated │ │
 │◄────────────────────────────────────────│ │
 │ │ │ │
 │ store_result() │ │ │
 │──────────────────────────────────────────────────────────►│
 │ │ │ │
 │ record_completed_test() │ │
 │───────────────────►│ │ │
 │ │ │ │

Planner Architecture

Base Planner

All planners inherit from BasePlanner and implement:

Installation

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites
Quick Installation (from PyPI)
Installation from Source
Offline Installation (Wheelhouse)
Installation with Spack (for HPC environments)
Optional Dependencies
Verifying Your Installation

Prerequisites

Before installing IOPS, ensure you have:

Python 3.10 or later
For benchmark execution: Required tools in PATH (e.g., ior, mpirun for I/O benchmarks)
For SLURM clusters: Access to a SLURM scheduler

Virtual environment (recommended)

Install IOPS into an isolated environment (venv or Conda, both work the same) so its dependencies do not conflict with other Python tools on the machine.

IOPS YAML Format Reference

Mon, 01 Jan 0001 00:00:00 +0000

Complete guide to the IOPS benchmark configuration format

File Structure Overview
benchmark
vars
constraints (optional)
command
scripts
output
reporting (optional)
hooks (optional)
machines (optional)

See also: Templating and Context Reference for Jinja2 syntax, dynamic values, conditionals, and context variables.

File Structure Overview

benchmark: # Global configuration (name, workdir, executor, search method)
vars: # Variable definitions (swept and derived)
constraints: # (Optional) Parameter validation rules
command: # Command template and labels
scripts: # Execution scripts and parsers
output: # Output configuration (CSV, Parquet, SQLite)
reporting: # (Optional) Report generation settings
hooks: # (Optional) Commands fired on run events
machines: # (Optional) Per-machine config overrides

Required sections: benchmark, vars, command, scripts, output

Templating and Context Reference

Mon, 01 Jan 0001 00:00:00 +0000

IOPS uses Jinja2 templating throughout the configuration to enable dynamic values, conditional logic, and computed expressions. This guide covers both the Jinja2 syntax and the context variables available in each configuration section.

Overview
Jinja2 Syntax Reference
Context by Configuration Section
Complete Example

Overview

Template Support by Field

Field	Jinja2 Support	File Path Support	Inline Support
`command.template`	Yes	No	Yes
`command.env` values	Yes	No	Yes
`scripts[].script_template`	Yes	Yes	Yes
`scripts[].post.script`	Yes	Yes	Yes
`scripts[].parser.file`	Yes	No	Yes
`scripts[].parser.parser_script`	No	Yes	Yes
`vars[].expr`	Yes	No	Yes
`output.sink.path`	Yes	No	Yes
`hooks[].command`	Yes (own context, see Hooks)	No	Yes

Standard Context Variables

These variables are available in all Jinja2 templates:

Contributing to IOPS

Mon, 01 Jan 0001 00:00:00 +0000

Guidelines for contributing to IOPS.

Getting Started

Fork the Repository

# Clone your fork
git clone https://gitlab.inria.fr/YOUR_USERNAME/iops.git
cd iops

Set Up Development Environment

# Create virtual environment
python3 -m venv iops_env
source iops_env/bin/activate

# Install in editable mode
pip install -e .

# Install development dependencies
pip install -r requirements-dev.txt

# Verify the setup
pytest tests/

Development Workflow

Create a Branch

git checkout -b feature/my-new-feature
# or
git checkout -b fix/issue-description

Make Changes

Probe System

Mon, 01 Jan 0001 00:00:00 +0000

This page documents the IOPS system probe mechanism for developers who want to extend the collected metrics or modify probe behavior.

Overview

The probe system collects system information (CPU, memory, filesystems, etc.) from the actual compute node where the benchmark runs. This matters for SLURM jobs, where the login node differs from compute nodes.

Key files:

__iops_exit_handler.sh - Centralized EXIT trap coordinator
__iops_node_launcher.sh - Resolves the job’s node list and starts one helper per node
__iops_atexit_sysinfo.sh - System info collection script
__iops_sysinfo.json - Collected data (generated at runtime)

Architecture

User Script Exit Handler Sysinfo Script
┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
│ #!/bin/bash │ │ # Sets EXIT trap │ │ # Registers cleanup │
│ #SBATCH ... │ │ trap '_iops_run_ │ │ _iops_register_exit │
│ │ │ exit_actions' EXIT│ │ "_iops_collect_ │
│ # IOPS injection │ │ │ │ sysinfo" │
│ source handler.sh │───────►│ # Registry array │ │ │
│ source sysinfo.sh │───────────────────────────────────────►│ # Collection func │
│ │ │ _iops_register_exit │ │ _iops_collect_ │
│ # User's benchmark │ │ () { ... } │ │ sysinfo() { ... } │
│ mpirun ./benchmark │ └─────────────────────┘ └─────────────────────┘
└─────────────────────┘ │
 │ │
 ▼ (on script exit) ▼
┌─────────────────────┐ ┌─────────────────────┐
│ __iops_sysinfo.json │◄───────│ _iops_run_exit_ │
│ { │ │ actions() calls │
│ "hostname": ..., │ │ all registered │
│ "cpu_model": ..., │ │ functions │
│ "duration": ... │ └─────────────────────┘
│ } │
└─────────────────────┘

The exit handler provides a centralized EXIT trap that other IOPS features (system probe, resource sampler) register cleanup actions with, avoiding trap conflicts.

Quick Start

Mon, 01 Jan 0001 00:00:00 +0000

Run your first benchmark with IOPS in a few minutes.

Step 1: Create a Configuration
Step 2: Preview Your Benchmark
Step 3: Run the Benchmark
Step 4: Explore Executions
Step 5: Analyze Results
Understanding the Output

Step 1: Create a Configuration

Generate a YAML configuration template:

iops generate my_config.yaml

# Fully-documented template with all options
iops generate my_config.yaml --full

Step 2: Preview Your Benchmark

# Dry-run to see what will be executed
iops run my_config.yaml --dry-run
iops run my_config.yaml -n

# Check configuration validity
iops check my_config.yaml

The dry-run shows how many test instances will be generated, the parameter combinations, and estimated resource usage (for SLURM).

Writing Scripts

Mon, 01 Jan 0001 00:00:00 +0000

The scripts[] section of your YAML tells IOPS how to launch your benchmark. IOPS owns the parameter sweep and the result database; you own the shell script that actually runs the workload. This page is the practical guide to writing scripts; for the schema (every field, every option), see YAML Schema Reference / scripts.

Why You Need a Script Template

IOPS does not assume anything about how your benchmark is launched: some benchmarks are a single executable, others need MPI launchers, module loads, environment setup, or pre-flight checks. The script_template is the bridge: a Jinja2-rendered shell script that IOPS materializes to disk and hands to the configured executor. Per test execution, IOPS writes it to <execution_dir>/run_<name>.sh so you can inspect (or rerun) exactly what was executed.

Writing Parsers

Mon, 01 Jan 0001 00:00:00 +0000

IOPS does not know the output format of your benchmark. Instead, you provide a small Python script (the parser) that reads whatever your benchmark produces and returns the metrics you care about. This page explains how parsers work, how to write one, and where the extracted data ends up.

How It Works

Every benchmark writes its results differently: JSON files, plain text logs, CSV tables, binary formats. The parser is the bridge between raw benchmark output and structured data that IOPS can store, plot, and analyze. After each execution completes, IOPS runs it through this sequence:

Data Sources

Mon, 01 Jan 0001 00:00:00 +0000

This page documents where IOPS CLI commands read their data.

Overview

IOPS commands read from result files and from metadata files generated during execution. Metadata files have the __iops_ prefix and are documented in the Metadata Files user guide.

Data Sources by Command

iops find ◄─── __iops_index.json
 ◄─── __iops_status_rollup.json (preferred; falls back to per-folder scan)
 ◄─── __iops_status.json (fallback)
 ◄─── __iops_run_metadata.json (optional metadata)

iops find --watch ◄─── __iops_index.json (polling)
 ◄─── __iops_status_rollup.json (preferred; polling)
 ◄─── __iops_status.json (fallback scan)

iops report ◄─── __iops_run_metadata.json
 ◄─── results.csv/parquet/sqlite

Status roll-up fast path

Both iops find and watch mode read the run-root __iops_status_rollup.json aggregate instead of scanning every execution’s repetition folders, which keeps them responsive on large parameter spaces. The per-folder __iops_status.json files remain the source of truth (and the node-to-runner channel in single-allocation mode); the roll-up is an accelerator. Watch always prefers it; iops find only trusts it when the run marked it complete, otherwise it falls back to the folder scan. When the roll-up is absent (older runs, archives), both fall back automatically.

Deprecations

Mon, 01 Jan 0001 00:00:00 +0000

This page lists deprecated features and provides migration guidance.

Backwards Compatibility Policy

IOPS follows a 2 minor version deprecation cycle:

Version	Behavior
N	Feature deprecated with warning, old behavior still works
N+1	Warning remains, old behavior still works
N+2	Feature removed, error with migration guidance

This gives users at least two releases to migrate their configurations.

Current Deprecations

`benchmark.executor_options` → `benchmark.slurm_options`


Deprecated in	3.5.0
Remove after	3.7.0
Reason	Renamed for clarity (options are SLURM-specific)

Before:

Execution Matrix Generation

Mon, 01 Jan 0001 00:00:00 +0000

How IOPS builds the parameter space from your variable definitions

Overview

IOPS generates an execution matrix from your variable definitions; each row is a unique parameter combination to test.

vars: Execution Matrix:
 nodes: [1, 2] → | nodes | threads |
 threads: [4, 8] | 1 | 4 |
 | 1 | 8 |
 | 2 | 4 |
 | 2 | 8 |

Variable Types

Swept Variables

Variables with sweep define the parameter space. IOPS creates a Cartesian product of all swept variables.

IOPS Studio

Mon, 01 Jan 0001 00:00:00 +0000

Experimental Feature

IOPS Studio is experimental. It may contain bugs or undergo breaking changes in future releases. Please report any issues you encounter.

IOPS Studio is a local web UI for driving IOPS end to end: connect to a target (your laptop or an HPC cluster over SSH), build a benchmark config from a form, run it, and view the results, all from your browser.

Overview

Studio runs as a small local web server and does its work through one interactive shell per target, the same terminal you see on the right of the screen. Connecting to a cluster is just an ssh session inside that terminal, so everything works exactly as it would by hand, and if a command fails you are already in the right shell to fix it. Because it uses the one already-authenticated channel, Studio also works on locked-down login nodes that only accept an interactive password or 2FA.

Search Methods

Mon, 01 Jan 0001 00:00:00 +0000

IOPS supports multiple search strategies for exploring your parameter space, each with different trade-offs between thoroughness and efficiency.

Exhaustive Search

Tests all possible parameter combinations. Thorough but expensive for large parameter spaces.

benchmark:
 search_method: "exhaustive"

Bayesian Optimization

Uses a surrogate model to explore the parameter space, focusing on promising regions. Default parameters are empirically tuned: with 20 iterations (~7% of search space), Bayesian optimization achieves ~90% of optimal vs ~79% for random search.

Bayesian Optimization

Mon, 01 Jan 0001 00:00:00 +0000

Bayesian optimization lets IOPS explore a parameter space to find configurations that maximize or minimize a target metric, using far fewer evaluations than exhaustive search. Instead of testing every combination, it builds a surrogate model from observed results and focuses on the most promising regions.

Overview
Basic Configuration
How IOPS Implements Bayesian Optimization
Choosing a Surrogate Model
Acquisition Functions
Convergence and Early Stopping
- Default behavior
- Early stopping with xi boost
Combining with Exhaustive Variables
Fallback to Exhaustive
Configuration Reference

Overview

Bayesian optimization (BO) is a sequential, model-based strategy for optimizing expensive black-box functions. In benchmarking, each “function evaluation” is a full benchmark run that may take minutes or hours; BO reduces the number of runs needed by learning from previous results. Use it when runs are expensive and you want the best configuration without testing every combination. Prefer exhaustive or random search when the space is small enough to cover fully, when you need complete coverage for statistical analysis, or when there is no clear optimization target.

Changelog

Mon, 01 Jan 0001 00:00:00 +0000

All notable changes to IOPS are documented here.

[3.5.10] - 2026-10-26

Added

iops deps lists the optional dependencies, what each one enables, and whether it is installed, so the packages behind Bayesian search, Parquet sinks, watch mode, plot export, gallery downscaling, and Studio can be checked in one place instead of by triggering the feature and reading the error. --missing narrows the table to what is absent, and --check exits with status 1 when anything is missing, for use in CI. The same catalog now backs the availability flags inside IOPS: iops/deps.py holds one entry per extra and the modules that gate on a package call is_available() instead of carrying their own try/except, so the command and the runtime cannot disagree about what is installed. Detection uses each package’s import name rather than its distribution name, which is why Pillow (imported as PIL) is reported correctly. Every install hint IOPS prints now quotes the package spec (pip install "iops-benchmark[bayesian]"), because zsh treats the square brackets of an extra as a glob and rejects the unquoted form with no matches found. A test asserts no source file emits an unquoted hint.
xaxis_range and yaxis_range set explicit limits on a report plot’s axes, as [min, max], instead of leaving the zoom to Plotly. Both are written in data units even when the axis is logarithmic (yaxis_range: [1e-6, 1e-2] with log_y: true shows 10^-6 to 10^-2), since the conversion to powers of ten happens at render time. This makes several plots directly comparable on identical axes, lets a bar chart be pinned to zero, and allows zooming into a band of interest. The bounds must be numbers with min below max, and on a logarithmic axis the lower bound must be positive, because such an axis cannot represent zero. When a range is set, the decade tick rule reads the range rather than the data. The options are also available in the Studio plot editor.
Hooks run a command of your choice when something happens during a run, which makes a long run on a cluster easy to follow without watching the terminal. A top-level hooks: list names an event (start, progress, failures, consecutive_failures, or finish), says when it fires with at (one-shot thresholds) and/or every (a repeating threshold), and gives a command. Thresholds are an integer count or a percentage string such as "25%", counted in attempts (executions times repetitions). A failures percentage is the failure rate among tests that ran and is held back until min_tests have run (default 10, lowered on small runs), consecutive_failures re-arms after a success, and finish hooks can be restricted with when to the outcomes completed, interrupted, budget_exceeded, fail_fast, or crashed. The command is a Jinja2 template rendered against the run context (progress, run, benchmark.name, workdir, report_path, hostname, os_env, and a ready-made message), with a quote filter for the shell, and the same values are exported as IOPS_* environment variables for scripts. Commands run with bash on the host running IOPS, in the background and in order, each in its own session with a timeout (default 60 seconds), so a slow or failing hook is logged as a warning and never affects the run, and finish waits for pending hooks before IOPS exits. Output goes to __iops_hooks.log in the run directory. Hooks are listed but not fired by --dry-run, iops run --no-hooks disables them, and hooks can be overridden per entry under machines. iops check renders every hook command against a sample context to catch misspelled names, and iops hooks test config.yaml (with --hook, --event, --render-only, and --machine) runs the commands once so a mail command can be tried before a long run. A bare on: key is rejected with a pointer to event:, because YAML reads it as a boolean. See the Hooks guide and examples/hooks/hooks.yaml.

Changed

Every IOPS command starts about twice as fast. Deciding whether an optional package was installed was done by importing it, and importing scikit-optimize pulls in scikit-learn and SciPy behind it, which cost roughly 0.8 seconds. That was paid on every invocation to set a boolean, including iops --version and iops find, whether or not the run used Bayesian search. The availability gates now locate a module instead of executing it, which measures as no cost at all, taking startup from about 1.0 to about 0.45 seconds; the remainder is pandas, reached through the results writer. iops deps still performs a real import, since there the answer is the product and a package that is installed but broken should be reported as missing. This became a one-line change because the gates had already moved behind a single catalog.
A logarithmic axis (log_x, log_y) whose data spans two decades or more is now labelled once per decade, as a power of ten. Plotly labels minor log ticks by default, so an axis covering four decades read 2, 0.001, 5, 2, 100µ, 5, 2, 10µ, mixing minor labels in with the decade marks. Data confined to less than two decades keeps the automatic ticks: those labels are ordinary numbers, and a decade rule would leave such an axis with a single label or none at all, since its whole range can sit between two powers of ten.

Fixed

The report_config.yaml written after a run dropped most plot options: only type, x_var, y_var, group_by, title, and colorscale were kept, so log_x, log_y, axis labels, sizes, error-bar and outlier toggles, z_metric, color_by, size_by, and the coverage heatmap options were lost. Because iops report auto-loads that file in preference to the run metadata, a regenerated report silently ignored them, even though the report produced at the end of the run honored them. Every non-default plot field is now written, for both reporting.metrics plots and reporting.default_plots. Runs created with an earlier version keep the truncated file; restore the missing options by hand, or delete the file so iops report falls back to the configuration stored in the run metadata.
Bar plot value labels used two decimal places, so metrics below 0.005 were all labeled 0.00. Labels now show three significant digits (0.000956).
With fail_fast enabled, the core-hours of the test that stopped the run were not added to the total, because the run stopped before they were counted. They are now included, so the budget total and hooks see the full cost.
The automatic HTML report produced at the end of a run now includes resource sampling data. It was generated before the run metadata was finalized and __iops_resource_summary.csv was written, so custom plots on resource metrics (such as gpu_avg_utilization_pct) reported “not found in results”, and the Resource Sampling table, System Environment section, and total runtime were missing until the report was regenerated with iops report. The report is now built as the last step of a run that completes normally.
The version probe wrote an __iops_versions.json that no JSON parser could read whenever a version command printed more than one line on macOS or BSD, or printed a tab on any system. Its escaping relied on a GNU-only sed idiom and a tab substitution that never took effect. Escaping now uses bash parameter expansion and produces identical, valid output on bash 3.2 and 5.x.
Studio no longer hangs on targets whose terminal truncates long input lines. Every command was sent as a single line, so one longer than the target tty’s input buffer was cut short and the shell waited forever for the missing newline, leaving the session unusable. Environment discovery hit this on clusters such as PLAFRIM, which truncate at 1024 bytes. Commands that would exceed the budget are now streamed to a temporary script on the target and invoked with a short line (sourced, so module load and export still persist). Streamed transfers to macOS and BSD targets also work now: they decoded with base64 -d "$file", which BSD base64 rejects.

[3.5.9] - 2026-08-19

Added

IOPS Studio (experimental): a local web UI for using IOPS end to end, launched with iops studio (requires the optional nicegui dependency). See the Getting Started guide.

Adaptive Variables

Mon, 01 Jan 0001 00:00:00 +0000

Adaptive variables let IOPS probe for threshold values by starting at an initial value and progressing until a stop condition is met. Use them to find limits such as maximum problem sizes, memory capacities, or performance cliffs without manually defining the search space.

Overview
Basic Configuration
Step Methods
Stop Conditions
Direction
Safety Limits
Probe Results
Multiple Repetitions
Combining with Swept Variables
- How probes advance
- Multiple swept variables
Staircase Search with an Escalating Variable
Complete Example
Constraints
- Why only one adaptive variable?
- Adaptive vs Bayesian optimization
Configuration Reference

Overview

Unlike swept variables (which test a fixed list of values) or derived variables (which are computed from other variables), an adaptive variable is driven by a feedback loop:

Caching

Mon, 01 Jan 0001 00:00:00 +0000

Overview
Configuration
Inspecting the Cache
Rebuilding the Cache

Overview

IOPS caches execution results in SQLite so that tests with identical parameters can be skipped on re-runs. This speeds up iterative development, lets you safely retry failed runs without re-executing successful tests, and handles repetitions correctly (each repetition is cached separately).

License

Mon, 01 Jan 0001 00:00:00 +0000

IOPS is licensed under the BSD 3-Clause License.

BSD 3-Clause License

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

Machine Overrides

Mon, 01 Jan 0001 00:00:00 +0000

Use a single configuration file across multiple systems with per-machine overrides.

Overview
Merge Rules
Configuration
Example: Local Development + HPC Cluster
Use Cases
Tips

Overview

The machines section lets you define per-machine overrides inside a single YAML file. Instead of maintaining separate configs for your laptop, HPC clusters, or cloud instances, write one base config and override only what differs per environment.

Select the target machine at runtime:

Execution Backends

Mon, 01 Jan 0001 00:00:00 +0000

IOPS supports two execution backends for running benchmarks: local execution and SLURM cluster submission.

Local Executor
Parallel Execution
SLURM Executor

Local Executor

Runs benchmarks directly on your local machine using subprocesses. Suited for development, small experiments, single-node benchmarks, and quick iterations.

benchmark:
 name: "Local Test"
 workdir: "./workdir"
 executor: "local"

scripts:
 - name: "benchmark"
 script_template: |
 #!/bin/bash
 set -euo pipefail
 {{ command.template }}

Parallel Execution

By default, IOPS runs tests sequentially (one at a time). Run multiple tests concurrently by setting the parallel option:

Running IO500 with IOPS

Mon, 01 Jan 0001 00:00:00 +0000

Running the IO500 Benchmark with IOPS

The configuration generated by:

iops generate --io500

[!NOTE] This is a ready-to-use template, but it should not be used as-is. Before running any benchmark, make sure to review and adjust the configuration to match your HPC system, storage setup, and testing goals.

The generated YAML file includes comments that explain each parameter and how to customize it. It is strongly recommended to read them before starting the benchmark.

Single-Allocation Mode

Mon, 01 Jan 0001 00:00:00 +0000

Experimental Feature

Single-allocation mode is experimental. It may contain bugs or undergo breaking changes in future releases. Please report any issues you encounter.

Single-allocation mode runs all tests within a single SLURM allocation instead of submitting separate jobs for each test.

Overview
When to Use
Basic Configuration
How It Works
Running MPI Programs

Overview

By default, IOPS submits a separate SLURM job for each test (per-test mode). Single-allocation mode changes this behavior:

Budget Control

Mon, 01 Jan 0001 00:00:00 +0000

IOPS can track core-hours consumption and automatically stop execution when a budget limit is reached, helping prevent exceeding compute allocations on HPC systems.

Configuration
How Core-Hours Are Calculated
Duration Source and Accuracy
Budget Enforcement Behavior
Dry-Run Estimation
Cache Interaction

Configuration

Set a budget limit in your YAML config:

benchmark:
 max_core_hours: 1000 # Stop after 1000 core-hours
 cores_expr: "{{ nodes * processes_per_node }}"

Or from command line (overrides config value):

iops run config.yaml --max-core-hours 1000

Option	Description
`max_core_hours`	Maximum core-hours budget. Execution stops when reached.
`cores_expr`	Jinja2 expression to compute cores per test. Defaults to `1` if not set.

How Core-Hours Are Calculated

core_hours = cores × (duration_seconds / 3600)

cores: Evaluated from cores_expr using test variables
duration_seconds: Actual script execution time (see Duration Source)

Setting cores_expr Correctly

The cores_expr should reflect the cores you’re actually charged for on your HPC system:

Hooks

Mon, 01 Jan 0001 00:00:00 +0000

Hooks run a command of your choice when something happens during a run: the run starts, a share of the tests has concluded, too many tests are failing, or the run ends. The typical use is monitoring a long run on a cluster, for example sending a mail when the run is half done and another when it finishes. They are equally useful for housekeeping, such as copying the results somewhere once the run is over.

Resource Sampling

Mon, 01 Jan 0001 00:00:00 +0000

IOPS can optionally sample CPU, memory, GPU, and I/O activity during benchmark execution, so you can correlate parameter configurations with resource footprint (e.g., heatmap analysis of how parameters affect resource usage).

Warning: Performance Impact

Resource sampling runs background processes that periodically collect system metrics and write to CSV files. Although the samplers run at low priority (via renice), they may still affect benchmark results: a small amount of CPU time for reading proc files, an nvidia-smi invocation per GPU sample, one CSV append per sample interval, and ~1-2 MB of memory per process. For performance-critical measurements, run your benchmark without sampling first to establish a baseline, then enable sampling in a separate run to collect resource data.

Metadata Files

Mon, 01 Jan 0001 00:00:00 +0000

IOPS generates metadata files with the __iops_ prefix to track executions, enable fast lookups, and support report generation. This page documents each file, when it is written, and how to control its generation.

I/O Overhead Considerations
Metadata Files Overview
Controlling Metadata Generation
File Reference
Best Practices

I/O Overhead Considerations

Metadata files are written to the workdir during benchmark execution. They are small (typically a few KB each), but consider disabling them when:

Exploring Executions

Mon, 01 Jan 0001 00:00:00 +0000

IOPS provides tools to explore, search, and inspect execution folders after running benchmarks, useful for large parameter sweeps where you need to find specific configurations or investigate failures. For the full flag reference, see the CLI guide.

Finding Executions
Filtering Executions
Advanced Options
Watch Mode
IOPS Metadata Files
Inspecting Archives
Archiving Workdirs

Finding Executions

Use the find command to locate and display execution folders with their parameters:

iops find /path/to/workdir

The find command works with four types of paths:

Command Line Interface

Mon, 01 Jan 0001 00:00:00 +0000

Complete reference for the IOPS command-line interface.

Overview
Basic Usage
Subcommand Reference

Overview

IOPS uses a subcommand-based CLI similar to git and docker. Each subcommand has its own options and help.

Basic Usage

iops <subcommand> [arguments] [options]

Command	Purpose
`iops run <config.yaml>`	Execute benchmark
`iops check <config.yaml>`	Validate configuration
`iops generate [path]`	Create config template
`iops report <path>`	Generate HTML report
`iops find <path> [filters...]`	Explore executions
`iops archive <create\|extract>`	Archive and extract workdirs
`iops cache <create\|rebuild\|list\|show\|stats>`	Cache management
`iops convert <file.xml>`	Convert JUBE XML to IOPS YAML

Subcommand Reference

run - Execute Benchmark

iops run <config.yaml> [options]

Options:

Custom Reports & Visualization

Mon, 01 Jan 0001 00:00:00 +0000

IOPS includes a reporting system that generates interactive HTML reports with custom plots and statistical analysis from your benchmark results.

Introduction
Basic Usage
Report Sections
Controlling Sections
Custom Plots
- Logarithmic Axis Scales
- Explicit Axis Ranges
Default Plots
Themes and Styling
Plot Sizing and Defaults
Output Configuration
Per-Test Image Gallery
Software Version Capture Probe
Examples

Introduction

Reports are generated as self-contained HTML files with embedded interactive Plotly visualizations. You can auto-generate reports after execution, define custom plots and themes per metric, control which sections appear, regenerate reports with different configurations without re-running benchmarks, and optionally export all plots as image files (requires kaleido).

JUBE Conversion

Mon, 01 Jan 0001 00:00:00 +0000

Experimental Feature

JUBE conversion is experimental. It may contain bugs or undergo breaking changes in future releases. Please report any issues you encounter.

Convert existing JUBE benchmark configurations to IOPS format using iops convert.

Overview

JUBE (Juelich Benchmarking Environment) is an HPC benchmarking framework that uses XML configurations. iops convert provides a best-effort translation of JUBE benchmarks to IOPS YAML, using JUBE’s own Python library to parse the XML for accurate interpretation of JUBE-specific syntax. The output includes TODO markers for features that require manual adjustment.

Measuring the runtime overhead of an I/O interception library

Sat, 30 May 2026 00:00:00 +0000

This study is part of our paper “TOTO: Transparent I/O Tuning for HPC Applications”, accepted at ICS 2026 (International Conference on Supercomputing, Belfast, July 2026). The overhead measurements below were produced with IOPS.

When you intercept I/O system calls to analyze application behavior, how much overhead do you actually add? That’s the question we needed to answer for TOTO (Transparent and Online Tool for I/O), a library that hooks into I/O calls via LD_PRELOAD to provide runtime analysis of parallel file system access patterns.

Using IOPS on Grid'5000

Mon, 26 Jan 2026 00:00:00 +0000

Grid'5000 is a large-scale and flexible testbed for experiment-driven research in all areas of computer science, with a focus on parallel and distributed computing, including Cloud, HPC, Big Data and AI.

On Grid'5000 platform the reservation of computing nodes and scheduling of batch jobs is managed by the OAR software suite, while IOPS is only supporting Slurm as a batch scheduler. Thankfully, the high level of similarity between OAR and Slurm (at least from a user perspective) makes it easy to run IOPS on Grid'5000 front-end with OAR being used to schedule & control the test jobs on the computing nodes. Let’s see how…

Testing IOPS's BO search against random sampling using cached HPC results

Sat, 24 Jan 2026 00:00:00 +0000

IOPS can use Bayesian Optimization when the objective is to find the parameter combination that yields the maximum (or minimum) metric value in a search space. In this article, we decided to test the efficiency of IOPS’s BO search by comparing it against IOPS’s random search. The idea is simple: given a search space, run both IOPS BO and Random, then compare the results to see which one reaches the optimal first. And of course, it should not be a surprise at this point that we used an I/O benchmark to do it.

Search

Mon, 01 Jan 2024 00:00:00 +0000

IOPS

Design Philosophy

1. One YAML to Rule Them All

Execution Loop

Overview

Component Roles

The Execution Loop

Main Loop (IOPSRunner.run())

Sequence Diagram

Planner Architecture

Base Planner

Installation

Table of Contents

Prerequisites

Virtual environment (recommended)

IOPS YAML Format Reference

Table of Contents

File Structure Overview

Templating and Context Reference

Table of Contents

Overview

Template Support by Field

Standard Context Variables

Contributing to IOPS

Getting Started

Development Workflow

Probe System

Overview

Architecture

Quick Start

Table of Contents

Step 1: Create a Configuration

Step 2: Preview Your Benchmark

Writing Scripts

Why You Need a Script Template

Writing Parsers

How It Works

Data Sources

Overview

Data Sources by Command

Status roll-up fast path

Deprecations

Backwards Compatibility Policy

Current Deprecations

benchmark.executor_options → benchmark.slurm_options

Execution Matrix Generation

Overview

Variable Types

Swept Variables

IOPS Studio

Overview

Search Methods

Exhaustive Search

Bayesian Optimization

Bayesian Optimization

Table of Contents

Overview

Changelog

[3.5.10] - 2026-10-26

Added

Changed

Fixed

[3.5.9] - 2026-08-19

Added

Adaptive Variables

Table of Contents

Overview

Caching

Table of Contents

Overview

License

BSD 3-Clause License

Machine Overrides

Table of Contents

Overview

Execution Backends

Table of Contents

Local Executor

Parallel Execution

Running IO500 with IOPS

Main Loop (`IOPSRunner.run()`)

`benchmark.executor_options` → `benchmark.slurm_options`