Design Philosophy

Mon, 01 Jan 0001 00:00:00 +0000

Five design principles guide IOPS development. They explain why IOPS looks the way it does and what trade-offs were chosen.

1. One YAML to Rule Them All

A single, self-contained specification captures the complete benchmark lifecycle: workload definition, parameter space, run rules, environment, and validation criteria. You should never need a second file, a sidecar script, or a hidden environment variable to understand what a study does. The YAML is the source of truth.

Contributing to IOPS

Mon, 01 Jan 0001 00:00:00 +0000

Guidelines for contributing to IOPS.

Getting Started

Fork the Repository

# Clone your fork
git clone https://gitlab.inria.fr/YOUR_USERNAME/iops.git
cd iops

Set Up Development Environment

# Create virtual environment
python3 -m venv iops_env
source iops_env/bin/activate

# Install in editable mode
pip install -e .

# Install development dependencies
pip install -r requirements-dev.txt

# Verify the setup
pytest tests/

Development Workflow

Create a Branch

git checkout -b feature/my-new-feature
# or
git checkout -b fix/issue-description

Make Changes

Deprecations

Mon, 01 Jan 0001 00:00:00 +0000

This page lists deprecated features and provides migration guidance.

Backwards Compatibility Policy

IOPS follows a 2 minor version deprecation cycle:

Version	Behavior
N	Feature deprecated with warning, old behavior still works
N+1	Warning remains, old behavior still works
N+2	Feature removed, error with migration guidance

This gives users at least two releases to migrate their configurations.

Current Deprecations

`benchmark.executor_options` → `benchmark.slurm_options`


Deprecated in	3.5.0
Remove after	3.7.0
Reason	Renamed for clarity (options are SLURM-specific)

Before:

Changelog

Mon, 01 Jan 0001 00:00:00 +0000

All notable changes to IOPS are documented here.

[3.5.10] - 2026-10-26

Added

iops deps lists the optional dependencies, what each one enables, and whether it is installed, so the packages behind Bayesian search, Parquet sinks, watch mode, plot export, gallery downscaling, and Studio can be checked in one place instead of by triggering the feature and reading the error. --missing narrows the table to what is absent, and --check exits with status 1 when anything is missing, for use in CI. The same catalog now backs the availability flags inside IOPS: iops/deps.py holds one entry per extra and the modules that gate on a package call is_available() instead of carrying their own try/except, so the command and the runtime cannot disagree about what is installed. Detection uses each package’s import name rather than its distribution name, which is why Pillow (imported as PIL) is reported correctly. Every install hint IOPS prints now quotes the package spec (pip install "iops-benchmark[bayesian]"), because zsh treats the square brackets of an extra as a glob and rejects the unquoted form with no matches found. A test asserts no source file emits an unquoted hint.
xaxis_range and yaxis_range set explicit limits on a report plot’s axes, as [min, max], instead of leaving the zoom to Plotly. Both are written in data units even when the axis is logarithmic (yaxis_range: [1e-6, 1e-2] with log_y: true shows 10^-6 to 10^-2), since the conversion to powers of ten happens at render time. This makes several plots directly comparable on identical axes, lets a bar chart be pinned to zero, and allows zooming into a band of interest. The bounds must be numbers with min below max, and on a logarithmic axis the lower bound must be positive, because such an axis cannot represent zero. When a range is set, the decade tick rule reads the range rather than the data. The options are also available in the Studio plot editor.
Hooks run a command of your choice when something happens during a run, which makes a long run on a cluster easy to follow without watching the terminal. A top-level hooks: list names an event (start, progress, failures, consecutive_failures, or finish), says when it fires with at (one-shot thresholds) and/or every (a repeating threshold), and gives a command. Thresholds are an integer count or a percentage string such as "25%", counted in attempts (executions times repetitions). A failures percentage is the failure rate among tests that ran and is held back until min_tests have run (default 10, lowered on small runs), consecutive_failures re-arms after a success, and finish hooks can be restricted with when to the outcomes completed, interrupted, budget_exceeded, fail_fast, or crashed. The command is a Jinja2 template rendered against the run context (progress, run, benchmark.name, workdir, report_path, hostname, os_env, and a ready-made message), with a quote filter for the shell, and the same values are exported as IOPS_* environment variables for scripts. Commands run with bash on the host running IOPS, in the background and in order, each in its own session with a timeout (default 60 seconds), so a slow or failing hook is logged as a warning and never affects the run, and finish waits for pending hooks before IOPS exits. Output goes to __iops_hooks.log in the run directory. Hooks are listed but not fired by --dry-run, iops run --no-hooks disables them, and hooks can be overridden per entry under machines. iops check renders every hook command against a sample context to catch misspelled names, and iops hooks test config.yaml (with --hook, --event, --render-only, and --machine) runs the commands once so a mail command can be tried before a long run. A bare on: key is rejected with a pointer to event:, because YAML reads it as a boolean. See the Hooks guide and examples/hooks/hooks.yaml.

Changed

Every IOPS command starts about twice as fast. Deciding whether an optional package was installed was done by importing it, and importing scikit-optimize pulls in scikit-learn and SciPy behind it, which cost roughly 0.8 seconds. That was paid on every invocation to set a boolean, including iops --version and iops find, whether or not the run used Bayesian search. The availability gates now locate a module instead of executing it, which measures as no cost at all, taking startup from about 1.0 to about 0.45 seconds; the remainder is pandas, reached through the results writer. iops deps still performs a real import, since there the answer is the product and a package that is installed but broken should be reported as missing. This became a one-line change because the gates had already moved behind a single catalog.
A logarithmic axis (log_x, log_y) whose data spans two decades or more is now labelled once per decade, as a power of ten. Plotly labels minor log ticks by default, so an axis covering four decades read 2, 0.001, 5, 2, 100µ, 5, 2, 10µ, mixing minor labels in with the decade marks. Data confined to less than two decades keeps the automatic ticks: those labels are ordinary numbers, and a decade rule would leave such an axis with a single label or none at all, since its whole range can sit between two powers of ten.

Fixed

The report_config.yaml written after a run dropped most plot options: only type, x_var, y_var, group_by, title, and colorscale were kept, so log_x, log_y, axis labels, sizes, error-bar and outlier toggles, z_metric, color_by, size_by, and the coverage heatmap options were lost. Because iops report auto-loads that file in preference to the run metadata, a regenerated report silently ignored them, even though the report produced at the end of the run honored them. Every non-default plot field is now written, for both reporting.metrics plots and reporting.default_plots. Runs created with an earlier version keep the truncated file; restore the missing options by hand, or delete the file so iops report falls back to the configuration stored in the run metadata.
Bar plot value labels used two decimal places, so metrics below 0.005 were all labeled 0.00. Labels now show three significant digits (0.000956).
With fail_fast enabled, the core-hours of the test that stopped the run were not added to the total, because the run stopped before they were counted. They are now included, so the budget total and hooks see the full cost.
The automatic HTML report produced at the end of a run now includes resource sampling data. It was generated before the run metadata was finalized and __iops_resource_summary.csv was written, so custom plots on resource metrics (such as gpu_avg_utilization_pct) reported “not found in results”, and the Resource Sampling table, System Environment section, and total runtime were missing until the report was regenerated with iops report. The report is now built as the last step of a run that completes normally.
The version probe wrote an __iops_versions.json that no JSON parser could read whenever a version command printed more than one line on macOS or BSD, or printed a tab on any system. Its escaping relied on a GNU-only sed idiom and a tab substitution that never took effect. Escaping now uses bash parameter expansion and produces identical, valid output on bash 3.2 and 5.x.
Studio no longer hangs on targets whose terminal truncates long input lines. Every command was sent as a single line, so one longer than the target tty’s input buffer was cut short and the shell waited forever for the missing newline, leaving the session unusable. Environment discovery hit this on clusters such as PLAFRIM, which truncate at 1024 bytes. Commands that would exceed the budget are now streamed to a temporary script on the target and invoked with a short line (sourced, so module load and export still persist). Streamed transfers to macOS and BSD targets also work now: they decoded with base64 -d "$file", which BSD base64 rejects.

[3.5.9] - 2026-08-19

Added

IOPS Studio (experimental): a local web UI for using IOPS end to end, launched with iops studio (requires the optional nicegui dependency). See the Getting Started guide.

License

Mon, 01 Jan 0001 00:00:00 +0000

IOPS is licensed under the BSD 3-Clause License.

BSD 3-Clause License

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

About on IOPS

Design Philosophy

1. One YAML to Rule Them All

Contributing to IOPS

Getting Started

Development Workflow

Deprecations

Backwards Compatibility Policy

Current Deprecations

benchmark.executor_options → benchmark.slurm_options

Changelog

[3.5.10] - 2026-10-26

Added

Changed

Fixed

[3.5.9] - 2026-08-19

Added

License

BSD 3-Clause License

`benchmark.executor_options` → `benchmark.slurm_options`