Changelog

All notable changes to IOPS are documented here.

[3.5.10] - 2026-10-26

Added

• iops deps lists the optional dependencies, what each one enables, and whether it is installed, so the packages behind Bayesian search, Parquet sinks, watch mode, plot export, gallery downscaling, and Studio can be checked in one place instead of by triggering the feature and reading the error. --missing narrows the table to what is absent, and --check exits with status 1 when anything is missing, for use in CI. The same catalog now backs the availability flags inside IOPS: iops/deps.py holds one entry per extra and the modules that gate on a package call is_available() instead of carrying their own try/except, so the command and the runtime cannot disagree about what is installed. Detection uses each package’s import name rather than its distribution name, which is why Pillow (imported as PIL) is reported correctly. Every install hint IOPS prints now quotes the package spec (pip install "iops-benchmark[bayesian]"), because zsh treats the square brackets of an extra as a glob and rejects the unquoted form with no matches found. A test asserts no source file emits an unquoted hint.
• xaxis_range and yaxis_range set explicit limits on a report plot’s axes, as [min, max], instead of leaving the zoom to Plotly. Both are written in data units even when the axis is logarithmic (yaxis_range: [1e-6, 1e-2] with log_y: true shows 10^-6 to 10^-2), since the conversion to powers of ten happens at render time. This makes several plots directly comparable on identical axes, lets a bar chart be pinned to zero, and allows zooming into a band of interest. The bounds must be numbers with min below max, and on a logarithmic axis the lower bound must be positive, because such an axis cannot represent zero. When a range is set, the decade tick rule reads the range rather than the data. The options are also available in the Studio plot editor.
• Hooks run a command of your choice when something happens during a run, which makes a long run on a cluster easy to follow without watching the terminal. A top-level hooks: list names an event (start, progress, failures, consecutive_failures, or finish), says when it fires with at (one-shot thresholds) and/or every (a repeating threshold), and gives a command. Thresholds are an integer count or a percentage string such as "25%", counted in attempts (executions times repetitions). A failures percentage is the failure rate among tests that ran and is held back until min_tests have run (default 10, lowered on small runs), consecutive_failures re-arms after a success, and finish hooks can be restricted with when to the outcomes completed, interrupted, budget_exceeded, fail_fast, or crashed. The command is a Jinja2 template rendered against the run context (progress, run, benchmark.name, workdir, report_path, hostname, os_env, and a ready-made message), with a quote filter for the shell, and the same values are exported as IOPS_* environment variables for scripts. Commands run with bash on the host running IOPS, in the background and in order, each in its own session with a timeout (default 60 seconds), so a slow or failing hook is logged as a warning and never affects the run, and finish waits for pending hooks before IOPS exits. Output goes to __iops_hooks.log in the run directory. Hooks are listed but not fired by --dry-run, iops run --no-hooks disables them, and hooks can be overridden per entry under machines. iops check renders every hook command against a sample context to catch misspelled names, and iops hooks test config.yaml (with --hook, --event, --render-only, and --machine) runs the commands once so a mail command can be tried before a long run. A bare on: key is rejected with a pointer to event:, because YAML reads it as a boolean. See the Hooks guide and examples/hooks/hooks.yaml.

Changed

• Every IOPS command starts about twice as fast. Deciding whether an optional package was installed was done by importing it, and importing scikit-optimize pulls in scikit-learn and SciPy behind it, which cost roughly 0.8 seconds. That was paid on every invocation to set a boolean, including iops --version and iops find, whether or not the run used Bayesian search. The availability gates now locate a module instead of executing it, which measures as no cost at all, taking startup from about 1.0 to about 0.45 seconds; the remainder is pandas, reached through the results writer. iops deps still performs a real import, since there the answer is the product and a package that is installed but broken should be reported as missing. This became a one-line change because the gates had already moved behind a single catalog.
• A logarithmic axis (log_x, log_y) whose data spans two decades or more is now labelled once per decade, as a power of ten. Plotly labels minor log ticks by default, so an axis covering four decades read 2, 0.001, 5, 2, 100µ, 5, 2, 10µ, mixing minor labels in with the decade marks. Data confined to less than two decades keeps the automatic ticks: those labels are ordinary numbers, and a decade rule would leave such an axis with a single label or none at all, since its whole range can sit between two powers of ten.

Fixed

• The report_config.yaml written after a run dropped most plot options: only type, x_var, y_var, group_by, title, and colorscale were kept, so log_x, log_y, axis labels, sizes, error-bar and outlier toggles, z_metric, color_by, size_by, and the coverage heatmap options were lost. Because iops report auto-loads that file in preference to the run metadata, a regenerated report silently ignored them, even though the report produced at the end of the run honored them. Every non-default plot field is now written, for both reporting.metrics plots and reporting.default_plots. Runs created with an earlier version keep the truncated file; restore the missing options by hand, or delete the file so iops report falls back to the configuration stored in the run metadata.
• Bar plot value labels used two decimal places, so metrics below 0.005 were all labeled 0.00. Labels now show three significant digits (0.000956).
• With fail_fast enabled, the core-hours of the test that stopped the run were not added to the total, because the run stopped before they were counted. They are now included, so the budget total and hooks see the full cost.
• The automatic HTML report produced at the end of a run now includes resource sampling data. It was generated before the run metadata was finalized and __iops_resource_summary.csv was written, so custom plots on resource metrics (such as gpu_avg_utilization_pct) reported “not found in results”, and the Resource Sampling table, System Environment section, and total runtime were missing until the report was regenerated with iops report. The report is now built as the last step of a run that completes normally.
• The version probe wrote an __iops_versions.json that no JSON parser could read whenever a version command printed more than one line on macOS or BSD, or printed a tab on any system. Its escaping relied on a GNU-only sed idiom and a tab substitution that never took effect. Escaping now uses bash parameter expansion and produces identical, valid output on bash 3.2 and 5.x.
• Studio no longer hangs on targets whose terminal truncates long input lines. Every command was sent as a single line, so one longer than the target tty’s input buffer was cut short and the shell waited forever for the missing newline, leaving the session unusable. Environment discovery hit this on clusters such as PLAFRIM, which truncate at 1024 bytes. Commands that would exceed the budget are now streamed to a temporary script on the target and invoked with a short line (sourced, so module load and export still persist). Streamed transfers to macOS and BSD targets also work now: they decoded with base64 -d "$file", which BSD base64 rejects.

[3.5.9] - 2026-08-19

Added

• IOPS Studio (experimental): a local web UI for using IOPS end to end, launched with iops studio (requires the optional nicegui dependency). See the Getting Started guide.

  • Setups: save targets (local machine or an SSH host from ~/.ssh/config), each with a Python environment, workdir, and setup commands. A wizard walks connection, environment discovery/creation, and IOPS install (pip, with an offline wheelhouse fallback transferred over the interactive channel for hosts with no network). Setups can be edited (renaming migrates their configs and tracked runs) and validated live.
  • Config builder: the full IOPS YAML schema as a section-navigated form beside a live, two-way-synced YAML editor, with three view modes (Form + YAML, Form, YAML). Covers every option including SLURM/single-allocation, budget and core-hours, variables (sweeps/expr/adaptive), scripts, output, probes, constraints, and the whole reporting block (sections, plots, gallery, log axes). Script templates get bash highlighting and parser code gets Python highlighting. Import a host YAML as an editable copy, or export a config to the host.
  • Resilient runs: benchmarks run inside a screen session and survive dropped connections; Studio records the login node and reattaches, hopping back if the alias load-balances. A run-options dialog exposes --use-cache, --cache-only, --dry-run, and --fail-fast.
  • Integrated results: browse a target’s runs, view the HTML report in-app (charts render offline via a bundled Plotly), edit the run’s report_config.yaml and regenerate the report in place, or pull a run’s small artifacts (results, metadata, report, logs) back to the host. Raw scratch data is deliberately excluded so transfers stay small.
  • Multiple terminals: keep several targets connected at once, one tab each, with per-tab status and reconnect. Enable --log-level DEBUG to trace every command Studio sends.

• log_x and log_y (both default false) on report plots switch the corresponding axis to a logarithmic scale. Set them on any plot in reporting.metrics.<metric>.plots or reporting.default_plots; they are also exposed as checkboxes in the Studio plot editor. log_y applies to the numeric metric axis on every plot type and is the common case when metric values span several orders of magnitude. log_x is meaningful on plots whose x-axis is numeric (scatter, heatmap, surface_3d); on bar, line, box, and violin plots the x-axis is rendered as ordered categories, so log_x has no visible effect there.

• iops report --output <path> (short form -o) writes the HTML report to an explicit path, taking priority over the config’s output_dir/output_filename.

• benchmark.probes.io_sampling (default false) samples I/O activity during execution. It reads two counter sources every sampling_interval, because neither one alone describes both a local disk and a network filesystem: /proc/diskstats for block devices, which reports nothing for NFS since that traffic never reaches a client block device, and /proc/self/mountstats for NFS mounts, which reports nothing for local disks. Each row of __iops_io_trace_<hostname>_<attempt_id>.csv is tagged with its source, so a study comparing storage backends gets both arms rather than zeros on one side. The run summary gains io_read_gb, io_write_gb, their per-source counterparts (io_disk_*, io_nfs_*), average and peak throughput, and average operations per second; all of them are registered as report metrics and appear in the Resource Sampling table. NFS byte counts come from the server-read and server-written counters, so reads the page cache satisfied locally are excluded and the two backends are compared like with like. Block devices are counted once: partitions and virtual devices (dm-*, md*, loop*) are excluded to avoid double counting. Counters are node-level, so they include anything else running on the node.

• benchmark.probes.io_paths names the storage I/O sampling measures, and is required whenever io_sampling is enabled, e.g. ["{{ execution_dir }}", "/scratch/input"]. There is no “measure everything” mode on purpose: a node usually has storage the benchmark never touches, so a run writing to /tmp on a node that also mounts NFS would report both and neither number would describe the run. Paths are Jinja2 templates rendered per execution, so they can name a directory the run creates, and they are resolved on the compute node because mount tables differ across an allocation. A path that does not exist yet resolves through its nearest existing ancestor. A local path resolves to the whole block device behind it (a partition to its parent disk, an LVM or md volume to the disks underneath), so block scoping is only as precise as the device layout; an NFS path resolves to that mount exactly. A path with no countable storage behind it, such as anything on tmpfs, is reported rather than silently zero: the run logs a warning and every node writes __iops_io_targets_<hostname>_<attempt_id>.json recording what each path resolved to. Throughput is reported two ways, because sampling covers the whole script and most benchmarks spend much of it not doing I/O: io_*_mbs_avg divides by the full traced window and answers what the job moved per second of runtime, while io_*_mbs_active divides by io_*_active_s, the time that direction actually moved data, and answers how fast the storage went while in use. Volumes and peaks are unaffected by idle samples either way. Trace rows gain a path column naming the configured entry they came from. Paths are a filter rather than a per-path breakdown: counters are deduplicated, so two directories on the same disk are counted once and labelled with both, since the kernel keeps no per-directory counters to tell them apart.

• Escalating variables (escalate), a new variable kind that pairs with an adaptive variable to perform a staircase search. A plain adaptive probe finishes the first time stop_when triggers; with an escalating variable, the escalating value advances and the same adaptive value is retested instead, and the probe ends only when the escalation values run out. This expresses searches where one variable should advance precisely when the other stalls, such as finding the smallest number_of_blocks each problem_size needs, equivalently the largest problem_size each number_of_blocks can handle. The result is reported as a frontier, one rung per escalation value, in the run summary and under adaptive_results in __iops_run_metadata.json, with the new stop_reason value escalation_exhausted. Only one axis moves per step, so the search stays well defined; the cost is P + B - 1 runs against up to P x B for sweeping the second axis. At most one escalating variable per config, and it requires an adaptive variable to extend. The staircase never de-escalates, so it assumes a value that fails at one adaptive value also fails at every later one. See Adaptive Variables.

• HTML reports now show a staircase search as a frontier. The Adaptive Probing Results section previously reported only the probe’s final state, a single stop_value/last_value_before_stop pair, which for a staircase is the point where the search ran out of escalation values and hides the result entirely. The section now leads with a Search Frontier table, one row per escalation value with the furthest adaptive value it reached, and a Search Path plot showing every test the search ran (successes as green circles, stop-condition hits as red crosses) with a step line tracing the frontier. The escalation axis is spaced by position, so values like 1, 4, 16 stay evenly spaced. The Probing Configuration block lists the escalating variable and its values, and the section text describes the staircase rather than a single-threshold probe. Plain adaptive runs are unchanged.

• A gallery extra installs Pillow, which reporting.gallery.max_width needs to downscale images: pip install iops-benchmark[gallery]. Pillow was documented as a requirement of that option but was not declared anywhere, so it had to be installed by hand.

Deprecated

• The adaptive probe result fields found_value and failed_value are renamed to last_value_before_stop and stop_value. The old names assumed that stopping meant failure, which holds only for the usual stop_when: "exit_code != 0". An inverted condition is equally valid: stop_when: "exit_code == 0" keeps stepping while the benchmark fails and stops at the first value that succeeds, and in that case the value stored under failed_value was the one that worked, so results read backwards. The new names describe the probe’s progression instead, and are correct whichever way stop_when points. This affects the end-of-run summary, the adaptive_results block in __iops_run_metadata.json, the “Probe Results Summary” table in HTML reports, and the ProbeResult attributes. Both key sets are written to metadata and both attribute names resolve until the old ones are removed after 3.7.0; reports generated from runs recorded earlier fall back to the old keys automatically. See Deprecations.

Fixed

• I/O sampling scoped with io_paths no longer reports NFS mounts the paths never resolved to. The NFS filter read an empty selection as “no filtering”, so a run scoped to a local directory silently counted every NFS mount on the node. On a cluster whose nodes mount a package store and NFS home directories, a run scoped to /tmp attributed nearly all of its reported reads to module loading rather than to the benchmark. Once paths are configured, an empty selection for a source now correctly means nothing from that source.

• __iops_resource_summary.csv is no longer lost when executions report different metric sets. The header was taken from the first row with data, so any later row carrying a column that row lacked failed to write and the whole summary was dropped with a dict contains fields not in fieldnames warning. This happened when a GPU reported a field as unavailable in one execution but not another, and when an execution finished too quickly for its samplers to collect anything and so reported only the sample counters. Columns are now the union across all rows, in first-seen order, and cells with no value are left empty.

• Resource and GPU sampling now cover every node of a multi-node job on OAR and PBS, not just the head node. The samplers fanned out only when SLURM_JOB_ID was set and SLURM_NNODES > 1, using srun; on any other scheduler that test failed and the sampler ran in the background of the head node alone, so a 4-node run reported nodes_traced: 1 and its memory, CPU, and imbalance metrics described a single node. A shared node launcher (__iops_node_launcher.sh) now resolves the node list per scheduler (scontrol show hostnames for SLURM, $OAR_NODEFILE for OAR, $PBS_NODEFILE for PBS) and starts one sampler per node (srun --overlap, oarsh, or ssh respectively), falling back to the local node when no scheduler is detected. The per-attempt id used in sentinel and trace filenames now also recognises OAR_JOB_ID and PBS_JOBID, and is forwarded to remote nodes so all of them agree on the sentinel path. Remote traces require the execution directory to be on a shared filesystem; nodes_traced in __iops_resource_summary.csv reports how many nodes actually returned samples.

• iops report now works for adaptive runs that have no swept variables. Report variables defaulted to numeric swept variables only, and an adaptive variable is never swept, so a config whose only varying parameter was the adaptive one failed with No report variables found. Either specify report_vars in config or ensure you have numeric swept variables. The default now includes the variables the search itself drives (adaptive and escalating), which in such a run is the only axis there is to plot against. An explicit report_vars still takes precedence, and string-typed variables are still excluded. Runs recorded by earlier versions report correctly without regeneration.

• An adaptive step_expr that cannot be rendered no longer aborts the entire benchmark. This mattered most for the common pattern of indexing a literal list ("{{ [16, 32, 64][iteration] }}") to walk a fixed set of values: if max_iterations allowed more steps than the list held values, the probe eventually indexed past the end and the run died with a bare jinja2.exceptions.UndefinedError: list object has no element N, discarding every probe still in flight. Two changes address this. iops check now renders step_expr for each iteration the planner will request and reports the mismatch before anything runs (skipped when max_iterations is unset, since the probe has no known endpoint, or when the expression references previous, whose value depends on results). At runtime, a step that still cannot be computed ends that probe with the new stop_reason value step_error and logs the variable, iteration, and previous value, leaving the other probes to finish and report normally. The same handling now covers a step_expr whose result cannot be cast to the variable’s type.

• iops report now honors reporting.output_dir and reporting.output_filename from the report config (whether passed via --report-config or auto-detected as report_config.yaml in the run directory). Previously these fields were ignored on the CLI path and the report was always written to <workdir>/analysis_report.html. The precedence is now: explicit --output flag, then the config’s output_dir/output_filename, then the default <workdir>/analysis_report.html.

• A report with reporting.gallery.max_width set no longer fails with name 'logging' is not defined when Pillow is missing. The gallery’s two fallback paths, the notice that Pillow is unavailable and the notice that an image exceeds the 25 MB embedding cap, called logging.getLogger() in a module that never imported logging, so the branch meant to degrade gracefully raised NameError instead. Report generation aborted before analysis_report.html was written, which read as a broken gallery even though the images and folder layout were correct. Setting max_width was the only way to reach the first path, and Pillow is not installed by default, so any run combining the two hit it. The report now renders with images embedded at full size and a warning naming the missing package.

• A SLURM execution whose parser failed is no longer reported as succeeded. When a job exited cleanly but parsing its output then failed, the success path ran the parser and discarded the result, leaving the status SUCCEEDED. The runner caches on that status, so the execution was written to the cache with no metrics and a later run with --use-cache skipped it as already done. A parser failure on the success path now downgrades the status to FAILED, matching the local executor, so the execution is neither reported as succeeded nor cached. This applies to the SLURM and single-allocation executors.

• GPU metrics are no longer lost when a device does not expose one of the sampled fields. nvidia-smi reports [N/A] for fields a device does not provide, such as the memory fields on unified-memory parts, and every field of a sample was parsed inside one try block, so a single unparsable field discarded the whole row including the valid power, temperature, and utilization readings. With every row dropped, __iops_resource_summary.csv carried only gpu_count and gpu_samples_collected. Fields are now parsed independently and a metric is emitted only when the device reported it. Unreported fields are omitted rather than written as zero, so a missing measurement is not mistaken for an idle device; only the timestamp remains mandatory, since it anchors the sample on the power and duration timeline.

[3.5.8] - 2026-06-20

Added

• bayesian_config.max_retries (default 10) controls how many times per iteration the optimizer is re-asked when its suggestion maps to an already-visited point before falling back to random sampling from unvisited points. Each retry refits the surrogate model, so on large n_iterations runs - where late iterations almost always collide with visited points - the retries dominated wall time (especially with the GP estimator, whose fit cost grows with the observation count). Lowering max_retries (e.g. to 2-3) cuts that per-iteration cost without changing study semantics; the previous hardcoded value of 10 remains the default.

Fixed

CLI, watch mode, find

• iops find --watch pause (p) now truly freezes the view. The pause key only disabled row reordering while the periodic refresh kept reloading underneath the user, so rows appeared, completed tests were hidden, and the scroll position jumped, making pause look broken. Pausing now stops the background refresh entirely (no filesystem I/O while paused) and resuming forces an immediate refresh.
• Watch mode now collects data on a dedicated background thread, so a slow scan no longer blocks rendering or input. Keypresses, pausing, scrolling, and Ctrl+C stay responsive on large parameter spaces and on parallel/network filesystems regardless of how long a refresh takes.
• Watch mode keypresses (q, p, scrolling) no longer become unresponsive on large runs. The render loop serviced data refreshes before the keyboard, so once a frame took longer than the refresh interval the view redrew continuously without ever polling for input. Input is now drained first, every frame.
• Watch mode no longer recomputes per-test aggregates on every frame. The core-hours total compiled and rendered a Jinja expression once per test per redraw, costing seconds per frame on runs with thousands of executions (the dominant cause of an unresponsive UI). The compiled expression is now cached, the result is memoized per distinct parameter set, and all per-refresh aggregates are computed once on the collector thread rather than per frame.
• Watch mode no longer scans every not-yet-started execution’s folder on each refresh. The run-root status roll-up is now authoritative for coverage: when it is present, executions it does not list are treated as pending without a per-folder stat or directory glob. Previously a large upfront run could scan thousands of folders every interval despite the roll-up; the folder-scan fallback now runs only when no roll-up exists.

Cache

• The execution cache no longer fails with sqlite3.OperationalError: disk I/O error on NFS and HPC filesystems (Lustre, GPFS). When the cache file lives on such a filesystem, IOPS now opens it through SQLite’s lock-less unix-none VFS and keeps the rollback journal in memory, so no POSIX fcntl() locks are taken and no on-disk journal file is created. The previous NFS-compatible mode only set journal_mode=DELETE, which still relied on advisory locking that these filesystems often disable, so the first write (table creation) aborted the run. This is safe because IOPS uses single-process access to the cache.

Reporting

• HTML report generation no longer crashes when a swept variable is named count, mean, or std. The best-configurations section renamed grouped columns to the bare variable names and then appended those aggregation labels, colliding with the variable name and producing a duplicate-label reindex error. Aggregation columns now use sentinel names to avoid the clash.

Configuration

• Jinja2 templates in sweep values and conditional when/default values are now resolved instead of leaking through as literal strings. A value such as {{ os_env.OUTPUT }}/ost_{{ ost_count }} was previously stored verbatim and only type-cast, so it reached params, commands, and input files unrendered; conditional variables (which require sweep/when/default and cannot use expr) had no supported way to template their values. Templated string base variables are now rendered with the full context (os_env plus previously resolved vars) in config order and re-cast to the declared type. Unset references raise a clear StrictUndefined error instead of emitting the literal.

[3.5.7] - 2026-06-11

Added

• Bayesian optimization now caps the BLAS/OpenMP threads used by the Gaussian Process surrogate fit (default 1, override with IOPS_BLAS_THREADS). Previously numpy/BLAS spawned one thread per core for every fit, so on a many-core (often shared login) node a single iops run appeared as dozens of threads of pure overhead. The cap does not change the optimizer’s suggestions, only the threading of the underlying linear algebra.
• iops find and watch mode now read a single run-root __iops_status_rollup.json aggregate instead of scanning every exec_XXXX/repetition_N/__iops_status.json on each refresh. This keeps them responsive on large parameter spaces (thousands of executions) and on parallel/network filesystems, where per-folder reads dominate wall-clock. The runner maintains the roll-up during the run and marks it complete when finished; watch always reads it, while iops find trusts it only when complete and otherwise falls back to the folder scan. The per-folder status files remain the source of truth, so older runs and archives keep working.
• Per-execution image gallery (reporting.gallery) for embedding simulation thumbnails and other per-test images into the self-contained HTML report
  • Thumbnail grid grouped by execution with click-to-enlarge; images are base64-embedded so the report has no external file dependencies
  • Two discovery methods that can be combined: convention folder (auto-scans <execution_dir>/<gallery.folder> for files matching pattern) and explicit sources (Jinja2-templated paths resolved per execution, glob characters honored)
  • New built-in template variable {{ artifacts_dir }} resolves to <execution_dir>/<gallery.folder> (default <execution_dir>/images); use it in script_template to write images without hardcoding the folder name
  • max_width option (requires Pillow) downscales wide images before embedding; degrades gracefully without Pillow
  • caption_vars controls which parameters appear as the caption under each execution’s card (defaults to report_vars)
  • Controlled by reporting.sections.gallery (default true); section auto-enables when images are found
• Software version capture probe (benchmark.probes.versions) for recording software and library versions as metadata once per execution
  • Mapping of component name to shell command (e.g. app: "myapp --version"); IOPS injects __iops_atexit_versions.sh and captures versions after the benchmark body via the exit handler, so version tools made available by the benchmark’s own module load commands are in scope
  • Failing commands record an empty string rather than aborting the run
  • Writes __iops_versions.json to each repetition directory, and surfaces the captured versions as version.<component> columns in the results sink (CSV/Parquet/SQLite) so they can be queried alongside metrics; exclude with output.sink.exclude: [version.*]
  • HTML report renders a Software Versions section with a per-execution table; a prominent drift warning is shown when any component reports more than one distinct value across executions (this is the cache-mixing detector: it catches studies that mix freshly executed results with older cached results from a different software environment)
  • Controlled by reporting.sections.versions (default true); section auto-enables when version data is present
• iops cache create subcommand to build a cache database from a CSV file
  • --params and --metrics map CSV columns to parameters (the cache key) and metrics; each row becomes one cached execution
  • Cell values are coerced to int/float/bool where possible so they hash the same way IOPS normalizes parameters at run time
  • --repetition-column uses an existing column as the repetition number; otherwise repetitions are auto-numbered per unique parameter set
  • Shows a progress bar while writing entries (requires the watch extra for rich); disable with --no-progress
  • Imported entries are stored with SUCCEEDED status and work with cache list|show|stats and iops run --use-cache
• Execution Status breakdown in HTML reports, showing the count and percentage of executions by status (SUCCEEDED, FAILED, ERROR, SKIPPED, RUNNING, PENDING). Counts come from the run index and per-execution status files, so failures and skips are reported even though the results table only contains successful executions. A success-rate summary is also shown in the Execution Overview when not every execution succeeded.
• Log the execution id and repetition before each submission (e.g. Submitting 684 (rep 1/3)), so the execution can be correlated with the executor’s submission output such as the SLURM job id.
• Highlight the actively running row in iops find --watch with a background color (in addition to the existing ▶ marker), making the in-progress execution easier to spot.

Changed

• iops archive extract without -o now extracts into a folder named after the archive (e.g. study.tar.gz -> ./study/) instead of scattering files into the current directory. Pass -o PATH to choose a different destination.

Fixed

Search and planning

• bayesian_config.objective now defaults to minimize as documented. The loader previously defaulted to maximize, silently optimizing in the wrong direction for configurations that omitted the field. If you relied on the undocumented default, set objective: maximize explicitly.
• The end-of-run “Best parameters found” summary now reports the actual best point for objective: minimize studies. The best-value comparison was inverted, so the summary tracked the last evaluated point instead of the best one (the optimizer itself was unaffected).
• Bayesian search no longer crashes on string (categorical) sweep variables: the nearest-valid-point mapping converted every dimension with float(), so any type: str variable with more than one value failed on the first iteration. Categorical values are now mapped to category indices for distance computation.
• Single-allocation (kickoff) mode now honors random sampling. Previously search_method: random with allocation.mode: single silently prepared and ran the entire parameter space, ignoring n_samples/percentage.
• search_method: adaptive is now rejected with allocation.mode: single (same as bayesian) instead of crashing at run start; adaptive probing cannot work in a pre-generated script.
• Constraints that reference an adaptive variable no longer abort the run during matrix generation with an undefined-variable error. They are deferred to per-probe evaluation, where the adaptive value is in scope.
• The Bayesian planner now hands out per-repetition clones of execution instances instead of mutating one shared object. Previously resource-sampling summaries collapsed to the last repetition per execution, and dry-run accounting could attribute results to the wrong execution id.
• When the Bayesian optimizer suggests an already-visited configuration, the planner now re-tells the optimizer that point’s known outcome so the next suggestion actually changes; previously all retries returned the identical point and the iteration silently degraded to random sampling or triggered spurious early convergence.
• An explicit random_seed: null no longer crashes the Bayesian convergence fallback.

Execution runtime

• Parser output capture is now serialized across worker threads. With parallel > 1, the process-wide stdout/stderr redirection could cross-contaminate parser_stdout/parser_stderr files between tests and leave the process writing to an abandoned buffer.
• IOPS metadata files (__iops_index.json, __iops_status.json, __iops_params.json, run metadata) are now written atomically (temp file plus rename). Concurrent readers such as iops find and watch mode could previously observe truncated JSON mid-rewrite and fail or flicker.
• The local executor streams benchmark stdout/stderr directly to files instead of buffering everything in memory: verbose benchmarks no longer risk exhausting RAM, and output captured before an interrupt is preserved.
• A typo in benchmark.cores_expr now fails fast with a clear error instead of silently counting every test as 1 core, which effectively disabled max_core_hours budget enforcement. The expression is also evaluated without builtins.
• Fixed two Ctrl+C races: the interrupt handler could deadlock on the job-tracking lock, and an interrupt arriving between the shutdown check and a thread-pool submission turned a clean shutdown into a traceback.
• Dry-run reports for planners that reuse test objects (Bayesian) now snapshot execution ids and variables at collection time, fixing unique-test counts and cache-hit accounting.

Configuration loading

• Config blocks with a null or wrong-typed body (sweep:, adaptive:, vars:, scripts:, command:, parser:, post:, constraints:, machines:, and all reporting sub-blocks) now produce field-located validation errors instead of raw AttributeError/TypeError crashes. Commenting out a block body while leaving the key is the common trigger.
• --machine/IOPS_MACHINE with no machines section in the config now produces the intended friendly error instead of KeyError: 'machines'.
• cache_file: null no longer crashes; an empty cache_file is rejected with a clear error instead of resolving to the current directory. Templated cache_file paths now render strictly: an undefined variable raises a field-located error instead of silently producing a wrong path like .db.
• One-line script templates containing Jinja expressions (e.g. bash {{ workdir }}/run.sh) are no longer misclassified as file paths and rejected with “file was not found”.
• output.sink.include now emits a warning that it is not supported instead of being silently ignored.
• random_config.percentage values above 1.0 are rejected with an explanation that percentage is a 0-1 fraction; previously percentage: 50 was silently clamped to 1.0, meaning 100% sampling.
• Non-string command.template/script_template values now produce the intended “must be a string, got int” hint instead of an AttributeError.
• A script without a name now fails with scripts[i].name is required instead of a KeyError.
• Machine overrides that replace a var’s sweep with expr now also clear inherited when/default, fixing a confusing validation failure for a reasonable override.
• Duplicate names within a machine-override list (scripts, constraints) now raise an error instead of silently dropping the second entry.
• benchmark.workdir no longer needs to pre-exist; it is created on first run as the (previously unreachable) creation logic always intended.

CLI, watch mode, find

• Bare iops archive and iops cache print the intended “No subcommand specified” help instead of crashing with an AttributeError.
• iops find --interval values below 1 are rejected; an interval of 0 previously busy-spun the CPU and disabled all keyboard controls.
• PageUp/PageDown now work in watch mode: their 4-byte escape sequences were truncated to 3 bytes and could never match, leaving a stray ~ consumed as a spurious keypress.
• The / go-to-test search in watch mode jumps to the correct row when completed tests are hidden (auto-enabled for runs with more than 20 tests).
• Watch mode no longer evaluates cores_expr text from metadata files with eval(); values are parsed as plain numbers.
• A held or repeated keypress can no longer starve the watch-mode data refresh; refresh is driven by a monotonic deadline.
• The /dev/tty keyboard fallback now also engages when stdin is closed or unusable, not only when it is a non-TTY.
• An interrupt arriving before the first watch refresh no longer raises UnboundLocalError.
• iops find parameter filters now match numerically and case-insensitively for booleans: nodes=4 matches a stored 4.0, flag=true matches True, 1e3 matches 1000.0. The same matching is used by watch-mode filters.
• Log messages containing | (such as per-test metric summaries) no longer have their first part duplicated as a prefix on wrapped or multi-line console output.

Results sinks

• CSV results sink: extending the schema with a new column (e.g. a metric that first appears mid-run) no longer rewrites existing rows through pandas type inference. Previously the rewrite silently mutated stored values, such as "0010" becoming 10 and integer columns becoming floats. Existing rows are now preserved verbatim. A zero-byte results file left behind by an interrupted run no longer makes every subsequent write fail.
• SQLite results sink: rows that introduce new columns no longer abort the run with “table has no column named …”. The table schema is extended with ALTER TABLE ADD COLUMN instead.
• Parquet results sink: appending a row that lacks a value for an existing integer column no longer permanently upcasts the column to float; integer columns are preserved as nullable integers.

Execution backends

• SLURM executor: a transient squeue failure (e.g. slurm_load_jobs error: Socket timed out) is no longer interpreted as “job left the queue”. Previously one controller hiccup abandoned a live job: polling stopped, the job was removed from Ctrl+C cleanup tracking, and a still-running state was recorded as final. Status polling now retries on transient errors and confirms job completion via scontrol before finalizing.
• SLURM single-allocation mode: a transient squeue failure no longer fails the current test and poisons every remaining test while the allocation keeps running on the cluster. The allocation state check is also throttled to poll_interval (previously squeue was invoked twice per second for the entire run).
• SLURM executor: job ids are now parsed from multi-cluster submission output (Submitted batch job 12345 on cluster c2). Previously the job was marked as an error and ran untracked on the cluster.
• SLURM executor: DEADLINE, REVOKED, and SPECIAL_EXIT are now recognized as terminal failure states. A deadline-killed job could previously be recorded as SUCCEEDED when its partial output happened to parse.
• SLURM executor: the job start timestamp is now recorded when a job is already running at the first poll, fixing wrong queue-wait/run-time splits for jobs that start immediately.

Reporting

• Total core-hours in HTML reports paired each row with the wrong duration (or silently skipped rows) whenever any execution had been filtered out as failed; rows and durations are now aligned positionally.
• Benchmark name, description, variable values, hostnames, and probe-collected system strings are now HTML-escaped in report sections that previously interpolated them raw, so a value containing markup cannot break or inject into the report.
• Custom plot configurations survive report regeneration: the serialized template now includes row_vars, col_var, aggregation, sort options, the bayesian_parameter_evolution and resource_sampling section flags, and best_results.min_samples, all of which were previously dropped.
• Scatter plots with a string-typed variable as y-axis or color no longer fail; non-numeric color columns are encoded with labeled colorbars.
• The variable impact plot no longer disappears when all impacts are zero.
• Report generation no longer fails when merging resource sampling metrics for runs that use integer execution ids. The results dataframe stores execution.execution_id as integers while __iops_resource_summary.csv stores zero-padded folder-style ids (exec_0002), so the join raised “You are trying to merge on int64 and object columns”. Both join keys are now normalized to the numeric execution id before merging.

Cache

• The repetition count for cached parameter sets now counts distinct repetitions, fixing over-counting on caches built with iops cache rebuild (which may hold several rows per repetition).
• The objective-metric cache lookup orders entries by creation time, making the fallback deterministic, and no longer crashes on string-serialized metric values.
• Fixed a connection leak in the cache’s locked-database retry path, and iops cache rebuild now closes its database connections.
• Execution cache: numeric-looking strings are only coerced for hashing when the conversion round-trips exactly, so distinct string parameters such as "1.1" and "1.10" (version strings) or "1e3" no longer collapse into the same cache key and return each other’s results as false cache hits. Negative integer strings now hash like their native integer counterparts.
• iops cache create: caches built from CSV now store the executor status under the key the runner actually reads, so cache hits report SUCCEEDED instead of UNKNOWN in status files, results, and iops find --status filtering. CSV cell coercion now matches runtime parameter normalization exactly (previously negative integers never produced cache hits).

Archive

• iops archive: partial archives (--status, --params, --min-completed-reps) no longer fail their own integrity verification on extraction. Checksums are now computed over the content actually stored in the archive (filtered index and included executions) instead of the unfiltered source tree.
• iops archive create --min-completed-reps: filtered result files are no longer empty. Repetition folder numbers (1-based) were compared against result rows using a 0-based index, so completed repetitions almost never matched.
• iops archive extract: extraction now applies the standard library’s data filter, which rejects symlinks and hard links pointing outside the destination directory, absolute paths, and device nodes. The previous custom filter only inspected member names, leaving a path traversal vector via crafted symlink members, and it also skipped legitimate files whose names merely contained .. (e.g. results..csv).

[3.5.5] - 2026-05-30

This release introduces declarative input files, letting benchmarks that read parameters from disk (configuration files, problem decks, job descriptions) generate those files from the execution context instead of relying on the command template alone. It also gathers a set of reliability fixes around interruption handling, report accuracy, system probing, and live monitoring, making long campaigns on shared clusters more robust.

Added

• Declarative input files via scripts[].inputs for parameter files that the benchmark reads from disk
  • Each entry declares a required name, required path (Jinja2-rendered destination), template (inline) or file (external path), and optional mode (octal string applied with chmod)
  • Files are rendered with the full execution context (vars, os_env, execution_id, repetition, etc.) and written at preparation time, so the exact input used for each run remains on disk even if the script aborts
  • Rendered paths are exposed in script_template, command.template, and post.script via {{ inputs.<name>.path }}
  • Names must be valid Python identifiers and unique within a script; Jinja2 syntax in template and path is validated at load time
  • Purely additive: existing configs without inputs: are unaffected

Fixed

• Best Configurations table in HTML reports now shows the exact command recorded for each execution. It previously re-rendered the command template from the swept variables alone, so templates referencing runtime-only values such as {{ inputs.<name>.path }} could not be reconstructed and rendered as [Error rendering command: ...]. The command is now looked up from __iops_index.json by execution id (normalized so the integer id in the results dataframe matches the zero-padded id in the index), falling back to template rendering only when no index is available.
• Ctrl+C no longer submits a new SLURM job during shutdown. The SIGINT handler set self.interrupted and canceled the in-flight job, but neither _run_sequential nor _run_parallel checked that flag, so the loop fetched the next test from the planner and submitted it before exiting. Both loops now guard at the top of the iteration and again right before _execute_and_cache / pool.submit, closing the race window between next_test() and submission. A single Ctrl+C is now sufficient to stop the run cleanly.
• System probe no longer emits invalid __iops_sysinfo.json on CPU-only nodes. The GPU detection block trusted command -v nvidia-smi alone, but on nodes where the binary exists yet cannot reach a driver, nvidia-smi printed an error into the integer GPU fields, corrupting the JSON and triggering a parse warning during result collection. The probe now checks nvidia-smi -L first, strips newlines from all GPU fields, and validates that gpu_count and gpu_memory_mib are digits before emitting them.
• Watch mode no longer hides executions beyond the planner’s estimate. In dynamic mode the live table bounded its display range by total_expected, so when a campaign produced more executions than estimated (common with adaptive and Bayesian probing) the extra executions existed in the data but were never rendered. The display range now extends to cover the highest existing execution id, while queued placeholders still fill up to the estimate when it is higher.

[3.5.4] - 2026-04-28

Added

• Parallel test execution via benchmark.parallel config field and --parallel N CLI flag
  • Run multiple tests concurrently using a thread pool (works with both local and SLURM executors)
  • Planner-aware: Exhaustive and Random support unlimited parallelism, Bayesian is capped to 1 (sequential), Adaptive parallelizes across independent probes
  • Thread-safe budget tracking, result writing, cache access, and SLURM job management
  • Incompatible with single-allocation mode (ignored with a warning)
• BasePlanner.max_parallel() and BasePlanner.next_tests(n) methods for planner parallelism negotiation
• YAML configuration section in HTML reports, showing the original config in a collapsible block
• Signal handler registration for all executors (previously SLURM-only)
• iops cache list, iops cache show, and iops cache stats subcommands for inspecting cache databases
  • list collapses entries per unique parameter hash, with averaged metrics across repetitions and VAR=VALUE filtering (same syntax as iops find)
  • show accepts a git-style hash prefix and prints every repetition’s metrics and metadata
  • stats prints totals, unique parameter sets, and date range
  • All three support --json for scripting
• Public helpers in iops.cache for programmatic inspection: list_cache_entries, get_cache_entry, get_cache_stats, resolve_hash_prefix

Fixed

• Report generation now works after Ctrl+C interruption. Metadata file is written at the start of execution with static fields and updated with dynamic results in a finally block, so partial runs always produce a usable metadata file.
• ExhaustivePlanner.next_tests(n) no longer returns aliased ExecutionInstance objects when the same matrix entry is picked for different repetitions in the same batch. Each returned test is now a shallow copy with its own metadata dict, so _prepare_execution_artifacts cannot overwrite an earlier batch entry’s execution_dir / repetition / script_file. This prevented duplicate SLURM submissions pointing to the same WorkDir under benchmark.parallel > 1.
• Resource and GPU samplers now isolate concurrent SLURM attempts on the same exec_dir. Trace files and the sentinel file are now suffixed with the SLURM job id (or the shell PID when running locally), so a requeued second attempt cannot truncate the first attempt’s trace or stop its sampler when it exits.
• Per-GPU columns in __iops_resource_summary.csv now include the hostname when traces span multiple nodes (node01_gpu0_*, node02_gpu0_*), so same-indexed GPUs on different hosts no longer overwrite each other. Single-node runs keep the original gpuN_* naming.

[3.5.3] - 2026-03-31

This release introduces GPU monitoring support to the IOPS probe system, enabling energy consumption tracking, power draw analysis, and thermal monitoring during benchmark execution. It also adds a new Resource Sampling section to HTML reports and a JUBE-to-IOPS configuration converter.

Added

• GPU sampling probe (probes.gpu_sampling) for real-time GPU monitoring during benchmark execution
  • Tracks utilization, memory usage, temperature, power draw, and clock speeds
  • Supports NVIDIA GPUs via nvidia-smi, with vendor detection designed for future AMD/Intel extension
  • Gracefully skips when no supported GPU is detected (no errors, no empty files)
  • Per-node GPU sample files (__iops_gpu_trace_<hostname>.csv) with SLURM multi-node support
  • Energy consumption calculation via trapezoidal integration of power over time (gpu_energy_j)
  • Per-GPU metric columns (gpu0_avg_power_w, gpu1_energy_j, etc.) for multi-GPU analysis
  • Aggregated GPU metrics in __iops_resource_summary.csv alongside CPU/memory metrics
  • GPU hardware detection in system snapshot (gpu_count, gpu_model, gpu_driver, gpu_memory_mib)
• Resource Sampling section in HTML reports (reporting.sections.resource_sampling)
  • Auto-detects available CPU/memory and GPU metrics from __iops_resource_summary.csv
  • Summary table with min/max/mean for each resource metric
  • Heatmap and bar chart visualizations correlating user variables with resource footprint
  • Enabled by default when resource summary data is available
• iops convert command for translating JUBE XML benchmarks to IOPS YAML format [experimental]
• Jinja2 templating support in benchmark.cache_file
• examples/gpu_stress/ synthetic CUDA SGEMM workload for testing GPU probes

Fixed

• GPU sampler CSV parsing for GPU names containing spaces (e.g. “Tesla V100-PCIE-16GB”)
• GPU aggregate metrics no longer diluted by idle GPUs on multi-GPU machines

[3.5.2] - 2026-03-02

Added

• Adaptive variable probing (search_method: "adaptive") for finding threshold values automatically
  • Supports multiplicative (factor), additive (increment), and custom (step_expr) progression
  • Configurable stop conditions via stop_when expressions (exit code, metrics, execution time)
  • Independent probes per swept variable combination
  • Probe results stored in __iops_run_metadata.json under adaptive_results
• list variable type for indexed access to correlated parameter arrays
• --add VAR[:TYPE]=VALUE option for iops cache rebuild to add typed variables to all cache entries
• machines section for per-machine config overrides with deep-merge semantics
• --machine NAME flag (and IOPS_MACHINE env var) for iops run and iops check
• iops check --resolve [FILE] to output the fully merged config as YAML
• iops generate --machines to scaffold a config with machine override examples
• metrics global variable in parser scripts (list of expected metric names for selective computation)
• Copy input YAML config to run folder for reproducibility
• Reports section on the website with interactive Plotly report viewer

Changed

• Made scikit-optimize an optional dependency, installed via pip install iops-benchmark[bayesian]

Fixed

• --resolve now renders multi-line strings (parser scripts, script templates) with YAML literal block style (|) instead of escaped single-line format
• Watch mode progress display for adaptive planner
• Parser script now accepts directory paths
• Bayesian test failures when scikit-optimize is not installed

[3.5.0] - 2026-02-01

Added

• Resource tracing for CPU and memory monitoring (trace_resources, trace_interval)
• Single-allocation mode for batch SLURM execution (slurm_options.allocation) [experimental]
• MPI block for automatic MPI launching in single-allocation mode (scripts[].mpi)
• iops cache rebuild command to exclude variables retroactively
• Parser script context injection (vars, env, os_env, execution_id, repetition globals)
• os_env context variable exposing system environment variables to Jinja2 templates
• --partial and --min-reps flags for partial archive creation
• Unknown key validation with “did you mean?” suggestions for YAML config
• Core-hours tracking for cache hits in dry-run estimates
• Boolean variables now included in report generation by default (treated as 0/1)
• Plot export to image files (--export-plots, --plot-format) with pdf, png, svg, jpg, webp support
• Random search evolution section in HTML reports
• NFS auto-detection with lock-free SQLite mode for cache
• Real-time execution status tracking with executor-specific updates
• early_stop_on_convergence option for Bayesian optimization to stop when optimizer converges
• convergence_patience option to control early stopping sensitivity (default: 3)
• xi_boost_factor option to dynamically increase exploration when stuck (default: 5.0)
• --cache-only CLI option for cache-only execution (skip tests not in cache)
• Keyboard navigation for watch mode (pause, page scroll, search by test ID)

Changed

• Renamed executor_options to slurm_options (old name deprecated, remove in 3.7.0)
• Refactored probe configuration to nested probes: section with clearer field names:
  • collect_system_info → probes.system_snapshot
  • track_executions → probes.execution_index
  • trace_resources → probes.resource_sampling
  • trace_interval → probes.sampling_interval (old names deprecated, remove in 3.7.0)
• Renamed [pdf] optional dependency to [plots] (supports pdf, png, svg, jpg, webp)
• Removed scripts[].submit field
• Removed output.sink.include option
• Removed Pareto Frontier analysis from reports
• iops generate now defaults to SLURM executor (use --local for local)
• Separate user labels from IOPS internal metadata in output
• Default output path when output.sink.path not specified
• Bayesian optimization now uses MAX/MIN aggregation for repetitions (matching objective) instead of MEAN

Fixed

• sections.parallel_coordinates and sections.variable_impact settings being ignored in reports
• Dry-run cache lookup using wrong repetition values
• Constraint evaluation order (swept vars before derived expressions)
• Refactored BayesianPlanner to use pre-built execution matrix (consistent with other planners)
• SQLite cache locking errors on NFS filesystems
• Bayesian optimization nearest-neighbor tie-breaking now deterministic (prefers higher parameter values)

[3.4.0] - 2026-01-13

Added

• Subcommand-based CLI structure (run, check, generate, report, find)
• find command to explore execution folders with parameter filtering
• Execution status tracking with --status filter (SUCCEEDED, FAILED, ERROR)
• Archive module for workdir portability (iops archive create/extract)
• Watch mode for real-time execution monitoring (iops watch)
• Conditional variables with when/default fields
• Enhanced Bayesian optimization with base_estimator, xi, kappa, n_iterations
• Search space efficiency statistics in Bayesian reports
• fallback_to_exhaustive option for Bayesian planner
• create_folders_upfront benchmark option
• System info collection and config validation
• Client-side search for documentation site

Changed

• YAML file shorthand: iops config.yaml works as iops run config.yaml
• Renamed sqlite_db to cache_file
• Renamed target_metric to objective_metric
• Removed rounds feature and simplified planner architecture
• Made pyarrow an optional dependency
• Portable workdir metadata with relative paths
• Centralized validation logic in loader.py
• Restructured documentation for v3

Fixed

• Bayesian planner edge case bugs
• Archive progress bar performance with compression level tuning
• Metrics not showing when using --hide in watch mode
• Duration tracking to use cached sysinfo
• Search index to include full page content

[3.2.0] - 2025-12

Major overhaul transforming IOPS into a generic benchmark orchestration framework.

• Generic framework supporting any parametric experiment
• iops generate to create configuration templates
• Interactive setup wizard for guided configuration
• Bayesian optimization for intelligent parameter search
• Random sampling planner for parameter space exploration
• Core-hours budget tracking for HPC clusters
• Dry-run mode to preview executions with resource estimates
• HTML report generation with interactive Plotly charts
• Result caching with --use-cache flag
• SLURM cluster support with automatic job management
• Parameter constraints to filter invalid configurations
• Configurable SLURM commands via slurm_options
• exhaustive_vars for hybrid search strategies
• User-configurable reporting system with 8 plot types
• reporting configuration section for plot customization
• --report-config CLI option to regenerate reports
• Theme configuration (colors, fonts, plotly styles)
• PyPI packaging as iops-benchmark
• Spack package support

[2.0.0] - 2024

Internal development version that introduced architectural changes leading to 3.2.0.

• Database-backed storage for results
• Improved executor architecture with better SLURM support
• Enhanced test coverage and validation
• Refactored planner with greedy strategy and caching

[1.0.0] - 2023-10-19

Initial release of IOPS as an IOR-specific benchmark orchestration tool.

• IOR benchmark automation with parametric sweeps
• SLURM and local execution support
• Multi-round optimization with binary search heuristics
• HTML report generation with bandwidth graphs
• Configuration via INI files
• Stripe folder management for parallel filesystems
• Test repetitions for statistical validity
• Progress tracking and interruption handling