Developer Guide on IOPS

Execution Loop

Mon, 01 Jan 0001 00:00:00 +0000

This page documents the IOPS execution loop architecture: how the Runner, Planner, and Executor components orchestrate benchmark execution.

Overview

The execution loop follows a pull-based design where the Runner requests tests from the Planner and delegates execution to the Executor:

┌─────────────────────────────────────────────────────────────────┐
│ IOPSRunner │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Planner │─────►│ Runner │─────►│ Executor │ │
│ │ │ │ Loop │ │ │ │
│ │ next_test() │ │ │ │ submit() │ │
│ │ │◄─────│ │◄─────│ wait_and_ │ │
│ │ record_ │ │ │ │ collect() │ │
│ │ completed() │ │ │ │ │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Matrix │ │ Cache │ │ Parser │ │
│ │ Builder │ │ │ │ │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────────┘

Component Roles

Component	Responsibility
Runner	Orchestrates the execution loop, manages cache, tracks budget, writes results
Planner	Generates test configurations, manages execution order, prepares artifacts
Executor	Submits jobs, waits for completion, collects output and metrics
Matrix Builder	Generates parameter combinations from config, applies constraints
Cache	Stores/retrieves results to skip redundant executions
Parser	Extracts metrics from benchmark output files

The Execution Loop

Main Loop (`IOPSRunner.run()`)

def run(self):
 while True:
 # 1. Check budget
 if budget_exceeded:
 break

 # 2. Get next test from planner
 test = self.planner.next_test()
 if test is None:
 break # No more tests

 # 3. Check cache
 if self.cache and self.use_cache_reads:
 cached_result = self.cache.get_cached_result(test.vars, test.repetition)
 if cached_result:
 # Use cached result, skip execution
 test.metadata.update(cached_result)
 test.metadata['__cached'] = True

 # 4. Execute if not cached
 if not used_cache:
 self._write_status_file(test, status="RUNNING")
 self.executor.submit(test)
 self.executor.wait_and_collect(test)

 # Store in cache if succeeded
 if succeeded:
 self.cache.store_result(...)

 # 5. Write status and results
 self._write_status_file(test)
 save_test_execution(test)

 # 6. Notify planner of completion
 self.planner.record_completed_test(test)

Sequence Diagram

Runner Planner Executor Cache
 │ │ │ │
 │ next_test() │ │ │
 │───────────────────►│ │ │
 │ │ │ │
 │ ExecutionInstance │ │ │
 │◄───────────────────│ │ │
 │ │ │ │
 │ get_cached_result() │ │
 │──────────────────────────────────────────────────────────►│
 │ │ │ │
 │ (cache miss) │ │ │
 │◄──────────────────────────────────────────────────────────│
 │ │ │ │
 │ submit(test) │ │ │
 │────────────────────────────────────────►│ │
 │ │ │ │
 │ wait_and_collect(test) │ │
 │────────────────────────────────────────►│ │
 │ │ │ │
 │ test.metadata updated │ │
 │◄────────────────────────────────────────│ │
 │ │ │ │
 │ store_result() │ │ │
 │──────────────────────────────────────────────────────────►│
 │ │ │ │
 │ record_completed_test() │ │
 │───────────────────►│ │ │
 │ │ │ │

Planner Architecture

Base Planner

All planners inherit from BasePlanner and implement:

Probe System

Mon, 01 Jan 0001 00:00:00 +0000

This page documents the IOPS system probe mechanism for developers who want to extend the collected metrics or modify probe behavior.

Overview

The probe system collects system information (CPU, memory, filesystems, etc.) from the actual compute node where the benchmark runs. This matters for SLURM jobs, where the login node differs from compute nodes.

Key files:

__iops_exit_handler.sh - Centralized EXIT trap coordinator
__iops_node_launcher.sh - Resolves the job’s node list and starts one helper per node
__iops_atexit_sysinfo.sh - System info collection script
__iops_sysinfo.json - Collected data (generated at runtime)

Architecture

User Script Exit Handler Sysinfo Script
┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
│ #!/bin/bash │ │ # Sets EXIT trap │ │ # Registers cleanup │
│ #SBATCH ... │ │ trap '_iops_run_ │ │ _iops_register_exit │
│ │ │ exit_actions' EXIT│ │ "_iops_collect_ │
│ # IOPS injection │ │ │ │ sysinfo" │
│ source handler.sh │───────►│ # Registry array │ │ │
│ source sysinfo.sh │───────────────────────────────────────►│ # Collection func │
│ │ │ _iops_register_exit │ │ _iops_collect_ │
│ # User's benchmark │ │ () { ... } │ │ sysinfo() { ... } │
│ mpirun ./benchmark │ └─────────────────────┘ └─────────────────────┘
└─────────────────────┘ │
 │ │
 ▼ (on script exit) ▼
┌─────────────────────┐ ┌─────────────────────┐
│ __iops_sysinfo.json │◄───────│ _iops_run_exit_ │
│ { │ │ actions() calls │
│ "hostname": ..., │ │ all registered │
│ "cpu_model": ..., │ │ functions │
│ "duration": ... │ └─────────────────────┘
│ } │
└─────────────────────┘

The exit handler provides a centralized EXIT trap that other IOPS features (system probe, resource sampler) register cleanup actions with, avoiding trap conflicts.

Data Sources

Mon, 01 Jan 0001 00:00:00 +0000

This page documents where IOPS CLI commands read their data.

Overview

IOPS commands read from result files and from metadata files generated during execution. Metadata files have the __iops_ prefix and are documented in the Metadata Files user guide.

Data Sources by Command

iops find ◄─── __iops_index.json
 ◄─── __iops_status_rollup.json (preferred; falls back to per-folder scan)
 ◄─── __iops_status.json (fallback)
 ◄─── __iops_run_metadata.json (optional metadata)

iops find --watch ◄─── __iops_index.json (polling)
 ◄─── __iops_status_rollup.json (preferred; polling)
 ◄─── __iops_status.json (fallback scan)

iops report ◄─── __iops_run_metadata.json
 ◄─── results.csv/parquet/sqlite

Status roll-up fast path

Both iops find and watch mode read the run-root __iops_status_rollup.json aggregate instead of scanning every execution’s repetition folders, which keeps them responsive on large parameter spaces. The per-folder __iops_status.json files remain the source of truth (and the node-to-runner channel in single-allocation mode); the roll-up is an accelerator. Watch always prefers it; iops find only trusts it when the run marked it complete, otherwise it falls back to the folder scan. When the roll-up is absent (older runs, archives), both fall back automatically.

Developer Guide on IOPS

Execution Loop

Overview

Component Roles

The Execution Loop

Main Loop (IOPSRunner.run())

Sequence Diagram

Planner Architecture

Base Planner

Probe System

Overview

Architecture

Data Sources

Overview

Data Sources by Command

Status roll-up fast path

Main Loop (`IOPSRunner.run()`)