Data Sources

This page documents where IOPS CLI commands read their data.

Overview

IOPS commands read from result files and from metadata files generated during execution. Metadata files have the __iops_ prefix and are documented in the Metadata Files user guide.

Data Sources by Command

iops find ◄─── __iops_index.json
          ◄─── __iops_status_rollup.json (preferred; falls back to per-folder scan)
          ◄─── __iops_status.json (fallback)
          ◄─── __iops_run_metadata.json (optional metadata)

iops find --watch ◄─── __iops_index.json (polling)
                  ◄─── __iops_status_rollup.json (preferred; polling)
                  ◄─── __iops_status.json (fallback scan)

iops report ◄─── __iops_run_metadata.json
            ◄─── results.csv/parquet/sqlite

Status roll-up fast path

Both iops find and watch mode read the run-root __iops_status_rollup.json aggregate instead of scanning every execution’s repetition folders, which keeps them responsive on large parameter spaces. The per-folder __iops_status.json files remain the source of truth (and the node-to-runner channel in single-allocation mode); the roll-up is an accelerator. Watch always prefers it; iops find only trusts it when the run marked it complete, otherwise it falls back to the folder scan. When the roll-up is absent (older runs, archives), both fall back automatically.

iops find

Single execution view (when pointing to exec_XXXX folder):

iops find --watch

Watch mode is designed to work during execution, so it only reads from files that exist before the benchmark completes. It does not read from __iops_run_metadata.json (which is written at the end).

iops report

The report command reads from the run metadata file and the results output file. It does not read from index or status files.

Results file types:

File Locations

workdir/
  run_001/                              # Run root
    __iops_index.json                   # Index file (find, watch)
    __iops_run_metadata.json            # Run metadata (find, report)
    results.csv                         # Results file (report)
    exec_0001/                          # Execution folder
      __iops_params.json                # Parameter values (find single)
      __iops_status.json                # Test-level status (SKIPPED only)
      repetition_1/
        __iops_status.json              # Repetition status (find, watch)
        __iops_sysinfo.json             # System info (stored in results)
      repetition_2/
        ...
    exec_0002/
      ...

Source Code References

Key Functions

find.py:

• find_executions() - Main entry point
• _read_status() - Reads and aggregates status from status files
• _read_run_metadata() - Reads run metadata file
• _show_executions_from_index() - Displays execution table

watch.py:

• watch_executions() - Main entry point with live updates
• _load_index() - Loads index file with execution metadata
• _collect_execution_data() - Collects current status from all executions
• _build_table() - Builds Rich table for display
• _build_progress_bar() - Builds progress bar with status counts

report_generator.py:

• ReportGenerator.__init__() - Initializes with workdir path
• load_metadata() - Loads __iops_run_metadata.json
• load_results() - Loads results from CSV/Parquet/SQLite
• generate_report() - Generates HTML report

Data Flow Diagrams

During Execution

Runner
  │
  ├──► __iops_index.json (updated incrementally)
  │
  ├──► exec_XXXX/
  │      ├──► __iops_params.json (before test)
  │      └──► __iops_status.json (if SKIPPED)
  │
  └──► exec_XXXX/repetition_N/
         ├──► __iops_status.json (after completion)
         └──► __iops_sysinfo.json (via probe script)

After Execution

Runner
  │
  ├──► __iops_run_metadata.json (at end)
  │
  └──► results.csv (incrementally during execution)