Tools

Offline helpers that run beside a training job rather than inside one. The guide builds an index in Parquet and Vortex.

Build an index.json for a dataset directory (fast shard discovery).

CLI: python -m zephon.build_index <format> <dataset_dir>.

build_index(format_name, dataset_dir, *, output_path=None, progress=True, max_workers=8)

Build an index.json for a JSONL, Parquet, or Vortex dataset.

Writes to dataset_dir/index.json unless output_path is given, printing progress when progress is true. Local paths and remote roots supported by Zephon’s storage backend are accepted. Returns the output path or URL as a string. At most max_workers shards are inspected concurrently; set it to 1 for sequential indexing.

Return type:

str