Core Modules

• mmirage.core.loader — Data Loaders
  • Base classes
  • JSONL loader
  • HuggingFace local loader
  • Loader utilities
• mmirage.core.process — Processors
  • Variables
  • Base processor
  • Mapper
  • LLM processor
  • Batch API processor
  • Batch processing
• mmirage.core.writer — Writer
  • Template renderer
• Shard Utilities
  • mmirage.shard_process
  • mmirage.shard_utils
  • mmirage.merge_shards
  • mmirage.cli_utils.status
  • mmirage.cli_utils.slurm
  • mmirage.cli_utils.runtime