MMIRAGE Documentation¶
MMIRAGE is an open-source platform for large-scale dataset processing using large language models (LLMs) and vision-language models (VLMs).
It provides a declarative, YAML-driven pipeline to extract variables from data samples, construct prompts, run model inference, render structured outputs, and distribute the work across shards — locally or on HPC clusters via SLURM.
What is MMIRAGE?¶
MMIRAGE (Modular Multimodal Intelligent Reformatting and Augmentation Generation Engine) lets you transform large datasets using generative models by:
extracting named variables from each sample with JMESPath queries
constructing prompts with Jinja2 templates
running inference locally via a SGLang engine, or asynchronously via a provider batch API (OpenAI, Anthropic)
rendering processed outputs into any JSON schema you define
splitting work across shards with automatic resume, retry, and merge
It natively supports text and image inputs, SLURM-based cluster execution, throughput benchmarking, and atomic crash-safe writes.
Where to start¶
Depending on what you want to do, start in different places:
to install MMIRAGE, read Installation
to run a first pipeline end to end, read Quickstart
to understand core concepts and terminology, read Concepts
to understand the full pipeline data flow, read Pipeline
to process images with a VLM, read Multimodal Processing
to generate images from dataset prompts, read Image Generation
to run at scale on a cluster, read SLURM & Cluster Deployment
to use a provider batch API, read Batch API
to use your custom python module, read Custom Module
to measure throughput and GPU efficiency, read Benchmarking
to configure every parameter, read Configuration Reference
to contribute to the codebase, read Developer Guide
Documentation map¶
Set up MMIRAGE and prepare your environment.
Run a first pipeline end to end in minutes.
Core vocabulary: shards, variables, schemas, execution modes.
Step-by-step walkthrough of what MMIRAGE does with your data.
Running VLMs on image datasets.
Generating image datasets from templated prompts.
Scaling pipelines across HPC nodes with SLURM.
Async inference via a provider batch API (OpenAI, Anthropic).
Running your own Python function in an isolated worker pool.
Measuring throughput, GPU utilization, and efficiency.
Full YAML parameter reference for every section.
All mmirage subcommands, flags, and examples.
Internal module layout and design decisions.
Testing, code style, extending MMIRAGE, and debugging.
Page guide¶
Installation: set up MMIRAGE and prepare your environment
Quickstart: run a first minimal pipeline from scratch
Concepts: learn the vocabulary used throughout the documentation
Pipeline: understand what happens at each stage of the pipeline
Multimodal Processing: configure image inputs and VLM chat templates
Custom Module: run your own Python function over the dataset in an isolated worker pool
Image Generation: generate images with an external or managed SGLang server
SLURM & Cluster Deployment: submit, monitor, and retry jobs on HPC clusters
Batch API: send requests asynchronously to a provider batch API
Benchmarking: collect and interpret throughput and GPU efficiency metrics
Configuration Reference: complete reference for every YAML parameter
CLI Reference: all subcommands, flags, and their behaviour
Architecture: internal package layout and key design decisions
Developer Guide: contribute code, run tests, add loaders or processors