How to A/B Test AI Image Models Like a Designer (Not a Dev)

Developers A/B models in code: same prompt, two IDs, a spreadsheet of latency and cost. Designers need the same experiment with eyes. If the outputs are not on one surface, it is not a test. It is two vibes you already forgot.

OpenRouter made the developer version a product. OpenRouter for creatives is the designer version: same brief, multiple models, one board. Pantr’s Change model action is built for that.

Set the experiment before you generate

Lock four things or you are not A/B testing:

• Prompt — don’t “improve” it between models.
• Ratio — 4:5 vs 16:9 is not a model difference.
• References — same `@` set, same order in your head. If a model’s ref cap is lower, that is a result (ineligible), not a style.
• Variant count — compare 1 vs 1 first. Four-of-each is a moodboard, not an A/B.

Write the job in one sentence: “photoreal hero,” “edit the bag,” “poster with the name.” Then pick the two models that claim that job. Starting pairs: Flux vs Nano Banana vs GPT Image.

Run it on the canvas

Generate on model A. Select the result → Change model → pick B (and C if you must). Pantr shows the total API price before it runs. New stills tile to the right of the original. Do not auto-select winners. Sit back from the screen. The one that matches the sentence you wrote is the winner — not the one that looks more “AI impressive.”

If you use your OpenRouter key, the same button still works. Billing just lands on OpenRouter instead of the wallet.

Score like an art director

A cheaper model that hits the brief beats a famous model that misses. That is the whole OpenRouter lesson, translated. Token-talk: pay-per-token for creatives.

Don’t pollute the test

• Don’t inpaint on A then compare to a fresh B. That’s two briefs.
• Don’t change the prompt “just a little.” Start a second row if you must iterate copy.
• Don’t mix image models with video models in one tile row. Video has its own rules: Sora, Veo, Kling, Seedance.

When you are done, keep the losers on the board until the client picks. Context is the archive. More on that: keep creative context when you switch. Start with €1.00 and one prompt. Two models. One winner per frame.