MIPROv2, GEPA and few-shot on one DSPy extractor

The problem

On the platform I maintain, an extraction step in the pipeline pulls three short text fields out of a raw text record with a DSPy program running on a small, low-cost model. The program is compiled by MIPROv2 in every training cycle, and a candidate is promoted when its held-out score is at least the current one's. GEPA is now in the DSPy version we run (MIPROv2 searches over instructions and demonstrations with Bayesian optimization; GEPA rewrites the instructions from a reflection model's critique of failed examples), and a lighter option, a few-shot compile that only bootstraps example demonstrations, is available in the same training code. The question: does the more expensive optimizer earn its cost on this task?

Two costs matter and only the first one is easy to see. The compile is paid every cycle. The compiled prompt is paid on every call, and the three optimizers write prompts of different lengths.

What I varied

One variable: the optimizer. For each seed, one train set (200 rows) and one validation set (100 rows) are drawn and every arm gets the same ones, few-shot carries up to 16 bootstrapped demonstrations, and every compiled program is run once over the same 100 held-out rows. The model, the temperature and the metric are fixed. Each optimizer runs its own light preset, the smallest search setting it ships with, and what it spends is measured, not matched. A row counts as correct when all three extracted fields match the label. A small slice of 66 rows was also checked by a person and is reported separately. The arms are zero-shot, few-shot (bootstrapped demonstrations), MIPROv2 and GEPA, 3 seeds each. GEPA needs a reflection model, and I use the same small model for it, so its calls are priced like the others. A second part takes the best arm, GEPA, and varies the training size instead, from 100 to 1,000 rows; it follows the result.

Measured per arm: rows with all three fields correct, accuracy per field, compile cost in dollars and minutes (task, proposer and reflection calls, from token counts times list price), compile metric calls (how many times the scoring function runs during the compile), prompt tokens per call, and the resulting cost per 100 inferences.

Figure 1: Design. Four ways of preparing the same extractor; per seed every arm gets the same train and validation draw, and all are scored on the same 100 held-out rows. Only the optimizer changes, each run on its own light preset.

Result

Rows fully correct out of 100, mean over 3 seeds: zero-shot 73, few-shot 76, MIPROv2 76, GEPA 79. The four arms sit within 6 rows of each other, and the 95% intervals for a single 100-row split (about 8 to 9 rows either side) overlap.

Figure 2: Score against prompt length. One dot is one seed (3 seeds per arm, the same 100 held-out rows, a row counts when all three fields match); the ring is the mean of the three. The dashed line is the zero-shot score. The seed spread within an arm is as wide as the gap between arms, and the prompt grows about 11 times from zero-shot to few-shot.

The seed-to-seed spread is as large as the gaps between arms. Few-shot ranged from 70 to 81 across seeds, GEPA from 75 to 82, MIPROv2 from 75 to 78, and zero-shot did not move at all because it is not trained. On paired exact (McNemar) tests on the same 100 rows, per seed, few-shot was ahead of zero-shot on two seeds, significant once (p of 0.008), and behind on the third. MIPROv2 and few-shot swapped places: MIPROv2 was ahead on one seed (p of 0.02) and behind on two, once significantly (p of 0.03). GEPA was ahead of MIPROv2 on two seeds (p of 0.06 and 0.02) and behind on the third. GEPA was ahead of few-shot on all three seeds (by 3, 5 and 1 rows), never significantly. Accuracy per field moved by up to 7 points between arms.

Arm	Rows correct (of 100)	Per seed	Compile cost	Compile time	Prompt tokens per call	Cost per 100 inferences
Zero-shot	73	73 / 73 / 73	none	none	767	$0.012
Few-shot	76	77 / 70 / 81	$0.004	0.2 min	8,580	$0.090
MIPROv2	76	75 / 78 / 75	$0.21	7.4 min	3,177	$0.037
GEPA	79	80 / 75 / 82	$0.19	12 min	1,555	$0.023

Table 1: Accuracy and cost per arm. Compile cost covers every task, proposer and reflection call. Inference cost uses the measured prompt and completion tokens per call at $0.10 / $0.40 per 1M tokens.

The costs separate more clearly than the scores. Few-shot compiled in 0.2 minutes for $0.004, MIPROv2 in 7.4 minutes for $0.21 and GEPA in 12 minutes for $0.19. After that the prompts differ a lot. Zero-shot sends 767 tokens per call, GEPA 1,555, MIPROv2 3,177 and few-shot 8,580, because the bootstrapped demonstrations ride along on every call. Per 100 inferences that is $0.012 for zero-shot, $0.023 for GEPA, $0.037 for MIPROv2 and $0.090 for few-shot. Few-shot's prompt is 5.5 times GEPA's in tokens and about four times in cost per call. Read against a single call, the compile costs as much as about 4 of that arm's own inferences for few-shot, 577 for MIPROv2 and 835 for GEPA.

Counted in tokens, prompt and completion, a compile plus 100 inference calls uses 0.09 million for zero-shot, 0.90 million for few-shot, 1.57 million for GEPA and 2.16 million for MIPROv2 (means over 3 seeds). At that volume the compile dominates. The longer few-shot prompt adds up with traffic: its total passes GEPA's after about 195 calls and MIPROv2's after about 333, and zero-shot stays the cheapest at any volume.

Taking GEPA further: more training data

GEPA had the highest mean, so I took it further with one more variable: the training size. GEPA was compiled at 100, 200, 400, 700 and 1,000 training rows, with a validation set half that size and one seed per size, and every compiled program was scored on the same 500 held-out rows: the 100 above plus 400 more from the same split. The sets are nested, so each larger set contains every smaller one, and the 200-row sets are the seed-0 sets from the comparison. The uncompiled program is the 0-row point.

Figure 3: GEPA with more training data. One GEPA compile per training size (light preset, the same seed), each scored on the same 500 held-out rows; the whiskers are 95% Wilson intervals. 0 is the uncompiled program. The score peaks at 100 to 200 rows, and 1,000 rows stay below it.

The score rose from 341 of 500 (68.2%) with no training to 404 (80.8%) at 100 rows and 411 (82.2%) at 200, then fell to 376 at 400 and 355 at 700, and came back only to 371 at 1,000. The step from 100 to 200 is not resolved (exact paired test, p of 0.34). Every step after it is, at p of 0.007 or lower on these 500 rows (unadjusted for the several comparisons), and 1,000 rows still finished 40 rows below 200 (p below 0.001, the same test). On the comparison's 100 rows the 200-row compile scored 84, against 80 for the seed-0 GEPA compile above on the same training and validation rows: one example of how much one compile can differ from the next.

Training rows	Validation rows	Rows correct (of 500)	Of the first 100	Compile cost	Prompt tokens per call	Cost per 100 inferences
0	0	341 (68.2%)	73	none	716	$0.012
100	50	404 (80.8%)	83	$0.15	1,388	$0.020
200	100	411 (82.2%)	84	$0.25	4,069	$0.050
400	200	376 (75.2%)	79	$0.28	1,889	$0.026
700	350	355 (71.0%)	76	$0.34	1,205	$0.018
1,000	500	371 (74.2%)	79	$0.58	1,742	$0.025

Table 2: GEPA by training size. "Of the first 100" is the score on the 100 held-out rows used in the comparison above. A row whose output could not be parsed counts as wrong (4 rows at 100, 1 at 1,000). Compile cost covers every task and reflection call; inference cost uses measured tokens at $0.10 / $0.40 per 1M. The five compiles ran at the same time, so their wall time is not comparable with Table 1 and is left out.

In this run, more data cost more and did not buy accuracy. Compile cost went from $0.15 at 100 rows to $0.58 at 1,000. The prompt GEPA wrote did not grow with the data: between 1,205 and 4,069 tokens per call with no order by size, so the cost per 100 inferences stayed between $0.018 and $0.050.

Two explanations fit, and this run does not separate them. The first is one seed per size: a single compile varies, and the comparison saw a seed spread of 3 to 11 rows out of 100, up to about the size of the drop here. The paired tests cover the held-out rows, not that variation. The second is the budget. GEPA's light preset sets its budget at about four passes over the validation set plus a fixed 380 metric calls, at every size. A larger validation set leaves room to score fewer candidate prompts in full, and the fixed part is ten reflection steps on minibatches of 35 training rows, so at most about 350 training rows are ever read, whether the training set has 100 rows or 1,000. What the run does show is that on this extractor, at the light preset, more than 200 training rows did not give a better program.

Tradeoffs

A longer compile. GEPA took about 5 minutes more per compile and 35% more metric calls than MIPROv2, which matters once rate limits are the bottleneck.
The labels are mostly machine-derived. Most gold rows come from earlier model output or from corrections, and only a small slice was checked by a person, so a score measures agreement with those labels, and labels derived from earlier model output reward agreement with that model, not correctness. The human-checked slice was too easy to separate the arms.
A hundred rows is a small test. Differences inside the seed spread, 3 to 11 rows, are not resolved. A bigger held-out set would be needed to rank the arms, not to make the cost comparison.
One seed per training size. The curve rests on one compile at each size, and one compile varies by up to about as much as the drop after 200 rows. More seeds per size would be needed to say whether the drop is the data or the draw.
One extractor, one model. The ordering of the arms can change for a different task or a larger model.