MIPRO

MIPRO is a DSPy optimizer for Language Model Programs that treats prompts not as hand-written strings but as tunable program parameters: free-form instructions plus few-shot demonstrations for each LM call. Its central contribution is joint optimization: rather than first choosing examples and then separately editing instructions, MIPRO searches over combinations of instructions and demonstrations using program-level feedback, a learned surrogate, and mini-batch evaluation. MIPROv2 is the current DSPy-facing version of that idea: it bootstraps candidate examples, proposes grounded instructions, and uses Bayesian optimization to select prompt configurations that improve a downstream metric. aclanthology.org

Coverage note: verified through May 18, 2026.

Naming and scope

The canonical expansion in Opsahl-Ong et al. is Multi-prompt Instruction PRoposal Optimizer, while DSPy’s documentation expands MIPROv2 as Multiprompt Instruction PRoposal Optimizer Version 2. This entry follows the article prompt’s title, “Multi-Step Instruction Proposal and Optimization,” as a descriptive label for the multi-stage process, but the cited literature uses “multi-prompt” rather than “multi-step” as the formal acronym expansion. aclanthology.org

MIPRO sits inside DSPy, a framework for programming LMs with modules, signatures, metrics, and optimizers rather than directly editing prompt strings. DSPy’s own overview describes the framework as a declarative system for building modular AI software whose algorithms “compile” AI programs into effective prompts or weights; the older DSPy paper frames the same idea as a move away from hard-coded prompt templates discovered by trial and error. dspy.ai

The algorithm is aimed at multi-stage LM programs: retrieval-augmented QA pipelines, verifier pipelines, answer ensembles, chain-of-thought classifiers, or agentic workflows in which several LM calls feed one another. In that setting, a single final metric is available, but intermediate labels for each LM call usually are not; the optimizer must infer which prompt parameters helped from end-to-end behavior. Opsahl-Ong et al. explicitly study this setting under weak assumptions: no access to LM weights, log probabilities, gradients, or hand-written intermediate-stage metrics, only the program, a train set, and a downstream metric. aclanthology.org

The problem MIPRO addresses

A modern LLM Pipeline often contains several prompt-bearing modules. A multi-hop QA program, for example, may have one module that generates retrieval queries and another that synthesizes an answer from retrieved passages; an agent may have separate prompts for planning, tool selection, verification, and final response. Hand-tuning each prompt independently is fragile because a prompt that improves one module can hurt another, and the observed metric is usually only the final task score.

Opsahl-Ong et al. formalize this as optimization over the string-valued variables of an LM program. Each module has prompt templates with open slots, and in the MIPRO paper the slots of interest are an instruction variable and a set of demonstration variables. The optimizer seeks an assignment of strings to those variables that maximizes the program-level metric over training data. aclanthology.org

This creates two hard subproblems. The first is proposal: the space of possible instructions is effectively unbounded, and the space expands with every module. The second is credit assignment: when only a final score is available, the optimizer must infer which instruction or demonstration choice helped which stage. Opsahl-Ong et al. identify exactly these two challenges and design MIPRO around them. aclanthology.org

Algorithm at a glance

Stage	What MIPRO does	Why it matters
Candidate demonstration generation	Runs training examples through the program and keeps successful traces as module-level few-shot demonstrations.	Converts end-to-end successes into local examples without requiring labels for every intermediate module.
Grounded instruction proposal	Uses an LM proposer, informed by data summaries, program summaries, successful demonstrations, and prompt-generation tips, to generate instruction candidates.	Narrows the enormous instruction space to a small, task-grounded candidate set.
Joint discrete search	Searches over combinations of instruction candidates and demonstration candidates for every module.	Captures interactions between instructions and examples rather than treating them as independent knobs.
Mini-batch surrogate evaluation	Evaluates candidate prompt configurations on random mini-batches and updates a Bayesian-style surrogate.	Reduces evaluation cost while steering search toward promising configurations.
Full validation checkpoints	Periodically evaluates top candidates on the full validation/train split and returns the best full-evaluated program.	Reduces the risk that the optimizer selects a mini-batch artifact.

DSPy’s MIPROv2 documentation presents the same high-level decomposition: bootstrap few-shot examples, propose instruction candidates using dataset/program/demo/tip context, then use Bayesian optimization over prompt combinations with mini-batch trials and periodic full validation. dspy.ai

Inputs and objective

A typical DSPy optimizer receives three things: a DSPy program, a metric, and training inputs. The program may be a single dspy.Predict module or a complex multi-module program; the metric is the score to maximize; the train set can be small and may contain incomplete labels depending on the task. dspy.ai

For MIPRO, the effective objective is:

arg⁡max⁡instruction/demo choicesE(x,y)∼D[μ(Φθ(x),y)]\arg\max_{\text{instruction/demo choices}} \mathbb{E}{(x,y)\sim D}\left[\mu(\Phi{\theta}(x),y)\right]arginstruction/demo choicesmax​E(x,y)∼D​[μ(Φθ​(x),y)] where Φθ\Phi_{\theta}Φθ​ is the LM program instantiated with a particular choice of instructions and demonstrations, and μ\muμ is the downstream metric. Unlike gradient-based prompt tuning, the optimizer treats the program as a black box. It can run the program, observe scores, and update its search state, but it does not require gradients or model internals. aclanthology.org

Demonstration bootstrapping

MIPRO inherits one of DSPy’s core ideas: successful traces can become demonstrations. The optimizer samples a training input, runs the current program, and records the internal input/output trace for each module. If the final program output passes the metric or exceeds a metric threshold, the trace is treated as a candidate demonstration for the modules that produced it. aclanthology.org

This is the same broad mechanism behind DSPy’s BootstrapFewShot family. DSPy’s BootstrapFewShot documentation describes it as composing demonstrations from labeled examples plus bootstrapped examples; each bootstrap round can rerun the LM at temperature 1.0 with a new rollout ID to collect diverse traces. dspy.ai

The important distinction is that MIPRO does not stop at “find some demos.” It builds multiple sets of candidate demonstrations and later searches over them jointly with instruction candidates. In the MIPRO paper, Bootstrap Random Search is the demonstration-only baseline: it bootstraps candidate demo sets and then randomly searches over their combinations. MIPRO adds instruction proposal and Bayesian-style joint selection on top. aclanthology.org

Grounded instruction proposal

The instruction proposal step uses an LM as a proposer. Rather than asking the proposer to invent prompts in a vacuum, MIPRO grounds the proposer in task and program context: summaries of the raw dataset, summaries of the program’s control flow, successful bootstrapped demonstrations, and sometimes histories of prior prompt evaluations. aclanthology.org

This is a methodological bridge between ordinary prompt engineering and algorithmic optimization. Human prompt writers inspect examples, program behavior, and failure modes; MIPRO attempts to automate part of that inspection and convert it into candidate instructions. The proposer LM does not directly “solve” the task; it generates instruction strings that will later be empirically tested.

DSPy’s MIPROv2 docs make this concrete: the proposer can receive a generated summary of the training dataset, a summary of the program code and predictor, bootstrapped reference inputs/outputs for the predictor, and a randomly sampled tip such as being concise or creative. The prompt model then writes instruction candidates. dspy.ai

This proposal stage is Bayesian-style only indirectly. The instruction strings themselves are proposed by an LM. The Bayesian component appears in the search over the discrete candidate pool: after candidates are generated, MIPRO uses a surrogate model to decide which combinations to try next.

Surrogate search and mini-batch evaluation

The core search problem is combinatorial. If a program has mmm modules, and each module has nin_ini​ instruction candidates and did_idi​ demo-set candidates, the joint search space is approximately:

∏i=1m(ni×di)\prod_{i=1}^{m} (n_i \times d_i)i=1∏m​(ni​×di​) Even small candidate counts become expensive when each evaluation runs an LM program over a validation set.

MIPRO addresses this with a surrogate model. Opsahl-Ong et al. use Optuna’s Tree-structured Parzen Estimator implementation to model the quality of parameter combinations, including multivariate interactions among choices. The paper frames this as a solution to credit assignment: rather than relying on a proposer LM to infer which module-level change caused a program-level gain, the optimizer uses observed scores to guide future discrete search. aclanthology.org

The evaluation loop is stochastic. MIPRO evaluates a candidate program on a random mini-batch, updates the surrogate from the observed score, and periodically evaluates the best candidates on the full train set or validation set. The paper states that this mini-batching lets the optimizer explore and exploit more configurations efficiently, while full evaluations every fixed number of steps guard against choosing a candidate that merely got lucky on a batch. aclanthology.org

MIPROv2 exposes this in its public API: compile accepts arguments such as num_trials, minibatch, minibatch_size, minibatch_full_eval_steps, program_aware_proposer, data_aware_proposer, tip_aware_proposer, and fewshot_aware_proposer. The defaults encode the same research assumptions: generate candidates, evaluate many candidate combinations cheaply, and periodically validate more fully. dspy.ai

MIPRO vs. MIPROv2

MIPRO in the EMNLP paper is the research algorithm: bootstrap demo candidates, generate grounded instruction candidates, then use Bayesian optimization over instruction/demo combinations. It is evaluated against instruction-only and demonstration-only baselines on seven tasks. aclanthology.org

MIPROv2 is the DSPy API-level optimizer that implements the same high-level recipe and is documented as capable of jointly optimizing instructions and few-shot examples, or performing zero-shot instruction optimization by setting demo counts to zero. It supports auto="light", "medium", or "heavy" modes and separates prompt_model, task_model, and teacher settings. dspy.ai

One naming trap: the MIPRO paper also discusses 0-Shot MIPRO++, a research variant that meta-optimizes how instruction proposals are generated, including whether to use dataset summaries, program summaries, tips, proposer temperature, and selected bootstrapped demos shown to the proposer. MIPRO++ is not simply the same thing as the DSPy class name MIPROv2; it is a particular experimental variant in the paper. aclanthology.org

Comparison with BootstrapFewShot, COPRO, OPRO, and manual tuning

Method	Optimizes	Search style	MIPRO relationship
BootstrapFewShot / BootstrapRS	Few-shot demonstrations	Bootstrap successful traces; optionally random-search demo sets	MIPRO keeps this mechanism but adds instruction proposal and joint instruction/demo search.
COPRO	Instruction/signature text	Breadth/depth candidate generation and evaluation	Earlier DSPy instruction optimizer; useful baseline conceptually, but not the main baseline in the EMNLP MIPRO table.
Module-Level OPRO	Instructions	LM proposer sees prompt-score history and proposes new instructions	MIPRO separates proposal from credit assignment using a surrogate.
Manual prompt tuning	Instructions, examples, formatting, task decomposition	Human iteration	MIPRO automates part of this loop but still requires a metric, training cases, and budget.

COPRO is documented in DSPy as an optimizer that “optimizes signature” for a student program and has breadth/depth hyperparameters for generating and evaluating prompt candidates. MIPRO differs by explicitly optimizing demonstrations and instructions together rather than only rewriting signatures/instructions. dspy.ai

The MIPRO paper’s direct instruction baseline is not COPRO but Module-Level OPRO, an adaptation of OPRO to multi-module LM programs. In Module-Level OPRO, each module receives a history of instructions and program-level scores, and an LM proposes new instructions; the method assumes the program score is a usable proxy for each module’s instruction quality. MIPRO relaxes that assumption by using a surrogate model for credit assignment. aclanthology.org

Manual tuning is best treated as the broader background rather than a clean MIPRO benchmark. The MIPRO paper motivates the work by noting that LM programs are commonly designed by manual prompt engineering, but its main table compares algorithmic optimizers rather than a controlled panel of human prompt engineers. The older DSPy paper provides stronger direct evidence against manual-style baselines: it reports self-bootstrapped DSPy pipelines outperforming standard few-shot prompts and expert-created demonstrations in its case studies. aclanthology.org

Empirical results

Opsahl-Ong et al. evaluate seven tasks: ScoNe natural-language inference, HotPotQA multi-hop QA, HoVer multi-hop claim verification, a conditional HotPotQA variant, Iris classification, Iris with a prompt typo, and Heart Disease classification. The benchmark includes both single-stage and multi-stage DSPy programs; the paper uses 500 training examples, 500 development examples, and a 2,000-example test set when available. aclanthology.org

The experiments compare instruction-only optimizers, demonstration-only optimizers, and MIPRO as the joint instruction-plus-demonstration optimizer. The primary task model is Llama-3-8B, GPT-3.5 is usually the instruction proposer, and GPT-4o is used as a teacher model for harder bootstrapping tasks such as ScoNe and HoVer. aclanthology.org

Test task	Unoptimized / seed row	Strong demo-only baseline	MIPRO test score	Main reading
ScoNe	69.1	77.4 Bayesian Bootstrap	79.4	Joint optimization improves over strong demo search.
HotPotQA	36.1	46.2 Bayesian Bootstrap	46.4	MIPRO barely edges the strongest demo-only result.
HoVer	25.3	37.6 Bayesian Bootstrap	39.0	Joint search helps multi-hop retrieval/verification.
HotPotQA Conditional	6.0	10.4 Bootstrap RS	23.3	Instructions matter strongly when rules are conditional.
Iris	40.9	94.1 Bootstrap RS	88.6	Demo-only wins; MIPRO is not uniformly best.
Iris-Typo	32.0	58.7 Bootstrap RS	68.7	Instruction optimization helps recover from prompt misspecification.
Heart Disease	26.8	79.2 Bootstrap RS	74.2	Demo-only wins; MIPRO underperforms the best baseline.

These figures come from the EMNLP paper’s Table 2, which averages five runs and marks statistically supported best values using Wilcoxon signed-rank tests. The paper summarizes the result as MIPRO outperforming baseline optimizers on five of seven tasks, by as much as about 13 percentage points in the abstract and conclusion. aclanthology.org+2aclanthology.org+2

The results support a narrower claim than “prompt optimization always wins.” Demonstrations alone are often very strong; on Iris and Heart Disease, the demo-only baseline beats MIPRO. Instruction optimization is most valuable when the task has subtle conditional rules, prompt misspecification, or behavior that cannot be easily captured by a small set of examples. Opsahl-Ong et al. explicitly draw this lesson: bootstrapped demonstrations are often essential, instruction optimization matters more for conditional rules, and joint optimization is generally best but not universally dominant. aclanthology.org

For code generation, the core MIPRO paper does not provide the same direct evidence as it does for HotPotQA, HoVer, ScoNe, Iris, and Heart Disease. Later agent-framework work has evaluated systems integrating MIPRO on MBPP-style code generation, but that evidence is mixed: in EvoAgentX’s reported table, MIPRO improved HotPotQA and MATH over the original baseline but scored 68.00 on MBPP pass@1 versus a 69.00 original baseline, while AFlow reached 79.00. The honest reading is that MIPRO-style optimization is plausible for code-generation programs, especially when code generation is embedded in a larger agent workflow, but code-generation gains are not the strongest primary result for MIPRO itself. arXiv

Methodological contribution

MIPRO’s important contribution is not merely “an LM writes better prompts.” APE, OPRO, and EvoPrompt had already shown that LMs can generate, mutate, or iteratively improve instruction strings. MIPRO’s sharper contribution is joint, program-level prompt optimization: instructions and few-shot demonstrations are both treated as parameters of every LM call in a multi-stage program, and the optimizer searches over their combinations using only end-to-end feedback. aclanthology.org+3arXiv+3arXiv+3

That matters because instructions and demonstrations are not independent. A demonstration set can compensate for a vague instruction; a precise instruction can make fewer examples necessary; examples can teach latent formatting and reasoning patterns; instructions can encode conditional rules that examples underrepresent. Searching these choices jointly is more faithful to how prompts actually behave inside LLM systems.

The second methodological contribution is the separation between proposal and credit assignment. OPRO-style methods ask the proposer LM to infer from a history of scored prompts and then produce better prompts. MIPRO delegates proposal to an LM but delegates credit assignment to a Bayesian surrogate over discrete candidates. This is less semantically elegant than an LM “reflecting” on its own failures, but it is more structured and better aligned with black-box hyperparameter optimization. aclanthology.org

The third contribution is the use of mini-batch evaluation for prompt-program search. Full validation after every prompt change is expensive; purely tiny-batch search is noisy. MIPRO’s compromise—mini-batch trials plus periodic full evaluation—resembles the practical engineering pattern used in many evaluation-driven optimization loops: cheap exploration, periodic confirmation, and retention of the best full-evaluated candidate. aclanthology.org

Relationship to OPRO

OPRO—Optimization by PROmpting—uses an LLM as an optimizer. The optimization problem is described in natural language, prior solutions and their scores are placed in the prompt, and the LLM proposes new solutions; in prompt optimization, those solutions are task instructions. The OPRO paper reports that optimized prompts can outperform human-designed prompts on GSM8K and Big-Bench Hard tasks. arXiv

MIPRO inherits OPRO’s basic insight that an LLM can propose natural-language candidate instructions. The difference is architectural. OPRO is mainly a history-conditioned proposal loop over strings; MIPRO is a program optimizer over multiple prompt variables, including demonstrations, and it uses a non-LLM surrogate for search and credit assignment. aclanthology.org

The MIPRO paper explicitly discusses Module-Level OPRO and Program-Level OPRO variants. Module-Level OPRO assumes program-level score histories are sufficiently informative for each module’s instruction; Program-Level OPRO gives the proposer longer trajectory histories and relies on the LM to assign credit. The authors regard those assumptions as strong, especially as histories grow, and focus the main experiments on other methods. aclanthology.org

Relationship to EvoPrompt

EvoPrompt connects LLMs with evolutionary algorithms. It starts from a population of prompts and uses LLM-mediated evolutionary operators to generate new candidates, then keeps or replaces prompts based on development-set performance. The EvoPrompt paper evaluates closed- and open-source LMs across language understanding, generation, and BBH tasks and reports improvements over human-engineered prompts and automatic prompt generation baselines. arXiv

MIPRO and EvoPrompt share the view that prompts are discrete objects suitable for black-box search. But they optimize different objects at different structural levels. EvoPrompt is population-based prompt evolution; MIPRO is modular LM-program optimization over instructions and demonstrations. EvoPrompt’s operators resemble mutation and crossover over prompt text; MIPRO’s search resembles Bayesian hyperparameter optimization over a pre-generated candidate set.

A useful mental model: EvoPrompt is closer to evolutionary search over prompt strings, OPRO is closer to LLM-as-optimizer over prompt histories, and MIPRO is closer to Bayesian program compilation over prompt parameters. All three are instances of Automatic Prompt Optimization, but MIPRO is the one most tightly coupled to DSPy’s program abstraction.

Practical use

A MIPROv2 optimization run usually looks like this:

1. Define a DSPy program with one or more predictors.

2. Define a metric that scores final outputs.

3. Provide a train set, and ideally a validation set.

4. Choose a prompt model, task model, demo counts, candidate counts, and budget.

5. Run MIPROv2.compile(...).

6. Save the optimized program artifact.

The official MIPROv2 example initializes a DSPy LM, creates MIPROv2(metric=..., auto="medium"), compiles a dspy.ChainOfThought("question -> answer") program on GSM8K training data, and saves the optimized program. dspy.ai

In production-style workflows, MIPRO is best understood as an offline optimizer, not an inference-time algorithm. You pay the optimization cost during development or scheduled retraining, then deploy the compiled prompt program. This is analogous to hyperparameter tuning: expensive during search, cheap after selection, assuming the chosen prompts generalize.

MIPRO is especially attractive when the program is modular, the task has a stable metric, and the team expects the system to be reused enough that optimization cost can be amortized. It is less attractive when the task lacks labels, the metric is unreliable, prompts change daily, model behavior shifts frequently, or the system is so latency-sensitive that even few-shot demonstrations are too expensive at inference time.

Limitations

Compute cost. MIPRO multiplies LM calls across bootstrapping, instruction proposal, candidate trials, mini-batch evaluations, and periodic full evaluations. The public API exposes this directly through candidate counts, trial counts, mini-batch sizes, demo limits, and prompt/task model choices. The practical cost scales with the number of modules, the size of the train/validation sets, the number of candidate instructions and demo sets, and the price/latency of the LMs used. dspy.ai

Evaluation dependence. MIPRO is only as good as the metric. If the metric rewards shallow formatting, overfits to lexical artifacts, or fails to capture production utility, MIPRO will optimize toward the wrong target. DSPy optimizers generally require a program, a metric, and training inputs; without a trustworthy eval signal, MIPRO becomes automated prompt churn rather than optimization. dspy.ai

Seed-prompt dependence. Opsahl-Ong et al. note that their optimizers have limited ability to infer complex task rules without a hand-written seed prompt. Grounding can expose dataset details and examples, but that may not be enough to infer a complete hidden specification. This is visible in tasks where classification criteria are not obvious from the seed instruction or a small number of examples. aclanthology.org

Benchmark narrowness. The MIPRO paper’s benchmark is diverse for a first study—multi-hop QA, claim verification, NLI, and classification—but it is still a small benchmark suite. The authors explicitly state that more remains to learn about optimizer performance on increasingly complex tasks and programs. aclanthology.org

No guarantee of monotonic improvement. Table 2 shows clear wins on several tasks, but also failures relative to demo-only baselines. MIPRO is an optimizer over noisy black-box evaluations, not a theorem of improvement. aclanthology.org

Prompt transfer remains uncertain. An optimized prompt can be model-specific. The MIPRO paper fixed proposer and task models for most experiments and calls for future work on whether the methods show consistent performance with different models. That caveat matters in production, where model providers, versions, and decoding defaults can change. aclanthology.org

Will MIPRO-style optimization become standard?

The case for standardization is strong. Production LLM systems increasingly look like programs, not single prompts. They have modules, tools, retrieval steps, routers, validators, and agent loops. Once a system is represented as a program and a metric exists, optimizer-driven prompt compilation is a natural next step. DSPy’s own optimizer documentation frames MIPROv2, GEPA, BootstrapFinetune, and related tools as ways to tune prompts or weights to maximize metrics. dspy.ai

The case for niche adoption is also strong. Many teams do not have stable evals. Many production tasks are preference-laden, changing, or under-specified. Some teams can get sufficient gains from retrieval improvements, model upgrades, schema constraints, or manual prompt edits without paying the optimization complexity tax. And in some domains, the bottleneck is not prompt wording but tool reliability, data freshness, latency, safety review, or human workflow integration.

The most likely outcome is neither universal adoption nor obscurity. MIPRO-style optimization is likely to become standard in eval-mature teams: teams with fixed tasks, regression suites, enough examples, and repeated deployment of the same LLM program. It may remain niche in exploratory product work, one-off agents, and systems where metrics are weak or change faster than optimization can amortize.

The deeper open question is whether prompt optimization becomes a first-class part of the LLM engineering stack, like hyperparameter tuning in classical ML, or whether it is absorbed into broader agent/system optimization methods that tune prompts, tool choices, memory policies, workflow graphs, routing, and model selection together. MIPRO is an important step toward that broader compiler view, but its own evidence still supports a bounded claim: joint instruction-and-demonstration optimization can produce measurable gains when the program abstraction and eval signal are strong.

Key references

[Opsahl-Ong et al., EMNLP 2024 — Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs]: primary MIPRO paper; formalizes LM-program prompt optimization, introduces proposal and credit-assignment strategies, and reports MIPRO outperforming baselines on five of seven tasks. aclanthology.org

[DSPy documentation — MIPROv2 API and algorithm sketch]: current public DSPy documentation for MIPROv2, including joint instruction/few-shot optimization, candidate bootstrapping, grounded instruction proposal, Bayesian optimization, and mini-batch/full-eval behavior. dspy.ai

[Khattab et al., 2023 — DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines]: foundational DSPy paper; introduces LM programs as declarative modules and reports self-bootstrapped pipelines outperforming standard few-shot prompting and expert-created demonstrations in its case studies. arXiv

[DSPy documentation — BootstrapFewShot and COPRO]: implementation-level references for DSPy’s demonstration bootstrapping and instruction/signature optimization baselines. dspy.ai

[Yang et al., ICLR 2024 — Large Language Models as Optimizers / OPRO]: primary source for Optimization by PROmpting, the history-conditioned LLM-as-optimizer method that MIPRO adapts and contrasts with in multi-module settings. arXiv

[Guo et al., ICLR 2024 — EvoPrompt]: primary source for evolutionary prompt optimization with LLM-generated mutation/crossover-style candidate generation. arXiv

Companion entries

Core theory: Language Model Programs, DSPy, Automatic Prompt Optimization, Prompt Compilation, Black-Box Optimization, Bayesian Optimization, Tree-structured Parzen Estimator, Credit Assignment in LLM Systems

DSPy practice: BootstrapFewShot, Bootstrap Random Search, COPRO, MIPROv2, DSPy Metrics, DSPy Signatures, Few-Shot Demonstration Bootstrapping, Prompt Artifact Versioning

Adjacent algorithms: OPRO, EvoPrompt, Automatic Prompt Engineer, GEPA, TextGrad, AFlow, Self-Improving LLM Systems

Evaluation and deployment: LLM Evaluation Harnesses, Prompt Overfitting, Regression Testing for LLM Apps, LLM Observability, Offline Optimization vs Online Adaptation, Production Prompt Management

Counterarguments and limitations: The Limits of Prompt Optimization, Metric Hacking in LLM Systems, Prompt Transfer Across Models, Compute Budgets for LLM Optimization, Human Prompt Engineering vs Algorithmic Prompt Search