LLM Prompt Optimization for Legal-Clause Classification

Nobody reads Terms of Service - yet they contain clauses that genuinely matter: the right to change terms unilaterally, mandatory arbitration, limitation of liability, an account that can be closed without notice. A system that can identify those clauses automatically lets people see what they are agreeing to.

The obvious approach is to ask a language model, but off-the-shelf prompts gave uneven results: strong on some clause types, weak on others. Retraining a model for the task is expensive and out of reach for most organisations.

Hence the research question: how far can you get by systematically optimising the prompt alone, without touching the model?

As part of a practical research course at the Technical University of Munich, I ran a comparative evaluation of automatic prompt-optimisation methods on legal clause classification.

1. A real legal dataset. I worked on CLAUDETTE, an academic dataset of Terms-of-Service clauses labelled fair or unfair. I set up two configurations: binary (fair / unfair) and nine-class multi-label - arbitration, unilateral change, content removal, jurisdiction, choice of law, limitation of liability, unilateral termination, contract by using.
2. Honest baselines first. To claim prompt optimisation adds anything, you have to know what it must beat: a classical TF-IDF + SVM, LegalBERT (a transformer pre-trained on legal text), and plain zero-shot and few-shot prompting.
3. Four automatic optimisation methods compared. PromptAgent (Monte Carlo tree search over the prompt space), EvoPrompt (evolutionary search), a textual-gradient approach (improving the prompt from the model's own error analysis), and OPRO - all evaluated on the same data, the same splits and the same metrics.
4. Output made measurable. The model was run to return schema-constrained JSON - class label plus rationale - rather than free text, so results could be scored automatically on accuracy, precision, recall and F1 without manual interpretation.

Result: the best automatically optimised prompt moved accuracy from 0.650 to 0.755 and F1 from 0.622 to 0.755 - roughly a 21% relative gain in F1, with no training step applied to the model.

The practical takeaway: the quality you get from a language model depends heavily on how you ask, and that "how" can be found by measurement rather than intuition. The same approach transfers to any classification task where changing the model isn't in the budget.

Stack: Python, Llama-3.3-70B and 3.1-8B (API), PromptAgent, EvoPrompt, OPRO, LegalBERT, scikit-learn, schema-constrained JSON output.