🎯 What you will learn

Prompts are the interface between your application and the LLM. Like any code, they break silently when changed, need version history, and must be tested before deployment. By the end of this lesson you will understand:

  • Why prompts must be versioned in a registry (not scattered in String literals)
  • The PromptTemplate record with id, version, template, and params
  • Parameterised templates with {placeholder} substitution via render(Map)
  • Automated evaluation: keyword-presence assertions in JUnit 5 CI tests
  • Version guard tests — fail CI when a template changes unexpectedly
  • Advanced: LLM-as-judge evaluation pattern and A/B testing prompts
💼 Interview reality: "How do you test LLM outputs?" is a standard question in ML/AI backend interviews in 2025. The answer: soft evaluation criteria (keyword presence, structure, length), deterministic temperature=0, and version guards in CI.

🎓 Real-world scenario

🎓
VidyaPath · Bangalore, India
Prompts as versioned, testable code

VidyaPath's "explain concept" feature broke silently after an accidental prompt change. Riya introduces a PromptTemplate registry with version IDs, parameterised placeholders, and automated evaluation tests — now prompt changes are tracked in Git and validated before deployment.

📋 PromptTemplate record + registry

The PromptTemplate record stores the template string, its version, and the list of expected placeholder names. The registry maps template IDs to their current versions — one source of truth for all prompts in the application.

javac -cp ".:jackson-core.jar:jackson-databind.jar" PromptVersioning.java
PromptVersioning.java — PromptTemplate record + registryLesson 37 · prompt registry
verified output — template registry loaded
--- Registered templates --- code-review version=v1 params=[issue_type, code] explain-concept version=v2 params=[concept, level]

Why a registry? When prompts are scattered as String literals across controllers and services, a change requires grep-searching the entire codebase. A centralised registry makes every prompt discoverable, versioned, and independently testable — the same discipline you apply to database schemas and API contracts.

🤖 Rendering + calling the LLM

The render(Map) method performs placeholder substitution. If any {placeholder} remains after substitution, it throws immediately — catching template bugs at call time rather than producing a broken prompt silently.

java -cp ".:jackson-core.jar:jackson-databind.jar" PromptVersioning
PromptVersioning.java — render + LLM callLesson 37 · template v2 rendered and called with llama3.2:3b
verified output — template v2 rendered and called with llama3.2:3b
=== VidyaPath — Prompt Versioning & Tests === Template: explain-concept (v2) Rendered prompt (truncated): Explain 'Java generics' to a intermediate Java developer. Use exactly 2 sentence... Response: Java generics allow you to define reusable classes that can work with different data types, making your code more flexible and maintainable. By using type parameters in place of concrete types, you can create classes that are open for modification, yet still provide compile-time type checking and strong encapsulation. ```java public class Box<T> { private T element; public void setElement(T element) { this.element = element; } public T getElement() { return element; } } ```

🧪 Automated prompt evaluation

LLM outputs are non-deterministic — you cannot assert exact string match. Instead, evaluate on soft criteria: required keywords present, response length in range, JSON valid (if expecting JSON), no forbidden words. Run tests at temperature=0 for maximum determinism.

java -cp ".:jackson-core.jar:jackson-databind.jar" PromptVersioning
PromptVersioning.java — evaluation + JUnit 5 testsLesson 37 · automated evaluation + version guard
verified output — evaluation step from main()
--- Evaluation --- Required keywords: [type, generic, <] PASS: true

LLM-as-judge: For richer evaluation, call a second LLM with the question, expected behaviour, and the generated response. Ask: "Does this response correctly explain the concept? Answer YES or NO with a one-line reason." Parse the YES/NO and log the reason. More reliable than keyword matching but costs an extra LLM call per evaluation.

📋 Prompt lifecycle workflow

PhaseActionTool / Pattern
AuthorWrite template with {placeholders}PromptTemplate record in REGISTRY
VersionAssign id + version (v1, v2, …)Git commit for every version bump
TestKeyword assertions in JUnit 5evaluate(response, mustContain)
GuardAssert version hasn't changed unexpectedlyassertEquals("v2", template.version())
DeployMerge after CI green on evaluation testsGitHub Actions / Jenkins pipeline
A/B testRoute traffic across versions, measure qualityFeature flag + statistical significance test

📝 Quiz — 5 questions

Question 1 of 5
Why should prompts be versioned like code?
Question 2 of 5
What does the render(Map) method do?
Question 3 of 5
How do you test LLM outputs in CI?
Question 4 of 5
What does evaluate(response, mustContain) check?
Question 5 of 5
What should a version guard test check?
0
/ 5
Quiz complete!

Next: Capstone — Spring Boot AI Service →

🎤 Interview Q&A

EasyWhat is prompt injection and how do you prevent it?
+

Prompt injection: a user crafts input that overrides your system prompt (e.g., "Ignore previous instructions and..."). Prevent it by: separating system instructions from user input using structured message roles (system/user/assistant), sanitising user input, and using a model that respects role separation.

EasyWhat is temperature and how does it affect reproducibility?
+

Temperature (0.0–1.0) controls randomness. temperature=0 → deterministic (same prompt = same output, good for tests). temperature=0.7 → creative and varied (better for generation tasks). Set temperature=0 in prompt evaluation tests for reproducible results.

MediumHow would you A/B test two prompt versions in production?
+

Deploy both v1 and v2 templates. Route 50% of traffic to each using a feature flag or random assignment. Collect user feedback and implicit signals (click-through, time-on-page). After N samples, run a statistical significance test (chi-square or t-test) and promote the winning version.

MediumHow do you store prompts externally so they can be updated without redeployment?
+

Store templates in a database table (prompt_templates: id, version, template, created_at) or a config server. Load at startup via a @Repository. Add a REST endpoint POST /admin/prompts/reload to hot-reload without restart.

HardWhat is the LLM-as-judge evaluation pattern?
+

Instead of checking for keywords, call a second LLM (the "judge") with the question, expected behavior, and the generated response. Ask: "Does this response correctly explain the concept? Answer YES or NO with a one-line reason." Parse the YES/NO and log the reason. More reliable than keyword matching but costs an extra LLM call per evaluation.

⚠️ Common mistakes

Hardcoding prompts in String literals scattered across controllers
When you need to update the prompt (for better quality, new model requirements, or tone changes), you have to grep the codebase. Centralise in a registry or external config.
No evaluation tests — only manual spot-checking
Manual testing misses regressions. Add at least keyword-presence assertions to CI. Run at temperature=0 for determinism.
Changing prompts without bumping the version
Silently changing a v2 template while all tests still pass on the old expected output leads to shipped regressions. Always bump version + update evaluation criteria together.
Using full response match for evaluation
asserting response.equals(expectedString) almost always fails for LLM output because the model varies phrasing. Use contains(), evaluate(), or LLM-as-judge.

✅ Key takeaways

  • Version prompts in a registry with id + version — track changes in Git like source code
  • Parameterised templates ({concept}, {level}) separate structure from data
  • Automated evaluation: keyword checks in JUnit tests run in CI
  • Version guard tests catch unexpected prompt changes before they reach production
  • For richer evaluation: LLM-as-judge or embedding-similarity scoring
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

From prompt registries to a deployed AI service

Join the Best AI Programs — three 5-month courses, fully online, with placement assistance. Week 1 is free.

Explore our AI-era courses →