AI for Evaluators: Prompt Patterns & Guardrails

A practitioner reference for using AI and large language models (LLMs) responsibly across the monitoring, evaluation & learning (MEL) cycle

How to Use This Handout

AI language tools can speed up parts of an evaluator's work, but they do not do evaluation. They generate plausible text; they do not judge evidence, understand context, or bear accountability. Treat every AI output as a rough draft from an over-confident junior colleague who has never met your respondents. This handout is a companion to the flagship course "AI for Impact: Data, Monitoring & Evaluation."

Vendor-neutral: The patterns apply to any general-purpose LLM assistant. No tool is endorsed; capabilities and limits shift with each model release.
Human-in-the-loop by default: Everything here assumes a qualified evaluator reviews, corrects, and signs off on the output.
Privacy first: Nothing that could identify a respondent should ever be pasted into a third-party tool. See the guardrails section.

1. Where AI Helps — and Where It Doesn't

AI assistance is strongest for drafting, restructuring, and first-pass processing of text you already have. It is weakest — and most dangerous — where accuracy, judgment, representativeness, or accountability are at stake.

MEL stage	Good uses (with human review)	Risky uses (avoid or heavily verify)
Evaluation design	Brainstorm evaluation questions; draft a theory-of-change narrative from your bullet points; suggest indicators to consider; outline a methods section	Letting AI choose the design or claim what is "rigorous"; generating sample sizes or power calculations without a statistician's check
Instrument drafting	Draft survey items or interview guide questions from your constructs; check items for double-barrelled or leading wording; suggest response scales	Assuming items are validated; skipping cognitive/pilot testing; generating "culturally appropriate" wording without local review
Qualitative coding	Propose a first-pass codebook; suggest themes across transcripts; flag candidate quotes for a theme — all as a starting point	Treating AI codes as final; reporting AI-generated themes without human validation or inter-coder checks
Transcription & translation	Rough transcripts and draft translations to speed a human transcriber/translator; glossary suggestions	Publishing machine transcripts/translations verbatim; trusting them for low-resource languages, dialects, code-switching, or technical terms
Summarization & synthesis	Condense long documents you provide; produce a first synthesis of findings you supply; reformat notes into a matrix	Synthesizing sources the model wasn't given (it will invent them); losing minority/dissenting views; over-smoothing disagreement
Report drafting	Draft an executive summary from your findings; tighten prose; adjust register for different audiences; draft recommendations you then vet	Any factual claim, statistic, or citation generated by AI — these must be traced to source data; passing AI text off as independently authored
Data cleaning	Draft cleaning/recode scripts (that you read and run yourself); explain an error message; suggest validation rules	Pasting raw datasets with PII into a tool; letting AI "clean" data invisibly; trusting generated numbers or aggregates

Rule of thumb: AI is useful when you supply the substance and it reshapes the form. It is hazardous when you ask it to supply the substance — facts, figures, citations, or judgments — because it will produce fluent, confident, and sometimes entirely fabricated content.

2. Prompt Patterns That Work

A useful prompt is explicit about five things. Vague prompts get generic, hedged, or invented answers; structured prompts get usable drafts.

The five building blocks (Role · Context · Task · Format · Examples)

Role: Tell the model who to act as ("You are assisting a program evaluator...").
Context: Give the relevant background — sector, program, audience, constraints. Paste the source material rather than describing it.
Task: State one clear job. Split multi-part work into separate prompts.
Format: Specify the output shape — table, bullet list, word count, columns, tone.
Examples (few-shot): Show one or two worked examples of the input-to-output mapping you want. This is the single biggest quality lever for coding and formatting tasks.

Techniques that raise quality

Ask for reasoning: "Explain your reasoning step by step before giving the answer" — helps you spot where it went wrong.
Constrain the output: "Only use the text I provided. If the answer isn't there, say 'not stated'."
Iterate: Treat it as a conversation — critique the draft and ask for revisions rather than expecting one perfect answer.
Ask for uncertainty: "Flag anything you are unsure about" surfaces weak spots.

Anti-patterns to avoid

Asking open-web questions expecting current facts — models can be outdated or wrong.
Requesting citations or statistics "from the literature" — these are frequently fabricated.
Bundling five tasks into one prompt — quality drops.
Accepting the first answer without asking "what did you assume?"

Concrete example prompts

Example A — First-pass thematic coding Role: You are assisting a program evaluator with qualitative analysis. Context: Below are 30 open-ended survey responses to the question "What changed for you after joining the livelihoods program?" Task: Propose an initial set of 5-8 themes. For each theme give a short label, a one-line definition, and 2 example response numbers. Constraints: Use ONLY the responses provided. Do not invent responses. Flag any response that fits no theme. This is a FIRST DRAFT for a human coder to review, not a final codebook. Format: A table with columns Theme | Definition | Example response #s. [paste the 30 de-identified responses here]

Example B — Draft a TOR section Role: You are helping draft a Terms of Reference for an external evaluation. Context: Program = 3-year adolescent-girls education project, 4 districts. Audience = prospective evaluation consultants. Task: Draft the "Evaluation Objectives and Questions" section only. Base it on the theory of change and objectives I paste below. Format: 200-300 words plus a bulleted list of 5-7 evaluation questions grouped under relevance, effectiveness, and sustainability. Note: I will review and edit; do not invent program facts not given. [paste objectives and theory of change]

Example C — Synthesize interviews, preserving disagreement Role: You are assisting an evaluator synthesizing key-informant interviews. Context: Below are notes from 12 KIIs (de-identified) on why uptake of the new referral system was uneven across facilities. Task: Summarize the main explanations. IMPORTANT: preserve points of DISAGREEMENT between informants rather than averaging them out. Note where only one or two informants raised a point, and where views conflict. Constraints: Use only the notes provided; attribute nothing beyond them. Format: (1) Areas of consensus, (2) Areas of disagreement with the competing views, (3) Outlier points raised by single informants. [paste 12 de-identified KII note sets]

Example D — Reframe a finding for a lay audience Role: You are an editor helping make an evaluation finding accessible. Task: Rewrite the finding below for a community feedback meeting: plain language, no jargon, about 120 words, respectful and non-blaming tone. Constraint: Do not add, soften, or exaggerate any result. Keep every number. [paste the finding as written in the report]

3. AI-Assisted Qualitative Coding — A Disciplined Workflow

AI can accelerate qualitative analysis, but it must sit inside a human coding process, never replace it. Use it for a first pass; the evaluator remains the analyst of record.

A defensible AI-assisted coding process

Human frames the analysis. Define the research questions and any a-priori codes yourself before touching AI.
AI first pass. Ask AI to propose themes/codes from a de-identified subset, as a draft only.
Human validation (mandatory). Read the transcripts yourself. Accept, merge, rename, split, or reject every AI-suggested code. Confirm each example quote actually supports its code.
Inter-coder check. Have a second human code a sample independently; compare and reconcile. AI does not count as the second coder.
Re-code with the agreed book. Apply the human-agreed codebook to the full corpus; use AI only to speed clerical steps, spot-checking its output.
Audit trail. Keep the prompts used, the AI drafts, and the human changes, so the analysis is reproducible and reviewable.

Never treat AI codes as ground truth. The model has no access to tone, context, or what the respondent meant. It can mislabel sarcasm, miss culturally specific meaning, drop minority voices, and confidently apply a code that the text does not support. Every code is a hypothesis for a human to confirm against the source.

South Asian context

Multilingual data, code-switching (e.g. Hindi-English, Bangla-English), regional idioms, caste/gender-sensitive phrasing, and indirect speech are easily flattened or misread by models trained predominantly on other languages and contexts. Validate coding of translated material against the original language with someone fluent in it.

4. Guardrails

4.1 Hallucination & fabrication

The core failure mode. LLMs generate the most probable next words, not verified facts. They will invent statistics, study citations, quotes, and program details that never existed — fluently and confidently.

Mitigation: Never accept an AI-generated fact, number, citation, or quote. Trace every one to your own source data or documents. Constrain prompts to supplied material ("use only the text I gave you; say 'not stated' otherwise"). Spot-check quotes against transcripts verbatim.

4.2 Bias & representativeness

Skewed by training data. Models reflect the biases of their training data and can under-represent marginalized groups, minority views, and non-dominant languages. In synthesis, they tend to over-weight the majority and smooth away dissent.

Mitigation: Explicitly prompt to preserve minority and dissenting views. Cross-check whose voices appear in the output against your sample. Have local experts review AI-shaped content for cultural and gender sensitivity. Do not let AI decide what is "representative."

4.3 Data privacy, consent & confidentiality

Do not paste personally identifiable or identifiable respondent data into third-party AI tools. Names, phone numbers, exact locations, GPS, ID numbers, photos, rare-combination attributes, or any detail that could re-identify someone. Consent given for an evaluation almost never covers sending data to an external AI vendor, whose terms may retain or train on inputs.

Mitigation:

De-identify before you prompt: strip names and direct identifiers; generalize locations; remove rare-attribute combinations that enable re-identification.
Check consent & ethics approval: confirm your consent language and IRB/ethics clearance permit AI processing; if unsure, don't.
Prefer safer channels: use enterprise/no-retention or self-hosted options where available; read the tool's data-retention and training terms.
Honour confidentiality: commercially sensitive or embargoed program data is subject to the same caution as personal data.

4.4 Over-reliance

Fluent output invites automation bias — the temptation to trust it because it reads well. Skills atrophy and errors propagate when AI drafts go unchecked.

Mitigation: Keep the human as analyst and author of record. Use AI to draft and speed, not to decide. Budget real review time; if you wouldn't have time to check it, don't generate it.

4.5 Transparency & disclosure

Disclose AI use. State in your methodology how AI was used (e.g. "AI assistance was used for first-pass coding and drafting; all outputs were reviewed and validated by the evaluation team"). Follow your commissioner's, employer's, and any professional evaluation association's guidance. Transparency lets others judge the reliability of the work.

Responsible-Use Checklist — run before and after every AI-assisted task

I removed all PII and identifiable respondent data before prompting.
Consent and ethics approval cover this use of the data.
I checked the tool's data-retention/training terms and used the safest available option.
I supplied the substance; I did not ask AI to invent facts, numbers, or citations.
I traced every factual claim, statistic, and quote in the output back to source.
A qualified human reviewed and corrected the output; AI codes/themes were validated, not accepted.
Minority and dissenting views were preserved, not averaged away.
Local/linguistic experts reviewed culturally or language-sensitive content.
I kept an audit trail of prompts, AI drafts, and my edits.
I disclosed how AI was used in the methodology.
A human remains accountable for the final findings and recommendations.

5. Limitations & Ethics

Issue	What it means for evaluators	Practical response
Reproducibility	The same prompt can yield different outputs; results are not deterministic.	Don't rely on AI for anything that must be exactly reproducible. Record prompts and outputs; validate with human coding.
Model drift & versioning	Models are updated or retired; behaviour changes over time, so a workflow may not replicate later.	Note the tool and approximate date in your methods. Re-validate workflows periodically.
Opacity	You usually cannot see why a model produced an output.	Require reasoning in the output where it helps; base conclusions on evidence you can inspect, not on the model's authority.
Context limits	Very long documents may be truncated or partially "read."	Chunk inputs; confirm the model actually used all material; spot-check coverage.
Equity of access	Tool cost and language coverage can advantage some teams and languages over others.	Be mindful when comparing AI-assisted and non-assisted work; don't let tooling shape whose evidence counts.

The human-accountability principle

AI is a tool, not an evaluator. It cannot be accountable to communities, commissioners, or professional standards. A named human must own the design decisions, the interpretation of evidence, and the final judgments. When AI is used, accountability does not transfer to the tool — it stays with the evaluator. Use AI to work faster and see more, never to decide what is true or what should be done.