AI Agents for Evaluators

S1What an agent actually is

You already use a chat window: you type, the model answers, you copy the useful bits. An agent is what you get when you take that same model out of the chat window and wire it into your work. Strip away the hype and an agent is five parts:

• An LLM — the language model (Claude, GPT, Gemini) doing the reading and writing.
• Instructions — a standing brief it follows every time, not a fresh prompt you retype.
• Tools — things it can touch: a Google Sheet, an email account, a WhatsApp channel, a file folder.
• A trigger — what wakes it up: a schedule (every Monday 9 am), an event (a new survey row lands), or you pressing a button.
• Memory (sometimes) — what it retains between runs: last week's flags, your codebook, the running list of action items.

The mental model that serves evaluators best: an agent is a junior analyst who never sleeps but needs supervision. It is fast, tireless and consistent about formatting. It is also confidently wrong sometimes, has no stake in the programme, and will not tell you when it is out of its depth. You would never let a first-week intern send a donor report unreviewed; the same rule applies here, permanently.

When an agent is the wrong answer

Agents pay off on work that is repeated, rule-describable and reviewable. They are the wrong answer when:

• The task is one-off. If you will do it once — a single ToR, one endline analysis — just use the chat window. Building an agent for a one-off is automating a task you no longer have.
• The task is a judgement call. Deciding whether a district partner's explanation for missed targets is credible, or whether a finding is safe to publish, is your job. An agent can assemble the evidence; it must not make the call.
• The output cannot be reviewed. If nobody will check the agent's work before it has consequences — a message straight to a community member, a number straight into a donor system — do not build it. Unreviewable automation is where M&E credibility goes to die.

S2Five agents worth building in M&E

These five earn their keep in real M&E teams. Each card rates time saved, risk if wrong, and oversight needed — note that the ratings move together: the more time an agent saves on consequential work, the more supervision it needs.

S3Build walkthrough #1 — the reporting assistant (no-code)

You can build this today with a Claude Project or a custom GPT / ChatGPT Project — no external platform, no payment beyond the AI subscription you may already have. Honest framing first: this is the "sometimes memory" kind of agent. The trigger is you opening the project; the tool is file upload. What makes it an agent rather than a chat is the standing instructions — you write the brief once, and every future report starts from it.

1. Create the project. In Claude: New Project. In ChatGPT: create a Project or a custom GPT. Name it after the donor, e.g. "Quarterly Update — FCDO WASH".
2. Paste the system instructions. Use the template below, edited for your programme. This is the single highest-leverage step — be specific about format and ruthless about rules.
3. Add standing knowledge. Upload your report template and one or two past reports (with any personal data removed) so it learns the house style.
4. Each quarter, feed the indicator sheet. Export your indicator tracker as CSV, strip any names or identifiers, upload, and say: "Draft the Q2 update from this sheet."
5. Review like an evaluator. Check every number against the sheet, every claim against reality, and resolve every [DATA MISSING] yourself. Expect a good first draft, not a finished report — typically 60–80% of the writing time saved, not 100%.

ROLE
You are a reporting assistant for the M&E team of [ORGANISATION],
which runs [PROGRAMME, e.g. a WASH programme across 6 districts of
Bihar]. You draft donor updates. You never make decisions, and you
never invent information.

DATA YOU WILL RECEIVE
Each quarter I will upload an indicator sheet (CSV) with columns:
indicator_id, indicator_name, unit, annual_target, quarter_actual,
cumulative_actual, data_source, notes.

STRUCTURE OF EVERY DRAFT
1. Headline summary (max 120 words, plain language)
2. Progress against indicators (one short paragraph per indicator:
   actual vs target, direction of travel)
3. Areas off-track (list indicators below 75% of prorated target)
4. Data gaps and caveats
5. Suggested points for the narrative section (bullet points only)

RULES — NEVER BREAK THESE
- Never invent numbers. Every figure in the draft must come from a
  specific row of the uploaded sheet.
- If a value is blank or missing, write [DATA MISSING: indicator_id]
  in place of the figure. Do not estimate, interpolate or infer it.
- If the sheet and my message contradict each other, stop and ask.
- Do not speculate about WHY an indicator is off-track. List it;
  explanation is the programme team's job.
- Keep to UK spelling and the calm, factual tone of the sample
  reports provided.
- End every draft with: "DRAFT — every figure must be verified
  against the indicator sheet before sending."

S4Build walkthrough #2 — the indicator tracker (n8n or Make)

This one runs without you. It follows the fundamental trigger → action pattern of every workflow tool: a schedule wakes it, it reads your Google Sheet, an LLM step checks thresholds, and a digest lands in WhatsApp or email. Both n8n (open source, generous free tier) and Make (visual, easy start) can build it in an afternoon. Your sheet needs four columns: indicator_name, target_to_date, actual_to_date, threshold_pct.

In n8n, node by node

1. Schedule Trigger node. Set it to weekly — e.g. Mondays, 9:00 IST. This is the alarm clock; nothing else runs until it fires.
2. Google Sheets node. Connect your Google account, choose the spreadsheet and worksheet, operation "Get Rows". The node outputs each row as an item the next node can read.
3. AI / LLM node (e.g. the OpenAI or Anthropic node). Paste the threshold-checking prompt below into the system message, and map the sheet rows into the user message. The model returns a short, formatted digest.
4. Send node. Either a Gmail / SMTP node to email the digest to the M&E lead, or a WhatsApp Business Cloud node to post it to your team number. Address it to the team — never to programme participants.
5. Test, then activate. Run once manually with the sheet deliberately containing one off-track indicator; confirm the digest flags it and nothing else. Then switch the workflow to Active.

In Make, module by module

1. Scenario schedule. Create a scenario and set its schedule to once a week rather than the default interval.
2. Google Sheets module — "Search Rows", pointed at your indicator sheet, returning all rows.
3. Array aggregator. Bundle the rows into one block of text so the model sees the whole sheet at once, not one row at a time.
4. AI module (OpenAI / Anthropic / Gemini). System message: the prompt below. User message: the aggregated rows.
5. Email or WhatsApp module. Send the model's output as the digest body. Run once to test with a known off-track row, then turn the scenario on.

You are an indicator-tracking assistant for an M&E team. You will
receive rows from an indicator sheet with the columns:
indicator_name, target_to_date, actual_to_date, threshold_pct.

For each row, compute: achievement = actual_to_date / target_to_date.
An indicator is OFF-TRACK if achievement is below threshold_pct
(expressed as a decimal, e.g. 0.75).

Write a digest with exactly this structure:
1. First line: "Indicator digest — [count] of [total] off-track."
2. One line per OFF-TRACK indicator:
   "- [indicator_name]: [actual_to_date] of [target_to_date]
    ([achievement as %]) — below the [threshold]% line."
3. If nothing is off-track, write exactly:
   "Indicator digest — all indicators on track this week."

RULES
- Use only the numbers in the rows provided. Never estimate a
  missing value; if a row has a blank, list it under a final line
  "Rows with missing data — please check:".
- Do not explain why an indicator might be off-track, do not
  suggest corrective action, and do not soften or dramatise.
  You flag; the team decides.
- Plain text only. No markdown, no tables.

S5Ethics and human oversight — non-negotiables

Everything below is a condition of building agents at all, not a nice-to-have. If a build cannot satisfy these five, the build is wrong — not the rules. They also align with the ImpactMojo AI policy, which the certificate track holds you to.

• PII never leaves your control. Strip names, phone numbers, Aadhaar-adjacent identifiers, village-plus-caste combinations and anything else that identifies a person before data reaches any API or upload box. Replace them with codes you hold locally. This is doubly true in India's current data reality: the DPDP Act 2023 makes your organisation the accountable data fiduciary, and community-of-practice standards (the CoSS conversations, sector data-protection codes) are converging on the same rule — consent given for programme monitoring is not consent for processing by a third-party AI service.
• Approval gates. The agent drafts; a human sends. Every message, report or flag that leaves your team passes through a named person who can stop it. If you cannot draw the workflow with a human gate before the consequential step, redesign it.
• Hallucination checks. Every number in an agent's output must be traceable to a source row — the "no invented numbers" rule from S3. Build the check into the workflow: spot-verify three figures per output at random, and treat one invented number as a full stop, not a glitch.
• Audit trail. Log what the agent did: when it ran, what data it saw, what it produced, who approved it. A simple log sheet is enough. When a donor or an ethics reviewer asks "how was this produced?", the answer must be a record, not a recollection.
• Disclosure. Donors and communities are told when text is AI-drafted. A line in the report footer — "First draft prepared with AI assistance; all figures verified by the M&E team" — costs nothing and protects everything. Quietly passing off agent output as hand-written is how trust is lost sector-wide, not just for your organisation.

S6Your capstone: build one, document it

This module is free to read for everyone. If you are on the AI for M&E Certificate Track, the module is assessed through the capstone below — a real build, honestly documented. Reviewers reward honesty about failure far more than a polished demo that hides its weaknesses.

The brief

Pick one of the five agents from S2 — or an agent of your own design that passes the S1 test (repeated, rule-describable, reviewable) — build it with any tools you like, run it on real (anonymised) data, and submit:

1. The problem. Who has it, how often it recurs, and what it costs in hours or errors today. (Half a page.)
2. The prompt or workflow. Your full system instructions, plus screenshots of the workflow or the exported JSON from n8n/Make.
3. One real input → output example. An actual (anonymised) input and the agent's actual output, unedited.
4. Your oversight design. Where the human gate sits, what your audit log records, how PII is stripped, and how AI drafting is disclosed.
5. What it gets wrong. At least two genuine failure modes you observed, and what you do about them. "It worked perfectly" is an automatic resubmission.

Rubric

Keep going