The MEL Toolkit

A monitoring, evaluation and learning system is not a document or a database. It is the set of arrangements by which a programme decides what it needs to know, gathers evidence proportionate to that need, judges what the evidence means, and changes something as a result. Everything in this toolkit sits in one of five layers. If a layer is missing, the layers above it will produce activity without insight.

Five questions that must be answered before any indicator is written

A scope of work (also called terms of reference) is a commissioning instrument, a contract schedule and a design brief at the same time. Written well, it constrains the bidder toward good practice. Written badly, it invites a methods section that says "mixed methods" and means nothing. DFAT's 2024 to 2025 review of evaluation quality found that poor-quality plans gave little detail on how findings would be used, lacked key design elements, paid limited attention to ethics or flexibility, explained sampling strategies insufficiently, made little use of monitoring or secondary data, left data collection tools underdeveloped, and offered no clear process for judging the strength of evidence. Each of those gaps is preventable at the scope of work stage.

The twelve required sections

1. Background and contextWhat the intervention is, its budget, duration, geography, delivery chain and current status. Include what has already been evaluated so the bidder does not re-tread it.
2. Purpose, audience and intended useName the decision, the decision-maker and the date the finding is needed. State whether the exercise is formative, summative, developmental or a mid-term learning review.
3. Evaluation questionsFive to eight, not twenty. Each question should be answerable, should map to a criterion, and should have a plausible method attached. Sub-questions belong in an annex, not the main list.
4. CriteriaState explicitly which criteria apply. The European Commission assesses interventions against relevance, effectiveness, efficiency, sustainability, impact, coherence and EU added value. The OECD DAC set of six is the common floor; add value for money, gender equality and inclusion, or localisation where the commissioner requires them.
5. Scope and boundariesTime period, components in and out, geographies, and what the evaluation will not cover. Boundaries prevent scope creep and price disputes.
6. Approach and methodsSay what is required (for example, a contribution analysis, disaggregated sampling, or a specific participatory element) and what is left to the bidder. Require a sampling rationale and a triangulation strategy, not just a tool list.
7. Ethics, safeguarding and data protectionEthical review route, informed consent, handling of personal and sensitive data, safeguarding reporting line, data retention and destruction, and applicable law. In Indonesia this means the Personal Data Protection Law; in the EU, GDPR.
8. Deliverables and quality standardsInception report, tools, draft, final, and a two-page summary. Name the quality standard the deliverables will be assessed against and attach it.
9. Timeline and level of effortDays by role, with the inception phase given real weight. Under-resourced inception is the single most common cause of weak evaluation design.
10. Team compositionRequired expertise, language, national representation and independence. DFAT's standards emphasise a clear specification of the mix of technical skills and attributes required, including the role of the team leader and local team members, and require consideration of the use of local consultants with the requisite skill sets.
11. Management, governance and conflict of interestWho approves what, the reference group, and the conflict of interest declaration each team member must sign.
12. Use, dissemination and management responseHow findings will be discussed, who owns the management response, and where the report will be published. Build the response step into the timeline, not after it.

What each commissioner expects in the methods section

→ Copy the scope of work template

Isabel Vogel's review for DFID remains the reference point for how theories of change are actually used in development. It found that the element shared across all approaches is the assumptions, with the standard instruction to make them explicit, and that getting depth and critical thinking on assumptions is widely agreed to be the crux of the process. Interviewees reported that many theories of change they had seen were superficial and mechanistic, without exploring the deeper questions. The practical implication is that a theory of change workshop that produces a tidy diagram in two hours has probably failed.

The review also noted a shift toward identifying several relevant pathways to impact for a given initiative rather than a single pathway, acknowledging non-linearity and emergence, and treating documented theories and diagrams as subjective interpretations used as evolving organising frameworks.

Building one in eight moves

1. Agree the long-term changeOne sentence, in the language of the people who would experience it. Not "improved enabling environment" but "smallholder cocoa farmers earn a living income from the same land".
2. Work backwards, not forwardsAsk what must be true immediately before the long-term change, then what must be true before that. Backward mapping surfaces preconditions that forward planning skips.
3. Name the social actorsChange happens through people and institutions behaving differently. For each outcome, name who is doing something differently. Vagueness here is the leading cause of unmeasurable outcomes.
4. Surface the assumptions at every linkBetween each pair of boxes, ask what has to hold for the arrow to be real. Write these down as numbered assumptions and rate each for confidence and evidence base.
5. Map the other actors and factorsWho else is pushing on the same outcome. This is what allows a later evaluation to reason about contribution rather than assert it.
6. Identify multiple pathwaysMost programmes have two or three routes to the same outcome. Draw them. A single linear chain is usually a simplification imposed by the reporting template.
7. Write the narrativeThe diagram alone is not a theory of change. The narrative explains the causal reasoning, the evidence behind it, the contested points and what the team is uncertain about.
8. Set review pointsDiarise when the theory will be revisited, usually annually and after any major context shift. A theory of change that has never been revised has never been used.

Quality check before you sign it off

• Could a sceptical outsider identify what would have to happen for this theory to be wrong?
• Is every outcome expressed as a change in what somebody does, not as an activity in disguise?
• Are the assumptions specific enough to be monitored, and is at least one indicator attached to the riskiest of them?
• Does the narrative say where the evidence base is strong and where the team is guessing?
• Do the people who deliver the programme recognise it, and can they explain it without the diagram in front of them?
• Has it been tested against at least one alternative explanation for the change you expect?

The underlying logic is stable across systems: resources produce deliverables, deliverables are taken up and used, use produces change in behaviour, and behaviour change produces change in conditions. What varies is nomenclature and the level at which the implementer is held accountable. Getting this wrong in a proposal is an avoidable scoring loss.

The logframe matrix, and what usually goes wrong in it

The seven tests USAID applies on top of SMART

Legacy USAID Program Cycle guidance asks whether an indicator is direct, objective, useful for management, practical to collect, attributable to the intervention, timely, and adequate in combination with others. The most useful of these in practice are direct (does it measure the result itself, or a distant relative?) and practical (can the field team actually collect it at the required frequency without distorting delivery?).

Five data quality standards, applied to every indicator

USAID assesses performance data against five standards: validity, reliability, timeliness, precision and integrity. These are the questions a data quality assessment answers, and they are the right questions regardless of who is funding the work.

Indicator types, and how many you need

→ Copy the performance indicator reference sheet template

USAID's discussion note on complexity-aware monitoring is the clearest short statement of the problem. It holds that complexity-aware monitoring is appropriate where cause and effect relationships are poorly understood, which makes it difficult to identify solutions and draft detailed implementation plans in advance, and where expected results may need refinement as strategies unfold. It is a complement to performance monitoring, not a replacement for it.

Three principles

Five recommended approaches

Outcome harvesting was developed by Ricardo Wilson-Grau and colleagues, drawing on outcome mapping and utilisation-focused evaluation, and refined through real-life practice. It enables evaluators, grant makers and managers to identify, formulate, verify and make sense of outcomes.

Unlike most approaches, outcome harvesting does not necessarily measure progress against predetermined objectives. Each application is customised to the needs of primary intended users and their principal intended uses, with useful and feasible harvesting questions guiding the collection of information. The initial sources are the individuals or organisations whose actions influenced the outcomes, because they know what was achieved and are motivated to share it.

What happens in each step

1. Design the harvestHarvest users and harvesters identify usable questions to guide the harvest and agree what information will be collected as the outcome description, in addition to the change in the social actor and how the intervention contributed. Design decisions continue to be made throughout, as new information about outcomes emerges.Yield: harvesting questions, outcome description format, sources map
2. Review documentation and draft outcomesDesk review of reports, memos, media, case files, promotional material and websites to identify and formulate draft outcome statements.Yield: a first list of draft outcome statements
3. Engage with informantsHarvesters work directly with change agent informants to review the outcome descriptions extracted from files, identify and formulate additional outcomes, and classify them. Informants often consult others inside or outside their organisation who are well informed about the outcomes.Yield: refined, precise, verifiable outcome statements
4. SubstantiateObtain the views of one or more independent people knowledgeable about the outcome, or about a representative sample of outcomes, and about how they were achieved, to enhance the validity and credibility of the findings. Substantiation also enriches understanding and can surface outcomes the primary users had not identified.Yield: independently verified subset, plus new outcomes
5. Analyse and interpretOrganise outcome descriptions in a database to make sense of them, then analyse and interpret to provide evidence-based answers to the harvesting questions. Classification by actor, type, geography, significance and contribution strength is what makes patterns visible.Yield: evidence-based answers to the harvesting questions
6. Support use of findingsPropose points for discussion to harvest users, grounded in the evidence-based answers. Use is designed in from step one; it is not a dissemination afterthought.Yield: decisions, adaptations, and a documented management response

Anatomy of an outcome statement

Outcome descriptions may be as brief as a single sentence or as long as a page. They may also carry the significance of the outcome, its context, the contribution of others, diverse perspectives, and anything else useful for answering the harvesting questions. Three components are non-negotiable.

OUTCOME  (September 2025, Sulawesi, Indonesia)

CHANGE
The provincial trade agency published a revised electronic licensing
standard operating procedure that removes three duplicate document
requirements for small exporters, and began applying it in two districts.

SIGNIFICANCE
Duplicate documentation was the single largest source of clearance delay
identified in the baseline. Removing it shortens the median clearance time
and lowers the fixed compliance cost that falls hardest on the smallest
exporters. It also sets a precedent other provinces have cited.

CONTRIBUTION
The programme ran the process mapping workshop that produced the duplicate
list, drafted two of the four revision options, and funded the legal review.
The reform was championed internally by the agency's deputy head, and the
national chamber of commerce lobbied in parallel. The programme's role was
technical and enabling, not decisive.

SUBSTANTIATED BY
Deputy head, provincial trade agency (interview, 12 Oct 2025);
independent customs broker association (interview, 19 Oct 2025);
published SOP document no. 44/2025.

Nine principles behind the six steps

The steps are rooted in nine principles derived from emerging practice: five relate to process and four to content. Rather than telling you what to do, they guide the decisions you make as the harvest proceeds. Two are worth singling out.

When to use it, and when not to

Most significant change was developed by Rick Davies and refined with Jess Dart, whose 2005 guide remains the standard reference and sets out the technique in ten steps. Dart and Davies describe it as a form of dynamic values inquiry, in which designated groups of stakeholders continuously search for significant programme outcomes and then deliberate on the value of those outcomes, contributing to both programme improvement and judgement.

The ten steps

1. Start and raise interestIntroduce the technique to stakeholders and secure buy-in. MSC depends on people willingly writing and reading stories; imposed participation produces thin material.
2. Define domains of changeBroad, deliberately fuzzy categories, typically three to five, such as changes in people's lives, changes in participation, or changes in institutions. Domains are not indicators and should not be defined precisely.
3. Define the reporting periodMonthly, quarterly or half-yearly. Frequency drives the volume of stories and the workload of selection.
4. Collect significant change storiesAsk the question directly: looking back over the last period, what do you think was the most significant change in this domain, and why is it significant to you? Record the story and the reason.
5. Select the most significant of the storiesGroups read the stories, discuss them and select one, then write down the criteria they applied. This deliberation is the core of the technique.
6. Feed back the results of the selectionTell the people who submitted stories which were selected and why. Skipping this step is the most common implementation failure and it kills participation.
7. Verify storiesVisit a sample of story locations to confirm the events described. Verification protects credibility and often deepens the account.
8. QuantifyOnce a significant change is identified, ask how widespread it is. Stories can be followed by counts of how many others experienced the same change.
9. Conduct secondary analysis and meta-monitoringAnalyse the whole body of stories for themes, and monitor the process itself: who submitted, who was selected, whose voice dominates.
10. Revise the systemChange the domains, frequency or selection process in light of what the first cycles reveal. MSC is meant to evolve.

What MSC does and does not deliver

→ Copy the MSC story collection form

Running a data quality assessment

USAID practice is a useful default even outside US-funded work: indicators reported upward should be assessed for data quality at some point within a three-year period, and new indicators within six months of establishing baseline data. The assessment looks at more than the numbers. It examines the quality of the indicator definition itself, the collection instruments, the collection methods, the database management and the actual data collected.

1. Select indicators and assemble the teamPrioritise indicators that drive decisions or carry reputational risk. Include someone who did not collect the data.
2. Review the indicator definitionIs it a direct measure? Is it unambiguous about what should be counted and what should not?
3. Trace one number end to endTake a single reported figure and follow it back through the database, the submission form and the original record. Most quality problems surface here.
4. Test against the five standardsScore validity, reliability, timeliness, precision and integrity, with evidence for each score rather than an assertion.
5. Document limitations and actionsRecord what the data can and cannot support, agree corrective actions with owners and dates, and carry the limitations into every report that uses the figure.

Responsible data, in practice

Much MEL data comes from at-risk or underrepresented populations, and it influences decisions that affect those same populations. A responsible data approach that takes ethics and protection into account is not optional. At the design and planning stage, formulate the plan and budget for the full data lifecycle, including the time, systems, people and money involved, and conduct a privacy or risk-benefit assessment to flag risk areas and develop mitigations before collection starts.

Tool landscape

The problem is well documented in the MERL Tech community: dashboards fail because design principles from graphic design and user experience are not applied, and because tools are built to do all the things without being tailored enough to be usable by the most important audiences. Prototype before you build. A paper prototype of a dashboard, tested with the person who is supposed to act on it, saves months.

Eight rules we apply

1. Name the audience and the decision cycle firstA board that meets quarterly needs a different artefact from a field manager who acts weekly. Build two views, not one compromise.
2. Lead with a verdict, not a chartThe top of the screen should say what the situation is in words. Charts justify the verdict; they are not the verdict.
3. Show exceptions, hide the compliantIf an indicator is on track, the dashboard should be quiet about it. Attention is the scarce resource.
4. Pass the five-second testShow it to someone unfamiliar for five seconds and ask what they took away. If they cannot answer, the hierarchy is wrong.
5. Cap the filtersTwo filters, chosen deliberately. Every additional control transfers analytical work to a user who does not want it.
6. Put disaggregation one click awayEquity questions are answered by breakdowns. If the breakdown is three menus deep, nobody will look.
7. Date and source every tile"As at" date, source system, and the data quality flag if one is open. Trust collapses the first time a number cannot be explained.
8. Prototype on paper, then buildTest a sketch with the actual decision-maker. Build only what survived the test.

Five principles

Seven evaluation phases

The FRAME checklist

Tick as you go. The counter is for your own tracking and resets when the page reloads, so print or export before you close it.

USAID's collaborating, learning and adapting framework treats strategic collaboration among internal and external stakeholders, continuous learning and adaptive management as the connective tissue between all components of the programme cycle. The 2026 Magenta Book update makes a comparable move for UK government evaluation, shifting emphasis from measuring what happened toward learning how to improve in real time, with new guidance on the responsible and ethical use of AI in social research and evaluation and a new section on place-based evaluation.

Four mechanisms that actually produce change

Designing for use from the start

• Identify the primary intended users by name at inception and involve them in setting the questions.
• Ask each of them, at the start, what finding would change their decision. If nothing would, rescope.
• Deliver an emerging findings session before the draft report, so the first time a commissioner meets an uncomfortable finding is not in writing.
• Write recommendations that name an actor, an action and a timeframe. "Strengthen coordination" is not a recommendation.
• Limit recommendations to those the commissioner has the authority to implement, and mark the rest as being for other actors.
• Publish. Evidence that stays inside the commissioning organisation cannot be built on by anyone else.