Value for Money criteria

In general terms, Value for Money (VFM) can be regarded an evaluative question about “how well resources are used, and whether they are being used well enough” (King, 2017). Evaluative questions require a judgement to be made – based on evidence, and using a transparent process of reasoning. VFM can’t be ‘measured’ using a formula – though quantitative evidence plays an important role in supporting well informed judgements.

Evaluation has been defined as “the systematic determination of the merit, worth or significance of something” (Scriven, 1991). In plainer language, the core purpose and function of evaluation is to determine “how good something is, and whether it is good enough” (Davidson, 2005). Therefore, evaluation involves making judgements: “it does not aim simply to describe some state of affairs but to offer a considered and reasoned judgement about that state of affairs” (Schwandt et al., 2016).

The fundamental problem in this endeavour is “how one can get from scientifically supported premises to evaluative conclusions” (Scriven, 1995) – in other words, how to make a sound judgement using a logical and traceable process.

Explicit evaluative reasoning provides the means to make robust judgements from evidence. Essentially, it involves developing definitions of what ‘good VFM’ looks like. Although there is more than one way to approach this task (e.g., see Schwandt, 2015; King, 2017), a widely used approach is to:

  1. establish criteria of merit, worth or significance – the dimensions of performance that are relevant and important to an evaluative judgement;
  2. define performance standards for each criterion, in order to distinguish the difference between ‘excellent’, ‘good’, ‘acceptable’ or ‘poor’ performance;
  3. gather and analyze evidence of performance against the standards; and
  4. synthesize the results into an overall judgement (Fournier, 1995).

Rubrics aren’t the only way to do this in practice, but in my experience they are are a good way (King et al, 2013). Values provide the basis for identifying criteria and standards (Schwandt, 2015). These values can come from a variety of sources, e.g., policy and strategy, past research and literature, and directly from stakeholders and communities.

What values might be important dimensions of VFM?

Often when thinking about VFM our minds turn quickly to efficiency. This can be defined in numerous ways but in the broadest sense reflects the idea of doing something that creates more value than it consumes, or provides a positive return on investment.

But there’s more to VFM than that. Just for one example, the 4Es framework also identifies economy (in plain words, minimising wastage) and effectiveness (meeting needs) as being important alongside efficiency. It even introduces a 4th E: equity.

Having multiple criteria makes VFM more complex, because it introduces trade offs. We’re no longer just thinking about whether society is better off overall, but also (for example) who in society is better or worse off, and perhaps how much improvement in equity might justify a loss of efficiency (or vice versa). It doesn’t end there. Using resources well could also mean using them ethically, lawfully, affordably, sustainably… and so on.

In June, 2016 I wrote a LinkedIn post to crowdsource a VFM criteria checklist. We can conceptualise these criteria as falling into three key groups:

  • Resource use (‘what did we put in?’)
  • Value derived from the investment (‘what did we get out?’)
  • Whether the value derived justifies the resource use (‘was it worth it?’) (King, 2016).

The following table organises VFM criteria into these categories. By understanding the resources used, the value derived from that use, and by having some basis for reconciling the two, we can make a well-informed judgement about VFM.

VFM criteria

In economic evaluation, efficiency is a comparative concept: the efficiency of an intervention is assessed relative to the next-best alternative use of resources (which could be another intervention or a ‘do nothing’ scenario). In the checklist above, efficiency is still comparative, while some other dimensions (such as ethicality and affordability) probably are not – they may be better considered in absolute rather than relative terms.

Some other concepts are more ambiguous (e.g., equity, sustainability) and the evaluator will have to make a contextual judgement about whether it is more valid to evaluate them in absolute or relative terms. For example, are we going to evaluate whether an intervention is more/less equitable than the next-best alternative intervention, or whether it meets some programme-specific requirements for equity?

Perspective matters in evaluation. The results and value of an investment – including what is considered valuable, and why – may be quite different from the perspective of affected communities than that of the investor. VFM evaluation should seek to empower the voices of those who are supposed to benefit from the investment – in determining what criteria matter, in gathering evidence of performance, and in using the criteria and evidence to make sense of performance and VFM.

There may be some overlap and ambiguity between the criteria. Nevertheless, this checklist might help us to scope a VFM evaluation and to grapple with VFM as a multi-dimensional concept. This list is not exhaustive, and not all criteria will apply in every context – but it does serve to demonstrate that determining VFM requires reference to more than efficiency alone. The “merit, worth and significance of resource use” encompasses all of the OECD DAC Criteria for Evaluating Development Assistance as well as additional factors.

References

Davidson, E.J. (2005). Evaluation Methodology Basics: The nuts and bolts of sound evaluation. Thousand Oaks: Sage.

Fournier, D. M. 1995. Establishing evaluative conclusions: A distinction between general and working logic. In D. M. Fournier (Ed.), Reasoning in Evaluation: Inferential Links and Leaps. New Directions for Evaluation, (58), 15-32.

King, J. (2017). Using Economic Methods Evaluatively. American Journal of Evaluation, March, 2017.

King, J., McKegg, K., Oakden, J., Wehipeihana, N. (2013). Rubrics: A method for surfacing values and improving the credibility of evaluation. Journal of MultiDisciplinary Evaluation, Vol 9 No 21.

Scriven, M. 1991. Evaluation thesaurus. Newbury Park, CA: Sage Publications.

Scriven, M. 1995. The Logic of Evaluation and Evaluation Practice. New Directions For Evaluation, 68, pp 49-70.

Schwandt, T. 2015. Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice. Redwood City: Stanford University Press.

Schwandt, T., Ofir, Z., Lucks, D., El-Saddick, K., D’Errico, S. (2016). Evaluation: a crucial ingredient for SDG success. Briefing: Governance, Policy and Planning. April, 2016.

May, 2017

Comments are closed.