employing additional staff can improve service delivery?

ebscohost.pdf

Home >Social Science homework help >employing additional staff can improve service delivery?

5 5

chapter 3

OPEN INQUIRY AND AUDIT REVIEW LOCAL AND CENTRAL AUDIENCES FOR EVALUATION We all know the litany recited to encourage us to do evaluation! It is to help us meet our aims and objectives ef� ciently and effectively, assist us with planning and accountability, identify both what we’ve achieved and what we haven’t, improve our activities, or support our next funding application!

However, some of the worst confusion that arises in evaluation stems from this common con� ation of the needs of two different audiences. One of these audiences typically may be the ‘local’ audience of on-the-ground providers and users of a service or activity. This audience needs to know the value of what they are doing in order to know how to improve or maintain it in their daily practices. The other of these two audiences typically may be the ‘central’ audience of those who are responsible for receiving and spending public monies on services and activities that are deemed appropriate and necessary by the elected or appointed representatives of the public interest.* This audience needs to know what is being done primarily in order to monitor for publicly approved appropriateness and value. The local audi ence is made up of the direct doers or participants. The central audience is primarily comprised of the auditors and transmitters and processors of information about the ‘doing’ further up the hierarchy and also of policy directions that come down the

* These comments also apply to those who spend private monies such as philanthropic trusts or community foundations. In their case, the forms of accountability are more discretionary and there is no system of public accountability via elected representatives.

TWO APPROACHES TO EVALUATION

Everyday Evaluation 3e.indd 55 19/1/11 9:16:21 AM

Co py ri gh t © 2 01 1. R ou tl ed ge . Al l ri gh ts r es er ve d. M ay n ot b e re pr od uc ed i n an y fo rm w it ho ut p er mi ss io n fr om t he p ub li sh er , ex ce pt f ai r us es p er mi tt ed u nd er U .S . or a pp li ca bl e

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

5 6 E V E R Y D AY E VA L U A T I O N O N T H E R U N

hierarchy from the elected representatives’ assess ment of public needs and wishes for new ‘doing’.

How you go about handling the requirements of central audiences (such as funders, administrators and any other authorities) may be different from how you respond to the needs of local audiences (such as local service-users or providers). For example: • There may be differences in the scope and detail—more about detailed areas needing

change and improvement for locals; more about ‘reporting in’ in a more summarised or abstracted way for central people (perhaps on things you are now taking for granted).

• Another difference may be the kind of study called for—open and change oriented for local people; ticking off against a checklist of objectives in a service agreement or standards manual for central people. Sometimes, however, these requirements can be reversed. Central authorities may

assist local services to be change oriented or responsive to users. In Australia, the Victorian Education Department’s School Improve ment Plan, the State Land Care Plan, the Depart- ment of Victorian Communities’ capacity-building and the Australian Taxation Of� ce’s restructuring called ‘People Action’ are good examples. Private Trusts or Community Foundations’ Deeds of Trust may commit them to community welfare, education or justice activities over longer periods of time. This can mean they have a greater commitment to local efforts to use evaluation for improvement.

On the other hand, local service-users and provid ers will sometimes merely want to monitor their own performance against their intended program logic using some standard criteria or broad statistics. Examples of this would be when Committees of Management use the Community Health Accredita tion and Standards Program manual (formerly CHASP now QUIPs) or the micro-computer Community Health Information and Research System (pcCHIRS) to circulate quarterly statistics reports or write up an Annual Report for their members.

There has nevertheless been an overall trend for evaluation to be used more as an ‘audit’ or ‘review’ mechanism by centralised administering authorities (as well as some local service managers) who primarily want to be assured that funds are being used for the purposes for which they were provided, and to check that services are doing what they said they would do. However, local users’ and providers’ interest in using evalu ation as an improvement method to check how or why something ‘worked’ or (didn’t work), and to learn from it to see how to do it better, is increasingly being ‘built in’ to central funding authorities’ work. In some areas, these two activities are at last being recognised as two halves of a cyclic action evaluation process where better attention to quality improvement processes decreases overly heavy reliance on after the event quality assurance measurement.

Indeed, some evaluation theorists would go so far as to say that audit, review or assur- ance activities are not evaluation at all, as they aren’t evaluating whether the services are actually of value, but only whether they are being done as promised, according to frame- works of pre-agreed logics and indicators of what was of value at that time. For example, the intended actions may have been taken, but change has rendered them less valuable

Everyday Evaluation 3e.indd 56 19/1/11 9:16:21 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 5 7

Everyday Evaluation 3e.indd 57 19/1/11 9:16:21 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

5 8 E V E R Y D AY E VA L U A T I O N O N T H E R U N

(a false positive). Or they are not being taken, but that turns out to be not such a bad thing (a false negative). However, because such frame works are intended to operate as implicitly valued objectives (which relate back to meeting service-users’ needs), formal ‘top-down’ evaluation-against-objectives techniques are examined in this chapter along with the more developmental ‘bottom-up’ every day kinds of approaches that identify what is of value in the � rst place. The strength of the value of objectives-based logical frameworks (logframes) will rest, therefore, on the strength of the previous cycles of research that led to them. Where this previous research was weak or badly theorised, or inadequately grounded in critical reference groups’ perspectives, then the objectives-based logical frameworks will also be � awed. Sometimes the higher purposes or more abstract philosophical principles are better carriers of accrued wisdom about value than the increasingly more highly speci� ed aims and targets that are so popular but often offer a false appearance of certainty about what will be of value.

It can now be seen that these two approaches represent two different perspec tives on the same evaluative research cycle (described in the previous chapter)—one looking ‘back’ at previously recommended and enacted practices to check whether they hap pened; the other looking ‘forward’ to a problem-posing and problem-solving effort and as yet unexplored and potentially valuable future practices. The bulk of this chapter examines these two different approaches to evaluation—recognising that both might be required by either local or central audiences, and ideally should be integrated into one evaluation design so each might draw on the other. The new challenge becomes one of retaining local and

central parties in a partnership working together around the cycles of ‘up close and personal’ inductive observation to abductive re� ection and ‘bigger picture’ contextualised theory-building about what would

Everyday Evaluation 3e.indd 58 19/1/11 9:16:24 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 5 9

be of more value (and why), to deductive � eld-based ‘experimentation’ on implementing the new valued actions and bedding them down and monitoring them as a culture of quality practice, without splitting off into a ‘division of labour’ and risking Chinese whispers or becoming a Tower of Babel as each speaks the language of only one of these discourses, and loses the whole.

EVALUATION FOR REPORTING AND ACCOUNTABILITY

Before moving on to consider the two approaches in more detail, it is worth pausing for a moment to consider further the risks of separating evaluation used as a reporting and accountability mechanism, from evaluation processes and � ndings at the local level for quality improvement—especially when those who are central (or otherwise elsewhere) do not know directly what is going on ‘on the ground’, and those who are local do not know what is being planned or introduced ‘from on high’.

A key practical issue from the perspective of the local level is, ‘How can central or other audiences best know what is being done down here without us spending a disproportionate amount of time doc umenting, recording and reporting?’

A key practical issue from the central level is, ‘How can we best know that local users and provid ers are getting and giving the most effective services or activities possible (a) given the available total pot of money, and (b) given the total range of needs and demands?’

A key political issue resulting from separating out evaluation for accountability from evaluation for improvement is that the former generates an inevitable pressure to demon- strate achievement and to show progress towards de� ned goals, playing down areas where there are problems and showing things in their best light by narrowly orienting to set achievement indicators—usually abstracted as snapshot quanti� cation. Evaluation for improvement, on the other hand, assumes there is room for learning, change and improve- ment, or the possibility of better achievement indicators, and thus concentrates on where there are still discrepancies between intentions and practice. People need to feel free to share their fears, worries and vulnerabilities, identify problems, come up with what might be done instead, work out solutions and carry out ‘in-� ight correction’ (Lynton Brown’s term) through several time point iterations, and then report on ‘whole stories’ of moving through such processes. These recursive developmental processes can be suppressed by too-regular surveillance or overly close adherence to pre-set logics of achievement objectives. Interest- ingly, when service agreements were introduced, they worked well as a genuine technique of evaluation for their � rst year or two when they were seen most clearly as experimental and provisional. Once pinned down, they largely become techniques for promising not just more of the same, but also for lowering sights to the achievable rather than the desirable.

While local audiences may always be more preoccupied with dynamic ‘case studies’ of local quality, and central audiences may be more preoccupied with counting and comparing services, there are nevertheless several possible strategies of mutual bene� t to both local and central levels, to try to integrate these perspectives and methods.

Everyday Evaluation 3e.indd 59 19/1/11 9:16:25 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

6 0 E V E R Y D AY E VA L U A T I O N O N T H E R U N

Direct contact

Direct contact is where the central audiences make direct site visits. For all its drawbacks, the old ‘inspector’ system (for example, in schools and infant welfare centres) had the irreplaceable value of giving direct observational data, and a potential chance to open up questioning and exploration with local practitioners. The best of the old inspectors would have been ‘critical friends’, using a quasi-anthropological but collaborative approach to gaining understanding, although the ‘friendship’ would always have been limited by powers to cut funds, affect promotion, and so on. Yet, in turn, these powers are limited by the need to ensure needs are met. Government ministers’ � eld visits with their exec- utive of� cers to services are another example of direct contact. On-site presentations of local reports in relation to service agreements for a Victorian gov ernment funding scheme is another example, as are the networks of regional pre-school child develop- ment or health promotion or other kinds of adviser positions. With a more decentralised self-organising approach, we might even call this role one of facilitation—and, like adult education, the learnings are mutual.

Vertical slice Vertical slice is a variation of direct contact that involves a group of of� cers from all levels from ‘top’ to ‘bottom’ meeting in small groups to collaborate on evaluation for policy development, re� ecting on monitoring and reviewing, or service problem-solving or vision-development. This has some shared features of focus groups, although some vertical slice groups may be ongoing collaborative inquiring groups. This approach exposes each to the other’s perspective—those both furthest from and closest to ministerial policy and funding levels and management, as well as to critical reference groups such as consumers or communities ‘on the ground’.

Network or tree A network or tree is often a feature of more con ventional management structures—whether corporate or hierarchical bureaucratic—and involves those at the ‘top’ or periphery being part of an intelli gence network that means that there is an assured chain of links from them to the centre. For example, there could be regional of� cers who do direct site visits and have good local knowl edge, and central of� cers who have direct contact with these regional of� cers, and central managers with direct access to these central of� cers, and so on. Certainty and reassurance lie in the knowledge that others know the detail (rather than trying to collect and deposit it all centrally). A major drawback is ‘editing’ that might take place going up the line (as in the famous pass-the-message-along-the-line Chinese whispers group dynamic exer cise), and inappropriate or poorly informed direction going back ‘down the line’.

Everyday Evaluation 3e.indd 60 19/1/11 9:16:25 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 6 1

Records/written documents The most unsatisfactory tendency is for central agencies to collect volumes of statistics that are stored in the name of ‘risk management’—in the past in endless � ling cabinets and now on endless computer drives—but rarely consulted unless a query arises. Some- times services and commu nity groups experience this as being a little like having to take aspirin every day in case their funder gets a headache! Such reporting should be minimised in the interests of service ef� ciency (see above for a more ‘distributed intelligence system’).

This would mean illuminative insights into what is done from natural records—such as those the service or activity would generate as a matter of course, like annual reports, etc.—could be read for meaning more easily. This may be enhanced by the provision of some consultancy to resource deeper re� ection. Narrative forms of ‘storytelling’ reporting, even in the form of the well-chosen anecdote, may also be more illuminative (‘We noticed such and such, so we did this and found that, and then decided then to try; and that led to . . .’, etc.) than resorting only to surveys with their many possible misinter- pretations. Visual presentations, videos, photos, and so on may also have more of a place than was once recognised.

To provide comparative planning data, central agencies should consider what is the minimum they need to know in order to divide up funds between regions or local areas, and leave it to smaller scale units to know more accurately whether a policy is being met. Thus local- or region-relevant data should be sought. Within a locality or region, staff should consider what is the minimum they need to know in order to divide up funds or know whether policy is being met. This would condition and limit the kind of data at this level, and overcome the kinds of problems typi� ed by the Herculean efforts being made by central agencies to collect standardised case information continuously about every single individ ual encounter between a service provider and service-user, in all services within a program, across whole states or nations!

Contractual assurances Contractual assurances typically have involved central agencies putting in place funding and service agreements requiring a set of requirements and commitments that are subject to regular change and revision in the light of changing purposes or long-term goals. Short- term operational objectives are always means to these changing ends, and should not be treated only as ends in them selves. They need not be overly detailed, but should provide a framework for looking at a year or two or three years of activity with a view to making a little more conscious and explicit where the local people are coming from and heading to, and why. Practice has demonstrated that such written agree ments work best if they are in a context of direct contact between central and local audiences (includ ing direct contact with consumers), and focus more on what is needing to change, leaving the remainder as more

Everyday Evaluation 3e.indd 61 19/1/11 9:16:26 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

6 2 E V E R Y D AY E VA L U A T I O N O N T H E R U N

or less ‘tried and true’ (so far). One contractual assurance might be to expect (and require) sensible reporting on putting in place more organically developed, grounded and change- oriented local inquiry-type evaluation.

WHAT IF YOU ALREADY KNOW THE VALUE OF WHAT YOU ARE DOING? Like all research, no useful evaluation can be carried out in the absence of a genuine unanswered ques tion. What is essential is that you work out whether you have a question or issue without a known answer, or whether you already know the answer. If you have a genu- inely unresolved problem, you will then be setting out on an exciting voyage of discovery—an inquiry into the unknown. If you already know the answers (or can � nd them out pretty easily), then you will be in the less-exciting but sometimes just as satisfying business of marshalling existing data or understandings—and carrying out a careful exercise in validation. We face a kind of fork in our path which Diagram 1 (see page 64) sets out.

So work out the answer to the question ‘Why do we want to evaluate?’ Keep going through the process in Diagram 1 (or use this as a map if you get lost) until you have satis� ed yourself that you either need to do a problem-posing evalu ation, or more of a validation and documentation evaluation (or a mixture of both, but be clear about which bits are which kind).

In terms of the evaluative research process cycle (see the wall chart on the Allen & Unwin website), the difference is between embarking on a new cycle of self-conscious evaluation and reporting on a cycle of evaluation already carried out (perhaps not so con sciously).

Either way, you will � nd that the question with which we started—‘What do I think of my service (or whatever)? How is it going?’—gets asked, but this time you will be clear about what has preceded the question and whether or not you have an answer.

EMBARKING ON NEW EVALUATION: OPEN INQUIRY AND/OR AUDIT REVIEW If, on the other hand, you cannot yet say that you do know the value of what you are consid- ering, we return to consider in more detail the two approaches to evaluation: change and improvement-oriented ‘open inquiry’, or a check on whether or not we have done what we said we’d do (e.g. by auditing or reviewing against a logframe of goals and objectives and their expected outcomes).

Recalling that evaluation proceeds initially from a sensed discrepancy between the way things are and an image or description of what was expected, we can see that ‘what was expected’ might be either: • something that was explicitly and consciously expected—something we realised we

valued, wanted and knew we were looking for (maybe even that we had already written down and planned to do), or

Everyday Evaluation 3e.indd 62 19/1/11 9:16:26 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 6 3

• something that was implicitly but not consciously expected—something we only realise in retro spect was what we must have valued, wanted, desired or hoped for (or not valued, desired or wanted, etc.)—that is, something we register mentally as a welcome or unwelcome surprise.

Reviewing on the basis of previous conclusions When we are comparing current practice with what we consciously planned or expected, then what we are doing is systematically looking back over our shoulder at the conclu- sions of a previous cycle of research that were turned into recommenda tions and explicit objectives. For each of these, we can identify the precedents in our past experience: ‘We did such and such in order to achieve xyz, and then discovered it had resulted in abc . . . We then realised such and such had happened, and concluded such and such . . . and then recommended that we try . . .’ We have thus provisionally already decided the value of doing whatever it is, and now are checking to see if we are accomp lishing or have accomplished what we set out to.

We are checking current practice off against previously articulated hopes or expectations (‘objec tives’ or ‘standards’). For example, we may have de cided (as a result of previous practice) that we should have objec- tives a, b and c, according to principles x, y and z. Now when we look at our current practice (descriptions of the world), we can ask, ‘Are we doing “a” according to x, y and z. Are we doing “b” according to x, y and z? And are we doing “c” according to x, y and z?’ See Diagram 2 on p. 66 for a typical set of service objectives and philosophical principles, and the kind of rational grid of evaluation questions it allows us to generate. This is all pretty logical stuff that lets us return to all those previously identi� ed conclusions and use them systematically as our current evaluation criteria.

I am calling all these kinds of retrospective evaluation ‘audit’ or ‘review’-type evaluation. It is important to reiterate that this can be done at any time or point in the life of an

activity or service. It does not have to be left until the end, and it doesn’t necessarily equate with ‘summative’ eval uation. For example, when you are about to do something new, you can run it through your mind against the goals and objectives already chosen: ‘Will this action accomplish these goals and objectives with these outcomes?’, or ‘How will this accomplish these goals, objectives and outcomes?’ This would ensure evaluation was built in from the outset and whatever actions are taken would be consistent with initial intentions, purposes and logical ways that have been calculated to get you there.

Everyday Evaluation 3e.indd 63 19/1/11 9:16:26 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

To start with ask:

Reflection

Now this is really the most important question of all since we do research and evaluation which stem from our values and what we are trying to achieve in the world. Sorting this out clearly will orient the evaluation in the right direction so the outcome will be useful, and not a waste of time. There may be two kinds of reasons. Work out whether it is for:

What is there about this that you don’t understand? What bewilders you about this? What do you genuinely not understand? If you already know the answer then assess the evidence, etc. With regard to what you do not know, ask it as a question:

Internal reasons ‘We want it’

Identifiable problem No identifiable problem

Why do we want to evaluate our project, service, etc?

External reasons ‘They want it’

‘A problem has come up and we need to solve it’ or ‘We feel uneasy about’ or ‘We need it because x... isn’t going as well as we expected’, etc.

‘Seems like a good idea’ or ‘We thought we could do it routinely/because it’s been going for a while/on an annual basis’, etc.

‘They said we have to’ or ‘We have to, to keep getting funded next year’ or ‘It’s a condition of subsidy’ or ‘Because they say it’s not working’, etc.

Identify who we are

Identify who has the

‘problem’

Have a think about how much convincing evidence (extent and kind) we will require

What is the nature of the

‘problem’? What is the difference between the way things are now and

the way you expect them to be, or thought

they should be?

Have a think about how much convincing

evidence (extent and kind) ‘they’ will require

Identify who ‘they’ are

With regard to what you and/or they want to evaluate, ask:

Now. Assess this evidence. Is it rigorous and comprehensive enough? Will it convince others?

How is it going? Is it working?

How do we know it’s going well? What makes us conclude this?

‘Well no, not entirely, perhaps it isn’t going entirely well.’

‘Not sure, perhaps not, perhaps there is a problem.’

‘Well is it like ..... instead of ....’ or ‘We are doing ...... instead of ....’

‘Yes it’s fine. It’s all going well.’

‘Yes we have heaps of evidence.’

Now ask:

’Well, because we’ve seen............. And they said................................. And we heard back that................. And.................................................’

Perhaps the evidence is a bit thin/threadbare/shaky. We need to:

(talk to more people) (talk to different people) (observe again, or for

longer, or of others) etc.

DIAGRAM 1 NEW EVALUATION OR WRITING UP OF OLD EVALUATION

Everyday Evaluation 3e.indd 64 19/1/11 9:16:26 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 6 5

Why is it so? or Why is it happening? or What is wrong with ......? or Why do they ......? or How can we ......?

‘There is this ..... and that ...... and ...... and ......and ...... (If they don’t believe this, they won’t believe anything. We’ve done the best we can)’

Pick the main question. That is your main research question, that you now need answers for (in order to know what to do next in your project, service, etc.).

O.K. Write it up

Say what was the nature of the original problem.

‘We set out to work out why x......... was happening instead of y............,

which we had expected or planned, etc.’

Say what your hunches were ‘We thought that if we ............

then ........... would happen, etc. ’

Describe in detail how your experiences confirmed and/or refuted your hunches.

Say who you had talked to, and how many, where and how they participated.

Say what questions were asked.

Describe any other information collected and circulated (and the sources).

Outline the various themes that emerged — showing why you came to your conclusions

Describe what you decided to do next (and progress regarding this, if it has already

been commenced)

Where does it all lead? What new actions should we take? What changes are

needed? What can we now recommend?

What can we conclude?

Now step back and ask ‘why?’ What else is going on (especially that people might not be aware of)? What pressures are being brought from outside?

What are the effects of the broader social contexts? What is the ‘bigger picture’? Check these ideas out

too. Talk some more. Collect some more information.

What is going on?

The key task here is to work out what you are hearing and seeing. What are people

saying? What kinds or answers are you getting?

Analysis, conclusions and planning new actions

Consider the whole range of other sources and techniques, (e.g. observation, things you can read, etc. — see list in Guide 4)

How else can you really get to know about the situation being studied?

‘We’ll need to ask ourselves ..... and .....’ (Try to keep to around 3–10 questions)

What questions will be useful?

How many? Where and how will you talk with them? Why these? Are there

others who would be relevant?

‘We’ll need to talk with ........ and ....... and ..........’

Identify the ‘researched’

O.K. To check out these hunches, what would you need to do? Now

you need some more fieldwork.

Designing a process of ‘finding out’

‘Well we reckon it’s because ............. and also ............. and ..............

Why do you think it is so? Why do you think it is happening?

Next, you need some ‘leads’ to know who to go out and ask what. Start with your hunches (remember they’ve come from previous ‘research’). In answer to your main research question ask:

Identify who is asking the question (you!). Is that different from who has the ‘problem’?

Everyday Evaluation 3e.indd 65 19/1/11 9:16:27 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

6 6 E V E R Y D AY E VA L U A T I O N O N T H E R U N

This is a typical set of principles reflecting over- arching valued ways of doing

things (which in turn are part of the rationale for the most over-

arching process of an organisation

or effort — see large-scale, long-

term purposes described in Diagram 5)

WHAT IS TO BE DONE

Goals

Objectives

To meet immediate crisis needs of individuals

To assist crisis resolution by individuals

To provide … (relevant crisis services, material assistance, etc.)

To provide … (related counselling referral,

placement, etc. services)

HOW IT IS TO BE DONE Principles of philosophy

Sensitive and receptive to needs

Responsive, flexible and adaptive

Integrated and comprehensive

Accessible

Address disadvantage and cultural difference

High quality

Participatory and empowering

Respect, workers’ rights

Preventative and developmental

Ethical

Some areas have received most attention to date — such as accessibility of individual services

(measured in No’s attending, No’s on waiting lists, No’s who have phoned, etc.).

More of a challenge to ask ‘How are individual services participatory and

empowering?’ How would we know? What are the signs?’

Just as each of the activity objectives can be specified in detail, so

also can each of the principles of philosophy, e.g. How can counselling

services be provided that respect dignity, integrity, chosen independence and

sovereignty, and are honest, just and fair? And so on.

DIAGRAM 2 FRAMEWORK OF PRINCIPLES AND ACTIVITY OUTCOME OBJECTIVES

Everyday Evaluation 3e.indd 66 19/1/11 9:16:27 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

To prevent recurrence or further development of problems

To strengthen conditions for well-being

These can be specified in detail (who will do what, when, where, how often, what outcomes do we expect or hope to see, etc.)

To provide … (information, advocacy, screening/

early detection, etc. services.)

To develop … (self-help, group work, community support,

community development, etc.)

To promote these conditions elsewhere

To initiate ... (community action, changes in structures, campaigns

of education, etc.)

For each ‘box’ we can ask ‘How do we do this in this way?’ e.g. How do we ensure group work is able to respond

to changing conditions? How would we know? What are the outcomes we

expect to see? Are we seeing them?

This is a typical set of activity goals and objectives which classifies the range of valued things to do (sometimes called ‘performance

areas’). The grid can be used to ‘cross’ principles of philosophy with objectives to produce operational aims, relevant indicators and

targets for outcomes to be achieved

This kind of matrix, and variations of it, show how questions can be generated about whether we are achieving our objectives and outcomes according to the kinds of principles of practice we value, i.e. whether we are doing what we want to, in the ways we want to do them. The material generated by matrices or ‘grids’ like this provide the stuff of:

prospectively as expectations or signs)

objectives and targets for agreed outcomes)

shoulds)

do x ...?’ questions)

‘How can we ...?’ or ‘Why haven’t we...?’ questions)

items to be identified and attended to, if all elements of a service are subjected

elements that fall into the realm of ‘achievable, but not yet taken for granted’.

Everyday Evaluation 3e.indd 67 19/1/11 9:16:28 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

6 8 E V E R Y D AY E VA L U A T I O N O N T H E R U N

Moving forward to new conclusions If instead we are asking about the value of current practice in terms of our observations and ‘descriptions of a world’, which we have not yet made very conscious but about which we may have strongly sensed or intuitive feelings, then evaluation begins to take us into an open search for new and possibly more valued ways of doing things. This calls for a process of thinking that is often only rational in hindsight, and that challenges us initially to describe the images that come to mind, name our feelings, articulate our hunches and explore ideas as we set off on a journey of discovery.

The aim of this kind of evaluation is to � nd out prospectively what is of value (merit, worth or signi� cance) and not to be constrained by existing conclusions and theories about what is already thought to be of value, or established goals, objectives and targets, but to ask unasked questions, observe the previously unnoticed, and consider the value and relevance of ideas and societal developments that may at � rst not appear to be of any direct relevance. In the course of exploring what people do value and think about things, we will work out what are our guiding images, intentions, purposes and vision. These in turn can then be made available for further discus sion, dialogue and checking against those of critical refer- ence groups (or checking back with them/us), and � nally constituted into more conscious frameworks for future planned and implemented actions.

I am calling these kinds of developmental or prospective evaluation ‘open inquiry’-type evaluation.

In Chapter 4, various evaluative efforts are identi� ed—ranging from the daily and small- scale, right through to the � ve-yearly or once-a-decade program ‘stocktake’—for which both the approaches to evaluation discussed in this chapter can be used, at whatever level of elaboration or scale and whether over the short term or long term.

The map and the compass To describe this metaphorically, if the � rst (‘audit review’) approach is to count the boats in the harbour to check they all followed the map to the already known, valued and desired destination, the second (‘open inquiry’) approach is for a boat to set sail for unknown and uncharted (but seeking more valued or desirable) waters with only feelings as a rudder, the critical reference group’s values for a compass, the urgency of its interests for an engine and its observations with which to keep adjusting direction!* When the boat arrives, there is a new map and we can count others when they follow suit—until someone else takes off to new, unmapped but hopefully better territory again!

Now, when we take off into uncharted waters, we may bring along for reference the map of the old harbour to see whether the new coastline resembles it—but if we only

* Interestingly the systems concept of cybernetics (or self-management)—which is about recur- sively aligning purposes to destination using feedback to chart one’s course—is derived from the Greek kubernetes, meaning ‘steersman’.

Everyday Evaluation 3e.indd 68 19/1/11 9:16:28 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 6 9

allow ourselves to stop in places that resemble precisely the old harbour, we may not be open to � nding something better.

As well, the map of the old harbour may be a quite useful and reassuring resource as long as nothing changes; however, if there is change in the situation or its context, then it will cease being the best guide for the future, and open inquiry evaluation will again become necessary to re-chart from what is more valued.

There is now much talk about ‘building in evalu ation’, but this is what it means: a perpetual process of learning

from our experience in the light of our guiding (and changing) values, purposes or philoso- phy (the ‘why’), in con junction with a check against a memory or reminder of what we thought we were doing, or had hoped or intended to do (the ‘what’).

Let’s compare the two approaches side by side. Diagram 3 on p. 71 sets out a picture of the characteristics of each of the two approaches.

Putting it together You may � nd you want to do an evaluation that combines both of these approaches. That’s ter ri� c! You stand a chance of both reassuring yourself that your service is doing what you thought it was doing (and � nding the gaps if it’s not) and also of breaking new ground that might suggest even better ways of doing it.

Some of us have been experimenting explicitly with how to do both kinds of evalua- tion within a single design process and have found that if we start with auditing against the ‘old-cycle conclusions’ (i.e. evaluation against objectives), it often has an effect of closing down our minds to thinking only about the value in terms of what we have set out to do. This is still critically important, but it is often very hard then to try to open our minds up again to thinking what else may be going on (but which isn’t covered by the program logic objectives). Nor do we get to discrepancy-identi� cation about unintended consequences (both good and bad) or creative thinking of ideas for improvement. That is, people often seem to slip into the well-ploughed furrows of looking at familiar objectives and intended actions, and forget to notice or remember the other little observations, insights or nagging uneasy feelings that offer to break out of these furrows and plough valued new ground.

Everyday Evaluation 3e.indd 69 19/1/11 9:16:28 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

7 0 E V E R Y D AY E VA L U A T I O N O N T H E R U N

Think about when your questionnaire only asks a lot of audit-style questions such as ‘Did you experience x’ or ‘Did you receive y’, and then asks at the end for ‘any other comments?’ and you can’t think of anything else.

To more effectively do both within the one evaluation exercise, it seems to work best to start with even just a few open questions like ‘How has it been going?’ or ‘What have you been doing/receiving and is it going well or not?’ or ‘What has been of most (and least) value?’, especially these questions are asked of the critical reference group, which may have a fresh ‘take’ compared with practitioners who are focusing on seeing particular things. This sets aside, for the time being, all the previous answers that might have been given to the question, including by carefully prepared service agreements that prescribe and proscribe service prac tice. Instead, it relies on people’s personal and local knowledge and what is at the forefront of their own inquiring minds, including the ideas already accrued through their various belief and value systems.

It turns out we do actually already know a great deal—particularly about what and why we are doing things and their value. It’s just that it is all stored on various ‘mental computer � les’ and the existing neural programs don’t always retrieve them!

Everyday Evaluation 3e.indd 70 19/1/11 9:16:28 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 7 1

DIAGRAM 3 TWO APPROACHES TO EVALUATION

OPEN INQUIRY EVALUATION • Inquiry—‘to seek’ • Starts with the questions: How are we

going? How is this service or activity, etc. going? What is happening? Is it working or not? In what ways? What do we think of this service? What is its value? What is of most value, merit, worth, or signifi cance about It?

• Asks the comparative questions: What are we doing? and Is that good or bad? What are the signs of this?

• Then asks problem-posing and problem-solving questions: How could we improve things? How could we do more of what we are doing right? How can we move from the things we don’t want to be doing to the things we appreciate more?

• Implies asking: What are the needs or deeper valued purposes?

• The questions are ‘opening up’ questions implying the need to build new evaluative theory inductively from diverse sources. There isn’t necessarily a concrete picture of what the answer would look like at the outset. The process will involve repeatedly asking the question why are we doing this?—to retrace the implicit program logic until the fullest and most satisfying set of explanations is achieved. And then asking ‘under the circumstances, what would be of more value (merit, worth or signifi cance) and why?

• Starts with immediate or obvious ‘pro- blematisation’ (either of good features or of bad features)—leaves the non- problematic as taken for granted.

• Examines practice in order to be able to ex tract assumptions and intentions.

Can then de velop new and improved evaluative criteria:

– Thus can ask about possible new goals, objectives, aims and activities that might differ from current ones.

– Is developmental, iterative or emergent (or feedback or ‘cyber netic’ systemic).

– Is improvement and change- oriented.

• Requires drawing on questioning, intuition, observant (interpretive), inquisitive, imaginative, spec ulative and creative aspects of mind.

– Use of logic of discovery. – Feels more like an art. – Aiming at excellence of

achievement. – Looks for ‘meaningfuls’. • Relies on who is/are the inquirer/s

(for example, inclusion of all relevant stakeowners and stakeholders).

AUDIT REVIEW EVALUATION • Audit—‘to check’ • Starts with the questions: Have we

done what we set out to do? Is this service, activity, etc. meeting its objectives according to the logic of the program theory and assumed outcomes?

• Asks the comparative questions: What did we set out to achieve? and What are the signs we have done this?

• Then asks the gap-fi lling and ‘irrelevance’-eliminating questions: What are we not doing (that we intended to do)? What are we doing that we shouldn’t be (that we didn’t intend to)?

• Implies already assuming what the needs are or deeper valued purposes.

Everyday Evaluation 3e.indd 71 19/1/11 9:16:30 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

7 2 E V E R Y D AY E VA L U A T I O N O N T H E R U N

Then the systematic ‘ticking off ’ exercise can be conducted—showing where intended things are or are not being achieved.

You may � nd that the open inquiry may have already covered some of this same territory, but now the most signi� cant matters will be to the forefront. In this way, you will have retained the ‘gold’ once the rest of the ‘gravel’ of current practice and objectives is sieved through. For example, a large residential agency for people with disabilities commenced the rewriting of its service agreement with a lengthy discussion of what single sentence it thought best represented what it was on about. When it was � nished and the agency compared it with its existing mission state ment, it found it had made an important conceptual leap from seeing itself as a provider of sheltered care to being a provider of home-like accom modation. The agency may not have got to this if it had instead begun by only checking that its existing mission statement was being addressed.

In another example, the volunteer staff of a community information service explored open ques tions about their work—and the issue of gender arose (almost all the volunteers were women, the coordi nators were male, the service-users were women, etc.), yet when they turned to see whether they were meeting their formal organisational objectives, there was no mention of the gender issue.

Diagram 4 on p. 74 shows a couple of very simple one-page evaluation sheets that have used the two approaches consecutively. Other examples in Chapter 4 show this too. A good

• The questions are ‘closed’ ques- tions; implying testing of logic based on theory from pre-existing inquiry cycle sources. (The correct answers are already known and are merely being checked for their existence and implemen tation). The process will involve repeatedly asking the questions, are we doing this? Did we achieve that?—until the full set of possible valued and aimed-for activities have been checked against outcomes.

• Sets out systematically to problematise all possible activities—leaves ‘nothing’ as taken for granted (except matters not covered by goals, objectives and aims and their logically derived indicators and outcomes).

• Examines practice in the light of objectives (applies known evaluative criteria):

– Thus starts and ends with existing goals, objectives and aims and their logically-derived indicators, activities and outcomes.

– Is a closed and causally linear system.

– Is status-quo (or frozen snapshot) reporting-oriented.

• Requires drawing on systematic, orderly, observant (monitoring), fastidious, highly organised, analytical aspects of mind.

– Use of logic of accounting. – Feels more like a craft. – Aiming at competence of

performance. – Looks for ‘measurables’. • Relies on the quality of previously

agreed-upon goals, objectives, aims, indicators and outcomes (and on the level of consensus previously reached).

Everyday Evaluation 3e.indd 72 19/1/11 9:16:30 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 7 3

example of integrating the two approaches, taken from the area of education, would be the difference between a teacher evaluating a student’s creative work of exploring a new idea (using an open-inquiry approach) by read ing an essay written by the student, and evaluating whether knowledge assumed to be of value has been acquired by marking a multiple-choice examination (audit review).

If you are conducting a largish effort, it can sometimes work if there is a division of labour around the two tasks. Those whose minds work more happily in an audit review mode could together assess a service against a set of objectives, extracted either from previ- ously written, formal statements of intention or from ‘reading off ’ the objectives from the kinds of services or activities components provided. Another group of more inquisitive souls with a nose for ‘problematisation’ (both negative and positive) could ferret around for the issues (the word ‘issues’ is itself almost always a euphemism for ‘problems’!) and start apply ing a ‘why, why, why’ inquiry mode of questioning to understand better the elements that need to be improved.

When the two groups eventually come back together, they might � nd overlap, but it will be the areas that don’t overlap that will point in the direction of positive change, and the areas that do overlap may con� rm the valid ity of the original evaluative research on which the service was based.

The two approaches in historical context ‘Audit review’-type evaluation after the event continues to be tremen dously popular, and is only just beginning to � nd a better balance with ‘front-end’ open inquiry approaches. In order for the latter to take their full place within a more comprehensive ‘full-cycle’ action evaluation framework, it may be helpful to examine the historical context of the rise of each.

The popularity of audit review-style evaluation may stem from the historical combina- tion of introducing new forms of human services in the 1970s, quickly followed by a desire to constrain their growth due to political economic pres sures on public spending.

The new forms of service emerged from the 1960s and 1970s when there was a particu- larly active phase of ‘open inquiry’, and consequent change in our views about how human services should be carried out. This was in many cases a change from a kind of industrial mode where modern profession als ‘knew best’, and intervened—often dramatically—in the lives of their patients, clients and other subject peoples with the authority of centralised, highly regimented and hierarchical institutions.

Ironically, these characteristics were associated with a way of providing human services based on the goals and objectives of an earlier era bent on creating a modern society in which ‘social problems’ were dealt with in a scienti� c, hygienic, ef� cient and utilitarian manner, in contrast to an even earlier nineteenth century industrial era when ‘problems’ were rife and people were left to fend for themselves with only the vestiges of feudal charity for comfort.

Everyday Evaluation 3e.indd 73 19/1/11 9:16:30 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

Everyday Evaluation 3e.indd 74 19/1/11 9:16:30 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 7 5

In the post-World War II period, ‘evaluation’ (rarely formalised or funded as such) by client populations and their sympathetic young profes sional advocates began to reject this well-meaning but often damaging approach. There was a shift towards understanding the social, political and economic conditions that created and shaped disadvantage (rather than seeing it purely as a matter of personal misfortune or individual wilfulness). Respon- sibility for people’s individual situations had become a social matter—starting with the introduction of government public health, education, welfare and social security schemes, and coming to rest in a set of understandings about how services should be provided to protect and promote people’s dignity, self-determination and empower ment.

In the 1980s and 1990s, these became the set of human services program principles, goals and objectives with which we have become so familiar (see Diagram 2 for an example).

Coinciding with this development came the end of the comfortable eco nomic condi- tions that had underpinned it. More money during the economic boom of the 1960s and early 1970s had seen the expansion of secondary and tertiary education, the growth of a well-paid, credentialled workforce, an intolerance of the ‘residue’ of poverty and disadvan- tage, and a determination to deinstitutionalise and resource a range of groups of people, who had literally been locked away from mainstream social and economic life, including people with physical, intellectual and psychiatric dis abilities.

However, the economic recession, and subsequent restructuring of industry and consequent demands for government resourcing of business recovery and debt, led to � nancial stringency, with the ‘luxury’ of social justice largely being postponed to await further presumed growth in a competitive economy. In 1980 in a Melbourne public lecture at Ormond College Martin Rein described the shift from ‘the vision of the social workers’ to ‘the vision of the economist’ at the height of the boom in the late 1960s and early 1970s, when a guaranteed minimum income seemed both desirable and possible, and � nally to ‘the vision of the auditors’. This latter ‘vision’ accompanied an era of ‘managerial ism’, ‘performativity’ and ‘leadershipism’, as the private sector moved ruth- lessly to maximise output and minimise production costs, and the public sector moved to copy this by controlling spending growth and containing the demands of a populace who had come to expect that their human needs might actually be met.

Thus the two groups—those who wished to ensure services were now targeting needs and abid ing by more humane principles, and those who wished to ensure that no new services were funded and indeed that some real decreases in funding levels could be attained (we should perhaps see the word ‘ration’ every time we are told an approach is ‘rational’!)— united to introduce tech niques such as Management by Objectives and Management for Results, program budgeting, performance indicators and targets. This approach has been in some ways cemented over the most recent decade by the move to tightly prescribing program logics, outputs, outcomes and evidence for these—leaving a silence about how the values underpinning the logics are created in the � rst place (when once this would have been achieved by needs studies).

Everyday Evaluation 3e.indd 75 19/1/11 9:16:32 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

7 6 E V E R Y D AY E VA L U A T I O N O N T H E R U N

This might explain some of the ambivalence with which audit review evaluation is still often greeted by local service providers or users. On the one hand, it seems terribly sensible and indeed helpful to check that one is doing the (good) things we set out to do, or getting the (good) services we were led to expect. But on the other hand, the ‘rationality’ of such a closed system approach may take critique and further change and development off the agenda. Yet not only growth, but also improvement and development that are within current budgetary restraints, may be impeded if our current logics, practice objectives and outcomes are insuf� cient or inadequate or inappropriate.

Interestingly, some of the most recent develop ments in managerialism (beginning with Management for Excellence and extending through quality improvement, to the contempo- rary strengths-based and appreciative approaches) have been away from reliance on tightly centrally controlled, technical rationalism (which had everyone constantly accounting in advance for everything they were going to do, and being held rigidly to doing every one of those promised things, and unable to do anything ‘outside the rules’ or unanticipated). The shift at least partially seems to be away from overwhelming reliance on ‘top-down’ carrot-and-stick motivation and command-and-control management with appearances of ‘performance’, towards including more ‘bottom-up’ autonomous and participatory self- management, inspirational lead ership and an emphasis on achievement within a broader organisational philosophy.

In important ways, this involves a shift from the excesses of an audit review approach towards integrating some of the char acteristics of open inquiry, such as qualitative observa- tion, critical re� ective insight and generation of new and better program theory, particularly through sensitively retouching base with the ‘customer’ or consumers.

Drawbacks of only evaluating against objectives, targets and outcomes While some of our services’ higher principles, deepest values and long-term purposes are the result of decades of painstaking research, theory, policy and practice, we only need to consider how it was that many objectives, indicators and targets were settled on to realise that some change might still be desirable!

Would you say that: • current practice is the result of planning which only came after in-depth, grounded

practice, � eld-based research in which all relevant parties participated? • this initial front-end developmental evaluation resulted in a high level of consensus and

commitment by consumers and practitioners? • everybody really understood what they were doing and why? • things proved to be easy to put into practice, as it was obvious what to do? • you could then tell whether they had or had not been met? • they were not overly ambitious or, alterna tively, too super� cial? • the needs they address haven’t changed since they were written?

Everyday Evaluation 3e.indd 76 19/1/11 9:16:32 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 7 7

Yes, Yes, Yes! Fantastic! No need to change yet. On the other hand, if you have been chuckling out loud, then you might be remembering how the new committee put together this year’s strategic plan based on the previous one; or a consultant came in to do the consultation that would give the rationale for what had already been decided; or your service manager met with the funding body, and after ‘constructive debate and discus- sion’, you ‘got up’ a few good objectives and you let ‘them’ get up some of theirs for the sake of peace and to get the funding application or service agree ment in before the closing date; or that the objectives were made so vague and general no one could disagree, and things wouldn’t be too unachievable; or the pressures of everyday work have meant there hasn’t been a chance to really stop and re� ect on deeper matters or see what the community thinks.

Yet, in terms of the ongoing evaluation research cycle (see the wall chart on the Allen & Unwin website), the program theory and resulting goals and objectives are theoretical predictions (if we do x, y, z, then needs will be met) until practice provides further evidence that we’ve actually got things right, or of how we need to adapt further. Until then, they are only as reliable and relevant as the research that led to them (and also depend on no change taking place to those original conditions).

As the saying goes, ‘The future is made, not predicted’—so we must retain a capacity to reform, change, develop and improve, or else risk the same ossi� cation that the commu- nity workers of the 1970s were suffering from with services designed for the 1940s and 1950s. This means treating ser vice policies, goals, objectives, targets and logical outcomes as valuable expressions of intention and purpose but as always provisional and not � xed for all time. For as long as this wisdom can be held, then services remain ‘alive’. In this way, they can move constantly between an old consensus and a new one—with evaluative research repeatedly bridging between the two.

So the drawback of audit review evaluation that can result in a preoccupation with measuring the discrepancy between formal objectives and practice (which can result in services adjusting their aims and activities to being more and more modest or achievable to seem to be ‘performing’) can be overcome by an open inquiry evaluation to continuously re-chart from deeper client-centred values, and free up re� ective thinking to develop better service responses.

While it is often an enjoyable, reassuring and valuable exercise (see Diagram 2), a drawback of audit review-type evaluation is that it can eventually therefore feel somewhat mechanistic and tedious about detail to the point of losing meaning. It is actually quite hard—even while worthwhile—to have to consider every little element of a service, especially all the bits that are going well enough and would otherwise be taken for granted. (It would be a bit like being stopped to evaluate your reading speed and comprehension levels every time you turned the page or asked yourself whether you were enjoying reading a novel.)

Sometimes this kind of evaluation—such as that implicit in a service agreement— also becomes repet itive or confusing, as each part of a service is considered under several

Everyday Evaluation 3e.indd 77 19/1/11 9:16:32 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

7 8 E V E R Y D AY E VA L U A T I O N O N T H E R U N

different headings in regard to its contributions to several different objectives that are connected. It is rare that one service or even a part of a service is only meeting one objec- tive, particularly if an inte grated approach is highly valued (possibly as an objective!). This might make audit review-type eval uations rather complex, but the complexity may actually be the price to pay in exchange for greater meaning. To be accomplished strictly within its own driving logic, it calls for a mind a little like a computer. Minds not quite so logically programmed can sometimes mix up objectives with princi ples of philosophy, or indicators with aims, and so on—but � nd it easier to tell a coherent story that narratively brings things together more meaningfully. This underpins the recent popularity of narra- tive evaluation which, while not a complete substitute for audit reviews, can help them make more sense.

So, despite all the detail and attention to systematically and logically evaluating ‘all’ ele ments of a service, the more glaring issue with evaluating a service against its objectives is that almost all the really interesting, colourful, ‘thick’, rich and juicy fabric of a service may actually lie outside the written-down objectives.

To give an example: if you were to evaluate this book, Everyday Evaluation on the Run, there are an enormous number of things that could come to mind: • how it makes you feel, what it touches off in you, where it connects, where it doesn’t,

what you thought of it, what you could learn from it • the appearance of the book, the colours, the type face • observations about the cartoons • the language, terminology or grammar • the style of writing, the examples used • whether you knew any of the people in the acknowledgements, or what you think of

the women to whom it is dedicated • comparisons you make between it and Do It Yourself Social Research • re� ections on other evaluation literature you might have read . . . and you would barely

have touched the surface of the colossal array of domains of potential ‘data’ and ‘evidence’ on which you might quite unconsciously be drawing to evaluate it. If you took longer (or I interviewed you!), you might expand into considering:

• its use in your own workplace or when evalu ating services you have used or situations in which you have found yourself

• some other people who could use it, or who might � nd it dif� cult • If you had already talked to others who had read it you might recall their comments and

compare them with your own thoughts. They might have mentioned its size and thick- ness, or cost, or value to students, or things it hadn’t even occurred to you to think of

and so on and on. From this wide range of potential aspects, you might volunteer some you had particu-

larly noticed that relate to your needs and interests. This would give strong clues about what is of most value or not of value. This approach, by inquiring about only those elements of interest, could then focus the evaluation on the elements needing change

Everyday Evaluation 3e.indd 78 19/1/11 9:16:33 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 7 9

or replication. On the other hand, if you wanted to do a really thorough open inquiry evaluation, you could ask, ‘What do you think’ of all or many of the widest possible range of aspects.

Both these exercises (‘What did you think of the book?’ and ‘What did you think of this aspect of the book, and that aspect of the book, and this, and this . . .’) would give you a more grounded evaluation from the reader’s own perspective.

If, however, you started with your purposes and intentions and only checked for these (‘We set out to do x and y and z: did you experience x, did you achieve y, and was z under- stood?’) you may well have missed out on valuable insights about values, needs and interests you had not realised were there but that were (or were not) catered for.

Sometimes our objectives are relatively cryptic—particularly those for more distant audiences such as funding bodies. For this book, there were four simple operational objec- tives (or targets): 1 to write a book with the title Everyday Evaluation on the Run 2 to have it in camera-ready form for the printer by a speci� ed date 3 to print 5000 copies, and 4 to have it purchased (including by critical reference groups).

The answers to a basic evaluation of the success of these objectives are, more or less, ‘Yes’, ‘Three months late’, ‘Yes, plus a further 5000 copies in the second reprint’ and ‘Yes’. Evaluation of this kind strikes many as not quite getting at the full meaning of their work! It is perhaps a little like describing Beethoven’s Fifth Symphony as 22 minutes and nineteen seconds long! Now if you’re a central program man ager who has to ensure that a perfor- mance of � ve pieces of music will � t into the program time, this may be the only evaluation that is required. But if you’re asking about the general value of the piece to an audience, then something of the ‘colour’ and meaning may have been lost! Indeed we don’t actually know from this exercise what the value of these things is—except by reference to the framework of principles and goals based on the existing (previously generated) theory (If we do this . . ., then that . . . will follow).

The � aw in assuming that one’s operationalisa tion of objectives will necessarily bring about one’s higher purposes or intentions might also be illustrated by the example of this book. It turned out that writing a popular and accessible book on evaluation was more dif� cult and time-consuming than originally thought. Ideally, it should have been left to develop to a further stage; however, the funding service agreement set tight limits around it and constrained its capacity to meet its overall purposes and guiding goal.

Alternatively, you might have proceeded to set objectives (indicators, and measurable targets to result in speci� c outcomes) for every one of the large range of possible aspects (listed in point form above). Yet would one neces sarily want to record formally all this mass of details as part of a rational and extensive standards or objectives-based evaluation framework at all, much less in advance? Or would this be a bit like counting all the leaves to know whether it is autumn?! Perhaps too much rationality of this kind locks us too tightly into a new iron cage.

Everyday Evaluation 3e.indd 79 19/1/11 9:16:33 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

8 0 E V E R Y D AY E VA L U A T I O N O N T H E R U N

Often a service will generate an activity that will suddenly take off and head in a really brilliant direction. When evaluated against its formal objec tives, such a development may be merely recorded as relevant to objective ‘A’, or it may even go unrecorded for want of a relevant category. But if the tight restriction of pre-existing audit review eval uation could be released, and a lot of attention devoted to that single successful instance, enormous ground might be covered that might have relevant repercussions for other areas of the service and may even lead to new or reframed objectives.

For example, a service may contribute a stall to a local community festival. If evaluated against its formal intentions, it may appear to have a rather weak ‘community education’ outcome. On the other hand, if it is examined for what effects it might have had per se, those who sat on the stall all day might have returned with insightful stories drawn from the half- dozen remarkable encounters with several members of the public who might never other- wise have approached the service with their stories. These stories may have yielded tip-offs regarding the parlous state of a local private nursing home for the aged, or the germ of an idea about addressing the possible volume of hoarded, unused and dangerous prescribed drugs in people’s medicine cabinets at home, or an exciting new idea about how to connect with young people who are carers using the internet.

Rather than the stall educating the community, the community had used the stall to educate the service! This insight might lead to the next stall being explic itly run as a ‘person in the street’ study, or an even more effective way of tapping community views. As well, another consequence may have been to have the stall appear in the background of a photo in the local paper the following week—reminding a single member of the public to want to join up. Again, when measured against a numerical ‘performance indicator’, it may not look very valu able. A manager may even move to abandon it in future. But say this single new recruit goes on to lead a successful campaign against the dumping of toxic chemicals in the local creek. And what if, with that new-found con� dence, she stands for local gov ernment? It might then be interesting to explore what it was about the stall that encouraged residents to talk; what characteristics of the workers allowed them to listen and hear, and so on.

It is this kind of locally contextualised, meaning ful and more complex information that may allow a more valid evaluation to be done of that service’s activities, yet one can see why such grounded material is both so important and also so elusive: • Effects may be long term. • Effects may only be known about by ‘chance’ or by having excellent local contacts to

hear ‘up close and personal’ or be intuited or deduced. • Effects may only be connected to the original activity by a � ne thread or a complex

network of interdependent effects rather than follow a straight line of simple cause and effect.

• Effects may be too costly to trace, but only anticipated ‘in theory’ (relying on having already been established in painstaking observed developmental practice). That is, it is too time-consuming and expensive to follow up, or only possible to do this very intermittently.

Everyday Evaluation 3e.indd 80 19/1/11 9:16:33 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 8 1

There is the now-famous example of the American War on Poverty’s pre-school chil- dren’s Operation Headstart (followed by Operation Homestart)—which was evaluated as ineffective and subsequently abandoned using an experimental design within only a few years of implementation. Twenty to 30 years later, the evi dence came in that children who went through those programs had in fact done signi� cantly better in terms of education and job success. We don’t require every teacher to check on their students twenty years down the track (interesting though it may be to be invited to their pupils’ school reunions!), yet we can be more or less con� dent of the value of teaching in the here and now because we have already established strong theory from previous cycles of evaluative feedback research pitched at the appropri ate level and the appropriate relationships between practices and later community and workplace settings.

The Australian example of community health centres being evaluated several years after inception and shown not to have made an impact also under lines this point. There is an addi- tional point that can be made here, which is not so much that such services are unevaluable (until they have operated for twenty years)—although this may also be true—but that there are varying indicators of success, and that these indicators match differing levels of inten- tion and action. As a book author, I check today’s output against an intention to produce so many pages about such and such a topic—not against whether I have managed to demys tify modern Western science to a statistically signi� cant extent! If a community group is funded to produce a newsletter for Heart Week, we don’t usefully eval uate it by checking the blood pressure of the population before and after its production. We may more usefully chart a sequence or logical pathway of effects and outcomes, and trace the more proximal of these connections in the � rst instance.

These kinds of observations underlie the current popularity of ‘logic models’ that spell out a hierarchy of purposes and practices to achieve them.

Drawbacks of open inquiry evaluation The fundamental drawback of open inquiry evalua tion lies in its capacity to overlook matters that are not obvious to the inquirer. Audit review evaluation will ask about ‘all’ elements (at least, those that are covered by existing intentions), and if those existing objec- tives are comprehensive, then the evaluation will also be comprehensive. However, open inquiry evaluation is generally ‘only’ able to be discrepancy based (whether appreciative or critical) or problem or improvement-focused. This makes it terribly important to ask, ‘Whose discrepancies or problems?’ and ‘Are the problems important?’ or ‘Signi� cant to whom?’, and ‘Improvement in what terms, and who will bene� t?’ If the problems are those of staff, then do they re� ect what is problematic to critical reference groups? Or if the ‘signi� cant changes’ are less important compared with the ‘white horse on the table’ that everyone is too polite or too afraid to mention, then the evaluation may be misdirected. Or if the voice of the critical reference group is not being heard at all, then the evaluation might well miss its mark.

Everyday Evaluation 3e.indd 81 19/1/11 9:16:33 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

8 2 E V E R Y D AY E VA L U A T I O N O N T H E R U N

This again emphasises the importance of genu inely involving critical reference groups in actively guiding the process and direction of inquiry in collaboration with other relevant perspective-holders. Users can introduce percep tions of situations that might never other- wise occur to staff, and can alert to discrepancies in priorities or presumptions that also might not have been obvious to staff.

Open inquiry approaches to evaluation require tolerance of considerable uncertainty and a realisa tion that evaluation essentially remains a matter of judgement. We can perhaps see why quanti� cation is so strongly associated with audit review styles of evaluation. When one has � rmly decided on clear objectives, and may even have converted the signs of achievement into clear performance indicators and worked out precise standards to be achieved, then one can try to detail with considerable precision the extent of compliance (or discrepancy). Sometimes this precision can be expressed without scaling or ranking, such as where questions call for Yes/No answers. For example, if the standard for a ‘partici pation’ objective is the presence of a community committee of management—it either exists or it doesn’t. Or it can be expressed in terms of how often or how many (people through the door, projects accomplished, etc.); or performances and actions can be ranked, graded, scaled, or scored.

However, for much everyday evaluation, a simple sense or feeling of discrepancy may be adequate (like the shepherd who knows there’s a sheep missing without counting them all, or the service-user who knows she is in pain or is being treated disrespectfully without having a device to measure it). However, whenever the extent, degree or amount of discrep- ancy between a goal or standard and an actual performance becomes crucial, then human beings prefer the reassurance of apparent certainty. Even when such matters remain matters of dubious contextual judgement, we seem to like to imagine that they aren’t! For example, when there’s a lot at stake, we like to ‘know’ that the swimmer really ‘won’ the electronically measured race at 24.003621 seconds and the swimmer who ‘only’ got there at 24.003622 seconds was de� nitely second-rate.

When there are scarce budget funds to be dis tributed among a lot of starving services, the apportioning-type evaluation that says that this ser vice is a ‘6.3’ and the one that loses half its budget was only a ‘5.8’ gives us a sense of ‘the right thing’ having been done, even if it hasn’t. Usually, but not always, there seems to be safety in numbers (especially if you’re a 6.3).* We can see again the sense in Albert Einstein’s saying ‘Where there is certainty there is no meaning. Where there is meaning, there is no certainty.’

If the managers having to make these budgetary decisions cannot get out and about to observe ser vices at � rst hand (and acquire direct naturalistic data), or have in place a local

* Sometimes the ambivalent term ‘measures’ is used. But this has several possible meanings including: • size or quantity • degree, extent, amount • unit of capacity • that by which a thing is computed • suitable action. Unless quanti� cation is intended, less ambivalent terms are ‘identi� ers’, ‘indicators’ or ‘signs’.

Everyday Evaluation 3e.indd 82 19/1/11 9:16:33 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 8 3

and regional network of people who can reliably do this and compare their naturalistic data, then they may rely more and more on abstracted numerical data. Objectifying in this way may make it easier for managers to avoid being confronted by the moral and ethically contextualised nature of their judgements. However, the further consequences of this may be twofold: 1 Those services that are longer established and have re� ned their practices so tightly as

to be able to time tasks and cost them to the smallest unit and attach a numerical value related to an assessment of needs may do well—even if the practices are no longer worthwhile or meritorious in relation to new more relevant values relating to current community needs.

2 Those services that are newest and most innova tive (and may be addressing vital and creative ways of doing things), but are still charting their paths and working out what counts as success and what doesn’t, may be most vulnerable to not being able to evidence why they should be allowed to continue. In Canada, the � rst services to be cut in a tight economic climate just happened to be

the most recently established: services such as those experi menting with self-determination, empowerment and participation of native Indian and minority groups; and low-income groups. There are similar experi ences in Australia, the United Kingdom and the United States. Slowly, it is being realised that today’s ‘tried and true’ only got there by being yester- day’s ‘innovative and creative’.

Yet even the newest and most creative projects can, with some modest assistance, self- evaluate around the particular signs of success that are meaningful within their own local contexts. Over time, and with dialogue, these may become more reliable and comparable signs.

However, not even the most useful evaluative statistics can necessarily be guaranteed a place in the sun. Even when abstracted numbers do bear some meaning, they may be rejected—or even abolished! For some years, the Australian federal government collected national emergency relief statistics. Every agency in the country painstakingly kept records of the age, sex, family status and income source of every applicant. Such data enabled the often-quoted statistic that nine out of every ten applicants had been or were awaiting a Department of Social Security pen sion or bene� t. The collection of these statistics was subsequently defunded and discontinued. No statistics—no problem! :-(

Guides 5 and 6 summarise the previous discussion of the strengths and weaknesses of the two phases of the evaluative research cycle.

Everyday Evaluation 3e.indd 83 19/1/11 9:16:33 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

8 4 E V E R Y D AY E VA L U A T I O N O N T H E R U N

GUIDE 5 STRENGTHS AND WEAKNESSES OF THE TWO APPROACHES

Audit review Strengths Weaknesses • Will call attention to ‘all’ matters

previously thought to be valued/ required—and submit them for attention and check they have been implemented. Within this framework, it can give a sense of illumination to areas that may have been overlooked or forgotten.

• Validity will be high if the prior research base was strong.

• Can be comprehensive and reassure that much is being done.

• Will identify matters (gaps) not yet attended to within the existing framework

• Can affi rm previously reached agreements and consensus; can strengthen collective sense of direction.

• Can feel very comfortable and comforting.

Open inquiry Strengths Weaknesses • Will call attention to the more

pressing or priority issues at the forefront of consciousness

• Problem-identifying focus increases the chances of problem-solving and better outcomes; change for improvement and in areas identifi ed as important.

• Doesn’t discriminate between the still- valuable and appropriate matters and the now not-so-valuable and even irrelevant or outdated matters (doesn’t identify how to change or improve; may lead to preoccupation with mere appearances of performance).

• Validity will be low if prior research establishing value was ungrounded, thin on evidence, etc.

• Can be tedious and feel wastefully time- consuming.

• Will not identify (without resort to inquiry mode) why these matters have not been implemented, or whether they should have been, or still should be; nor identify any gaps outside the framework (e.g. unmet needs).

• Can’t assist much if the objectives don’t represent consensus or generative agreement, or if they are only superfi cial, or ‘paper over’ confl ict, or if the certainty is trivial, irrelevant, meaningless or leave out the real issues of critical reference groups.

• Can give a feeling of complacency. May encourage inappropriate conserving of the status quo.

• There may be pressing issues that have been repressed into non-consciousness, that need time to surface, or be ‘felt through’ and refl ected on.

• Problem-focus may overlook important matters that no one has yet identifi ed as a problem or alternatively overshadow the strengths, assets and good things happening.

Everyday Evaluation 3e.indd 84 19/1/11 9:16:33 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 8 5

• Assists innovation and creativity and dynamism in special areas (depth).

• Sense of exhilaration at a journey into the unknown.

• Meaningfully grounded rather than abstract, involving rather than excluding.

• Identifi es what is of value, merit, worth or signifi cance.

• May not be systematic and comprehensive for all areas (scope).

• Involves uncertainty, suspension of judgement, lack of clarity, and even a sense of chaos, and possibly seeming ‘irrationality’, disagreement and confl ict.

GUIDE 6 WHAT TO DO TO IMPROVE THE TWO APPROACHES

Audit review • When auditing against objectives,

always ensure there is a method of also collecting records about matters that will either require further inquiry or may be signifi cant on refl ection—for example, where there is feedback about faulty logic or outdated objectives, or problematic implementation that might imply diffi culties with current objectives and intended outcomes, or matters that are otherwise silent or have no obvious place in the current framework.

• To strengthen chances of validity, put in place a parallel or sequential process for open inquiry.

• To mitigate false certainty, recognise the need to move to some kind of change, and publicly identify the existing framework as provisional and contributory to the next phase of evaluation.

• To overcome tedium and wasteful time con sumption, keep a full audit review for only occasional use (for example, annually or two- or even three-yearly), and supplement it with more frequent, smaller scale, ongoing, built-in problem-solving or appreciative inquiry evaluation. Focus annual reviews on the areas for change and improvement that have come out of open inquiry.

Open inquiry • To identify important matters that no one has yet seen as discrepant, touch

base with the critical reference group: work out ways for them to speak in their own undis torted voice and be heard without too many intervening ‘sound barriers’; encourage and respect even unconventional and (at fi rst) unsystematic forms of input; don’t be too quick to judge what is said as being irrelevant or unrepresentative—even the most solitary consumer voice may turn out to be the canary in the mine; later you can check the extent of the view (‘Somebody has suggested that . . . What do you think?) if that is necessary. Seek out comparable

Everyday Evaluation 3e.indd 85 19/1/11 9:16:33 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

8 6 E V E R Y D AY E VA L U A T I O N O N T H E R U N

Questions that address each of the two approaches in sequence To return to the point about the need for both approaches, a comprehensive evaluation framework would include two sets of questions with both service-users and service provid ers/other stakeholders. This section sets out to convey the logic of the two different kinds of questions that � ow from the two approaches discussed. Note that these may not be the precise questions you or your group might ask (or might ask of yourselves). These would have to be tailored to your own situation (and they can all be expanded or contracted in detail, depending on your time and purposes).

1. Starting with questions to users or critical reference groups Open inquiry questions for end-users These are the kinds of questions that ask, ‘What is the value of this service or activity to you?’, ‘What is not of value to you?’ or ‘Is the service or activity working for you?’ (‘What works?’ or ‘What doesn’t work?’).

If these kinds of questions are too broad, you could break down the service or activity into aspects or elements—‘What is the value of x to you?’ or ‘What is not of value to

situations or groups: have they identifi ed problems you haven’t? (Again, check out the value of these to your critical reference groups.) Draw on as many sources of different perceptions as you can: read the literature; think up analogies or metaphors that might be useful to reframe old ways of thinking that aren’t working.

• Evaluate against higher-order intentions—for example, you may be able to tick off suc cessfully that you have put an equal opportunity offi cer in every technical college—but has it really decreased women’s inequality?

• To counter lack of thoroughness, make sure that, periodically, a systematic review is done—but possibly less often rather than more. Constant, clarifying, problem- solving dialogue and critical and appreciative evaluation shape an organisation in the desired direction and decrease the chances of it ending up in the wrong place.

• To tolerate uncertainty, recognise the desire to improve, and accept moving to some level of agreement through dialogue, then treat what is possible as provisional but give it a chance to survive further trial and error. For example, a group may be reluctant to say precisely when a project will be completed or what its effects will be, but it can be known that those doing it have a track record for completing projects or good work, or it is known that the activity is addressing a notoriously diffi cult but vital topic. Remember that the more deeply innovative the effort, the longer it will need to prove itself—possibly even years or decades. Measuring Aboriginal blood pressure before and after a twelve-month project where a community worker is attempting to assist Aboriginal communities identify the condi tions for improving health may be not just unhelpful but actively re- damaging. If it took 200 years of dis empowering by white societies, it may take 200 years of resourcing to show identifi able outcomes. Generate fuller descriptive narratives, especially where abstracted snapshot statistics do not make enough sense on their own.

Everyday Evaluation 3e.indd 86 19/1/11 9:16:34 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 8 7

you about x (or y or z)?’ or ‘Is element x working?’, ‘Is element y working?’, ‘Is element z working?’ and so on.

Then you would pursue context by asking, ‘Why is that?’, ‘Can you talk more about that?’, etc.

You should get a picture of what is being done (description of activity or service), how this is known to be of value (description of the signs) and why it is working or not working (explanation for it). You may need to ask (or ask yourselves) ‘Why?’ several times to get a full explanation.

An example of an exchange that captures all three of these elements would be:

Q. ‘Can you say what the value of the self-help group has been for you (or us)?’ A. ‘Well, the meetings of the self-help group I attend are good.’ Q. ‘Can you say why?’ A. ‘Oh well, I suppose it’s helped me recover from the grief quite a lot.’ Q. ‘What are the signs of that for you?’ A. ‘Um well, I feel much brighter the day after the meetings! And it just feels less

alone when your thoughts drift back—you just know that others have the same thoughts, and that you can move on from them. Some like me take longer, but there’s a woman at the group—she only came once—but she’s sort of my personal inspiration. I just remember how she’s handled it. It gives me a picture of where I want to get to.’

Q. ‘What about the different aspects of the groups—what about the group leader? What about the time, the venue, the topics, etc.?’

There may be contextual matters that need to be clari� ed (‘Which self-help group?’; ‘How did you come to join it?’; ‘What were the signs or effects of the grief ?’ etc.). Or you might want to get more detail on how often the person comes to the group and so on, but the focus of all contextualising questions is to further illuminate the value of the group to them. The answers imply their own goals and objectives as well as deeper purposes, intentions and background needs and desires.

Then draw out what was or would be better and why: ‘What would you (or we) have preferred?’, or ‘What would be preferable in future?’, or ‘What would you (or we) like to see change?’, or ‘How could it be improved?’ Again, ask about ‘Why?’—that is, what effects might be expected, and why these changes might be improvements. These questions further clarify the implicit goals and objectives as well as deeper purposes, intentions and back- ground needs.

If service-focused questions prove too narrow, you may need to move more quickly to ‘bigger picture’ type questions, either about people’s own life situations or about the implications of these for better visions for future action.

Everyday Evaluation 3e.indd 87 19/1/11 9:16:34 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

8 8 E V E R Y D AY E VA L U A T I O N O N T H E R U N

Audit review questions for end-users Remember that these kinds of questions test whether the objec tives of those who planned or carried out the service or activity were realised. You might say: ‘The service or activity was trying to do x, y, z because … (logic based in context of what is valued). Can you (or we) say whether you/we think these things are happening or have happened?’ Don’t miss the opportunity at the same time, of asking open inquiry questions about whether these objec tives are still of relevance or value to the person: ‘Can you say whether that was still a worthwhile thing to try to achieve?’, ‘What else would be of importance or more important to you?’

For example:

Q. The self-help group’s organiser is trying to provide a group that achieves a number of purposes. Please comment on the following:

(a) Have you found the group friendly and approachable? (How much, in what ways? etc.) Is this important to you?

(b) Have you found you have learned more about the condition/situation, etc. (that is common to the group)? Is learning more about it of interest to you?

(c) Do you have contact with any of the group members outside meetings? Would contact outside the group be of value to you?

(d) Do you feel more in control of your situation as a result of being in the group? Is that important to you?

Finally, examine any similarities and/or differences between the two kinds of material resulting from audit review and open inquiry questions.

2. Moving to questions for providers and other parties The format for questions for providers and other parties is the same as for users or critical reference groups above—with the exception that they need to clarify whose valuing is sought.

Open inquiry questions for other stakeholders As soon as you move to questioning providers or other parties, you are asking about their perceptions of value which may be judged by criteria that relate either to themselves or to users or critical reference groups—or to co-workers, funders, managers or administrators. There are several ways of approaching this, each re� ecting different purposes and contexts for asking:

Everyday Evaluation 3e.indd 88 19/1/11 9:16:34 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 8 9

• You can frame all questions in terms of the critical reference groups on the assumption that this is the fundamental purpose of their activity (for example, ‘Do you think this service is working well for the critical reference group?’ ‘How do you know?’, ‘What have they said?’).

• You can ask for people’s own evaluations (‘How do you think it’s going?’) and then ask questions to identify who are the ref erence groups implied in the answers (‘Who are the end-users meant to be bene� ting?’) You may then need to ask explicitly about the value to critical reference groups (‘How would you know what end-users thought of it?’).

• You can ask what value they think it has for the critical reference group (‘How do you think it’s going for the end-user group?’), then ask what value they think the critical reference group would place on it (‘How do you think the end-user group thinks it’s going?) then ask what they think any other parties would think of it. You would have to think carefully about whether your choice of questions strength-

ens the assumption that the effort or work is for critical reference groups, or whether it might strengthen the assumption that other parties can judge what’s best for critical reference groups—pos sibly without reference to or respect for critical reference groups’ views.

Audit review/questions for other stakeholders These questions are the same as for users or critical reference groups (as above), but in the context of identifying: • what are the purposes and intentions (‘What are you or were you trying to do for

end-user groups?’) or what are the goals, objectives, indicators and targets set by you or your service that you are using or have been using? (‘What did you set out to do?/What were you meant to be doing?’ ‘Why were you doing it in this way, in terms of end-users’ expressions of needs or interests?’)

• were those intentions put into practice (in whose eyes—yours? The critical refer- ence groups’? Others’?)—and was this valuable? (This now reverts to an open inquiry question.) Break it down into separate components and ask about each one—see Diagram 2 (‘Did you do x?; Did you do y?’; etc.—and ‘Was this valuable, impor tant, worthwhile?’ etc.).

Tables 1 and 2 are two matrixes that show the two different kinds of questions with two worked examples. Note the value of asking both sets of questions as a sequence.

Everyday Evaluation 3e.indd 89 19/1/11 9:16:34 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

9 0 E V E R Y D AY E VA L U A T I O N O N T H E R U N

Table 1: Open inquiry-style evaluation framework

Describe the area, activity or program

How is it going?

Why? How do you know?

What do you think is going on?

What are the implications for the future?

ACT/OBSERVE

(What was done? or What are we doing?)

EVALUATE

(Is it working or not working?)

OBSERVE-

ANALYSE

(Why do you say that/What have you noticed? What signs, indicators, factors or evidence?)

REFLECT-

SYNTHESISE

(What else is going on that is creating this situation? What is really going on?)

BUILD NEW

THEORY/PLAN

(What changes would you make? What would you do differently?)

I had this new idea for a lesson plan in the classroom …

Some things about it seemed to go well —but perhaps more did not

The children appeared inter- ested, and two said it was great —but I’m not sure they really ‘got’ it, I had to talk a lot, they didn’t realise what was in the pictures

I think the topic was of interest to them, but maybe we explored aspect x when they might have preferred to explore aspect y

Ask them at the beginning of next class to say what they thought of it and why; have small discussion groups and then show the pictures later.

Everyday Evaluation 3e.indd 90 19/1/11 9:16:34 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763

O P E N I N Q U I R Y A N D A U D I T R E V I E W 9 1

Table 2: Audit review-style evaluation framework

Describe the overall vision or goal/s of the activity or program

Describe the detailed objectives so they show the program logic

What were your indicators and targets for each of these?

Did you achieve these? Show comparative data/measure- ments Then recom- mence open inquiry… by asking why (or why not)?

Continue with open inquiry… By asking What could you try instead?

PLAN PLAN/ACT PLAN/

OBSERVE

OBSERVE/

REFLECT

REFLECT/

PLAN

We set out

to …

reduce home- lessness

By doing …

a) reconnecting young people with their families

b) by building their self-regard and con� - dence

c) by expanding housing options

d) by restoring Under-18 living allowance

We expect to

see …

a) at least one family reunion for 50 young people

b) improvement as per St Luke’s self- assessment scales

c) twelve new community units built within two years

d) an approach made to the local member to introduce a Bill

What happened

…

a) only eight—the con� icts were too great in the other families

b) 72% higher— main factor was we’d ensured they had the power to choose

c) only three built

d) meeting held; approach rejected

Try next time

a) research building resilient peer groups

b) repeat same program

c) bring stake- holders together to sort out issues

d) write policy and run a public-awareness campaign

Everyday Evaluation 3e.indd 91 19/1/11 9:16:35 AM

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 11/23/2018 9:32 AM via FORDHAM UNIV AN: 398756 ; Wadsworth, Yoland.; Everyday Evaluation on the Run Account: s8944763