Archive for May, 2010

Building a culture of evaluation

Wednesday, May 5th, 2010

SHOW  ME  THE  CHANGE

Complexity and the Art of Evaluation – Reporting Sheet

Leader: Steton Kantmon

Participants: Amy O’Brien, Pam Beattie, Nicole, Kate Patken, Keren Winterford, Catherine Doran, Natalie Moxham, Jen Orange, Narelle Chambers, Liz Franzmaan, Rob Catchlove, Julie Richmond.

Key Points:

Drivers

Barriers

  • Being able to learn from failure safely

  • Joint leadership management

  • Knowing why you’re doing it

  • Learning, reporting

  • System/roles in place that encourage doing, own acting on results

  • Requiring recommendations, input data, action
  • Success driven culture, not acknowledging failures

  • Lack of resources to act on evaluation

  • Pressure to report results before expected change can happen

  • Tenure of champions to start

  • Failing to integrate accountability reporting goal with learning/improvement goal

  • Reactiveness to short term/lack of strategy

  • Tangle up of identity/investment in approach

Learning

  • Have dedicated roles/ongoing

  • Recognise need for results of different scales

  • Ownership of evaluation by users; focus on doing it for us – primary audience is the program delivery

  • Don’t confuse evaluation for management, define framework at beginning

  • Keeping progress log as living document

    • win it

    • own it – bring governance/funder along

  • Distinguish monitoring from evaluation, but integration

  • Whole of organisation provision, accountability and affectiveness, local tailoring and modification, to help solve local problems and make decisions

  • Links to planning and goal setting – integrate with business planning

    • program log = budget

    • evaluation frame = actions

  • Rather than showing individual contribution, show collective contribution of partners/stakeholders

  • Have agreed/scaled indicators for different levels of intervention, different positives

  • Multi-layered collaboration for intervention avoids the need for competing claims to have solved problems

Evaluation toolbox: let's talk about it

Wednesday, May 5th, 2010

SHOW  ME  THE  CHANGE

Complexity and the Art of Evaluation – Reporting Sheet

Leader: Damien Sweeney

Background to Toolbox:

  • Four councils involved with their project and Swinburne University is the academic partner

  • Current stage – putting tools together, have piloted some already with participating councils

  • Phone contact for case studies

  • Online forum would be useful

  • Have limitations of evaluation technique is useful

  • Qualitative is common – interview and qualitative

  • Cost benefit analysis is useful for CSIRO examples

  • Attribution – were you influenced by something else or this project? (Categorises the other influences)

  • Links to online surveys and tips to use that would be useful

  • Need to say what evaluation to have when ie. Evaluation plan for duration of project

  • City of Ballarat – project on planning and how engaged people are in programs – would be willing to pilot toolbox

  • Behaviour Change blog – through conference, could be useful to have this on site for blog

  • Survey – difficult to follow up on outcomes. Survey monkey is useful

  • Needs to be able to change the toolbox – not have it static, eg, add case studies, reviews of tools etc – make it interactive

  • Have volunteer from City of Ballarat willing to pilot the toolbox

  • Wicki page – people can add their own information and use as a blog

  • The National Social Marketing Centre in the UK is a good example of an online toolbox

  • Clear Water – storm water project

Tried and true facilitation methods

Wednesday, May 5th, 2010

SHOW  ME  THE  CHANGE

Complexity and the Art of Evaluation – Reporting Sheet

Leader: Candice Bruce

Key Points:
•    Ask for input – pose a question = ownership – outcome
•    Run perpetual initiates
– normalise behaviour
•    Develop trust
–    Adaptation
–    Solution focus
–    Visuals – “in a perfect world?” Grading from 1 to 10
–    where are you now? What are the good things and the learnings? What is one thing that you can do to move forward?
•    Collective Social Learning Process – Professor Valerie Brown – “Leonardo’s Vision”
•    Spacial marking (buy special chalk to mark on carpet) – participants step in and out
•    Structured workshops
–    home work before
–    brain storming
–    objectives –  business plan
–    options
–    time frame
–    post-it notes
•    Action learning (W.I.L)
•    Getting action at the end – pair up – what I’m going to do
– follow up in 2-3 weeks – accountable
•    Never make a statement when you can ask a question
•    Two ears, one mouth – listening – hear how you are being heard.
•    Telling stories about our own journey
–    photos – pick a picture you identify with and say why
–    share with 1 other person
–    then share with the table
•    Disposable camera – photos of what are their favourite areas or their home
•    Shared Resources:
– “Estelo” specialist training supplier Sydney
– Book – Climate Action – Diesendorf
– Edward de Bono – 6 Thinking Hats
– Storms of my grandchildren – Hanson
– The Rational Manager – Kepner and Tregoe
– “Kids Teaching Kids” Aaron and Richard Wood
– MS Project – Microsoft Excel
•    Make time for individual reflection and to do visual stuff
–    Role model deliberate actions
•    Transition handbook – share slides – use art and participant centred
•    Use of PCs – SKYPE – Web Cam – interactive learning solutions
•    Conversation Tech World Café – small group, slightly different question
•    Avoid Death by PowerPoint – ½ page is read by each – feedback to group
•    Lend out lecture notes eg. Harvard – ask for question out of the lecture (day before) – get student to answer questions – 5 minute lecture
•    Participants to present chapters in book
•    Question – what is in it for them? List main outcomes – collect at beginning, everyone walks out satisfied
•    Appreciative enquiry process
•    Look for criteria – start where they are at
•    Australia Standard on Governance – transition towns: Heart, Head, Hand, grassroots, self-sustaining
•    Favourite warm up – yes, but = blocker, yes, and opens up possibilities

Turning project outcomes into reality and identifying next steps

Wednesday, May 5th, 2010
Topic: Turning project outcomes into reality –
Identifying the next steps
Leader: Zandy Tibballs
Participants: 12
Key Points:
Identify tools to use that will lead to wider uptake.  Tools such as:
Focus group work – contact primary leaders to help shape the type of information that is needed to be obtained,  > contact secondary group to determine banners and the type of information that they need to have to be willing to change.
Target specific members / leaders and tune into their radio station to identify and understand the needs of the target audience > identify pathways that the change leaders go down – what made them change, what’s needed to help others change.
Be adaptive to needs of target group audience – be willing to switch needs – identify “hook” & type of messaging to be used.
Identify & work with user groups capable of influencing the target audience group.
Pick the right champions / community leaders to help influence the target market.
Consider the use of creative evaluation, eg films, media to influence.
Utilise envisioning tools to help people identify type of future they woud like.
Make links to accreditators / certificators / cost savings / rewards / recognitors.
Undertake more independent measurement to highlight benefits and provide proof of effects that the program / products provides.
Go to where they go eg conferences that they attend and present findings.
Establish / or link “people” into a reference group where they can share information and network – “create opportunities” for target users to learn from each other.
Identify champions that the target group would value and respect.
Use formative evaluation / action research to help determine next steps.
Utilise simple marketing tools eg
1.peer review articles in relevant magazines;
2.develop fact sheets adapted from case studies, suitable for different audiences;
3.include different information for different audiences;  and
4.explain & promote the benefits / use diagrams

SHOW  ME  THE  CHANGE

Complexity and the Art of Evaluation – Reporting Sheet

Leader: Zandy Tibballs

Participants: 12

  • Identify tools to use that will lead to wider uptake.  Tools such as:
  • Focus group work – contact primary leaders to help shape the type of information that is needed to be obtained,  > contact secondary group to determine banners and the type of information that they need to have to be willing to change.
  • Target specific members / leaders and tune into their radio station to identify and understand the needs of the target audience > identify pathways that the change leaders go down – what made them change, what’s needed to help others change.
  • Be adaptive to needs of target group audience – be willing to switch needs – identify “hook” & type of messaging to be used.
  • Identify & work with user groups capable of influencing the target audience group.
  • Pick the right champions / community leaders to help influence the target market.
  • Consider the use of creative evaluation, eg films, media to influence.
  • Utilise envisioning tools to help people identify type of future they woud like.
  • Make links to accreditators / certificators / cost savings / rewards / recognitors.
  • Undertake more independent measurement to highlight benefits and provide proof of effects that the program / products provides.
  • Go to where they go eg conferences that they attend and present findings.
  • Establish / or link “people” into a reference group where they can share information and network – “create opportunities” for target users to learn from each other.
  • Identify champions that the target group would value and respect.
  • Use formative evaluation / action research to help determine next steps.
  • Utilise simple marketing tools eg
    1. peer review articles in relevant magazines;
    2. develop fact sheets adapted from case studies, suitable for different audiences;
    3. include different information for different audiences;  and
    4. explain & promote the benefits / use diagrams.

An audit and compendium of evaluation tools in current practice

Wednesday, May 5th, 2010

One open space session was called to produce an inventory of evaluation tools in current practice.  The two attached files are the results of that session.  Both are in .pdf format.

Show Me the Change Conference begins

Tuesday, May 4th, 2010

IMG_2493As I sit here it’s mid-afternoon on the first day on the long-anticipated Show Me the Change Conference. About 180 people have gathered in Melbourne to explore topics and pose questions, share ideas and tools around evaluating behaviour change. I’ve met people with a wealth of experience and others bring new enthusiasm; there’s conversations of all shapes and sizes; there’s the usual challenges of working in large groups.

Chris Corrigan opened space this afternoon and about 40 topics went up on the wall to be discussed today and tomorrow. I can sense more topics brewing.

While many people have experienced Open Space as a process before, for many people it’s their first open space gathering. Open space taps into people’s passion around the topic and enables them to set the agenda (rather than a designated group pre-determining what everyone wants to talk about/listen to). Open space is an example of complexity in action.

Every now and again it’s good to be reminded about what makes open space work. Many of us try and intellectualise too much, and make it more complicated than it needs to be. Harrison Owen reminds us about the four principles and one law of open space, and what these mean in terms of the practice of open space in our lives and organisations.

The Principles:
Whoever comes are the right people
Whatever happens is the only thing that could have
Be prepared to be surprised!
Whenever it starts is the right time
When it’s over, it’s over.

The Law of Two Feet
If, at any time, you find yourself in a place where you are not contributing or not learning, then use your two feet and go somewhere else.

In response to a comment on the Open Space List regarding internalising these principles and law, Harrison Owen, wrote the following. I think there’s great wisdom in this.

I suspect that it is more a matter of remembering what we already know and for one reason or another have chosen to repress. All of this goes with the idea that Open Space is truly not something new and radically different. In fact it is a forceful confrontation with a pre-existing condition. We are already in Open Space by virtue of the fact that we have forever been in a self organizing world (the usual 13.7 billion years stuff).

The Law and the Principles are descriptive of normative behavior in a self organizing world, and therefore Open Space, I think. In short, we do all of the above all the time — unfortunately we usually feel guilty about it, and because of this, we tend to do it/them badly, or at least awkwardly and grudgingly. Thus with the Law: when faced with a nonproductive situation (no learning, no contribution) we always leave (hearts and mind out the window) — but the body remains feeling miserable, and making others miserable as well. Once we get the picture, things work better, and we feel a lot better. But it is not about doing something new, or internalizing some new truth — but rather remembering what we already knew and doing what we should/could have been doing in the first place.

Why bother with all this? Well if nothing else, I think it makes our job as consultants and facilitators a lot easier. First of all we are not inviting our clients to engage in risky behavior. Quite the opposite, we are opening a space in which they can really be themselves. And the real risk is to continue with the non-productive, guilt inducing, dependant behavior. The old Marxist Battle Cry might have some application here (with modification): People of the World Unite — You have nothing to lose but your chains.” In a word — Be yourself!

Hear, hear!

Viv McWaters