Archive for the ‘Uncategorized’ Category

Building a culture of evaluation

Wednesday, May 5th, 2010


Complexity and the Art of Evaluation – Reporting Sheet

Leader: Steton Kantmon

Participants: Amy O’Brien, Pam Beattie, Nicole, Kate Patken, Keren Winterford, Catherine Doran, Natalie Moxham, Jen Orange, Narelle Chambers, Liz Franzmaan, Rob Catchlove, Julie Richmond.

Key Points:



  • Being able to learn from failure safely

  • Joint leadership management

  • Knowing why you’re doing it

  • Learning, reporting

  • System/roles in place that encourage doing, own acting on results

  • Requiring recommendations, input data, action
  • Success driven culture, not acknowledging failures

  • Lack of resources to act on evaluation

  • Pressure to report results before expected change can happen

  • Tenure of champions to start

  • Failing to integrate accountability reporting goal with learning/improvement goal

  • Reactiveness to short term/lack of strategy

  • Tangle up of identity/investment in approach


  • Have dedicated roles/ongoing

  • Recognise need for results of different scales

  • Ownership of evaluation by users; focus on doing it for us – primary audience is the program delivery

  • Don’t confuse evaluation for management, define framework at beginning

  • Keeping progress log as living document

    • win it

    • own it – bring governance/funder along

  • Distinguish monitoring from evaluation, but integration

  • Whole of organisation provision, accountability and affectiveness, local tailoring and modification, to help solve local problems and make decisions

  • Links to planning and goal setting – integrate with business planning

    • program log = budget

    • evaluation frame = actions

  • Rather than showing individual contribution, show collective contribution of partners/stakeholders

  • Have agreed/scaled indicators for different levels of intervention, different positives

  • Multi-layered collaboration for intervention avoids the need for competing claims to have solved problems

An audit and compendium of evaluation tools in current practice

Wednesday, May 5th, 2010

One open space session was called to produce an inventory of evaluation tools in current practice.  The two attached files are the results of that session.  Both are in .pdf format.

If so much change occurs through word of mouth, how do we evaluate it?

Tuesday, April 13th, 2010

There is general agreement that word of mouth marketing is a critical element of changing behaviour. Whether it is a family member, colleague, neighbour, or friend, we are more likely to take on the advice and behaviours that are modelled by those we trust. This is the basis of effective communication (think also of the 6 degrees of separation experiment). Mark Earls, the author of Herd: how to change mass behaviour by harnessing our true nature provides great examples of how social networks are key to changing mass behaviour.

Mark recently posted a blog about how important it is to understand social networks.

Mark notes: Social networks are not channels for advertisers or for the adverts/memes you, your clients or any of your so-called “influentials” create, social networks are for all of the people who participate in the network.

So if word of mouth is an element of your behaviour change program (as it should be), how can you track its spread, and find out whom the key people are in networks? Well, social network analysis is one way! So what is a social network analysis?

Andrew Rixon, from Babelfish Group, notes in an e-booklet on enhancing collaboration that Social Network Analysis is the technique of analysing roles and social networks…. The outcomes of social network analysis provides surprising and insightful results allowing structure(s) to become visible and discussable.

Making such networks visible should surely be one of the goals of  evaluation. In this way, for those who have read Gladwell’s Tipping Point, you can find out who the mavens, connectors and salesmen are.

Interested in finding out more on Social Network Analysis?

Andrew Rixon will be holding a post-conference workshop on this very topic, so check out the program of post conference workshops and register online.

Who else do you want to see at Show me the Change?

Thursday, April 8th, 2010

The conference has already attracted a great number of participants from a wide variety of backgrounds. And as this conference is not one where “experts” will be doing all the “talking to you”, the diversity of people will make for passionate and thought-provoking conversations.

If you have already registered, who else do you know who has something to share or someting to learn from this conference? Why not let them know that you’ll be there, and that you would value their participation and exchange of ideas.  You can download a conference e-card from here, which you can use to email those whom you think should also attend.

Download the conference e-card

Download the conference e-card

The more things change, the more they stay the same

Thursday, April 8th, 2010

This phrase, from the original in French “plus ça change, plus c’est la même chose” is very pertinent in our quest to change behaviours.

In a WWF report, Crompton (2008: 5) states: “The results of experiments examining the ‘foot-in-the-door’ approach (the hope that individuals can be led up a virtuous ladder of ever more far-reaching behavioural changes) are fraught with contradictions”.

What does this mean in terms of evaluation? Well, for one, self-reporting of changes may lead to socially-desirable answers that overestimate the amount of actual change.

So how do we undertake better evaluations? Well, hopefully this will all be revealed in the conversations that take place at Show me the Change. Which leads me to the following…………

The Abbotsford Declaration on Behaviour Change Evaluation

A recent post on Rick Davies Evaluation News Site on the Paris Declaration on aid effectiveness made me think about what we can collectively achieve from Show me the Change.

Based on the diversity of people attending I am sure that the conversations will be passionate and inspiring and the amount of knowledge exchange based on practical experiences will lead to better practices…. So how about working towards an Abbotsford Declaration on Behaviour Change Evaluation as a marker for this event, and as a building block for future behaviour change programs and their evaluation!

Pathways beyond Show Me The Change

Friday, March 26th, 2010

Show Me The Change is linked to a range of conferences in Australia and around the world. It seems like it is ‘the time’ for people to explore complexity and rethink the way we have been approaching behaviour change and it’s evaluation.

One such gathering is in Melbourne. Hosted by the folks at ALARA (Action Learning Action Research Association), this event is sure to attract people from our’s and visa-versa!

Ross Colliver was kind enough to provide some words to describe the conference …

Eighth ALARA World Congress 2010
Participatory Action Research and Action Learning

Appreciating our Pasts, Comprehending our Presents
Prefiguring our Futures

Eighth ALARA World Congress 2010 on Participatory Action Research and Action Learning, September 6-9th 2010, Melbourne, Australia. Plus two pre-congress skill-sharing days (4th-5th)

Meet people actively engaged in Action Research / Action Learning and its applied fields. Be stimulated by critical dialogues and reflections the the Praxis Streams of Social Ecology, Community Development, International Development, Health and Wellbeing, Systems /Business / Organisational Development, Education and Learning, and Decolonising Practice, including cross-cultural learning with Indigenous Peoples.

Meet practitioners of the ‘new’ generation. Talk and listen to ‘Elders’ who have contributed to the early growth periods of these methodologies and who continue to be initiators and innovators. Become part of local, regional and global networks practicing PAR /AR / AL.

Engage with like-minded people in conversations about philosophies underpinning action research, debate issues of power, deepen your understanding of methodologies. Brush up on the basics, gather materials, shared-knowledge and contacts to take Action Research / Action Learning with you into workplaces, associations, organisations and lives.

Ross Colliver
0411 226519

Which indicators are important?

Wednesday, March 24th, 2010

Thanks to Glenn Elliot at LaTrobe Uni for sending me this link!

Here is a 10 minute, narrated slideshow that gets into the nitty-gritty detail of confidence stats, confidence intervals and p-values.

The thing to note here is the ‘key message’ – in the psychology field (and probably many others), decisions are often made about the ‘success’ of research experiments based on a single number … the “P-Value”. This video highlights the randomness, variability and complexity of research.

It also uses a nice musical theme to reinforce it’s message. I wonder what numbers we get too focused on when running our behaviour change programs AND are they really measures of our success and brilliance … or are they simply an example of randomness and chance?

About to go 'LIVE' with registration

Tuesday, February 23rd, 2010

Yes, we are about to ‘go live’ with the full conference program, post conference workshops and an online system so you can book your place at Show Me The Change!

Conference Design Team & Steering Committee

The Art of Metaphor

Monday, January 25th, 2010

Here is a short presentation by James Geary @ TED on ‘Metaphorically Speaking

About 5 minutes in, James talks about how our langauge influences the way we interpret the world and in the decisions and predicitions we make. Metaphor is another lens through which we can view human behaviour.


It's about conversations and relationships

Wednesday, January 20th, 2010

A while back, Chris Corrigan wrote this post called ‘Lessons About Invitation‘.

It’s about relationships and conversation. Elders tell me time and again after years of working in organizations and living in communities that the quality of life and work always comes down to relationships.  Younger adults and youth will talk about action and outcomes and getting things done productively and efficiently, but older people, who have time to reflect on their careers constantly tell me that focusing on relationships is more important, for quality, sustainability and effectiveness.  With this in mind, invitation needs to be about relationships and conversation too.  An one page written invitation is a sterile beast.  It does not reflect the mode of being that we are inviting people too.  If we want people to enter a conversation, we need to invite them there WITH conversation.  So reach out beyond sending out the email, embody and practice invitation with relationship building and conversation.”

Thought I’d share this with you all.