Archive for March, 2010

« Older Entries

Overcoming existing evaluation cultures and processes

Monday, March 29th, 2010

A post on evaluation and complexity on Rick Davies monitoring and evaluation news site has a link to a great powerpoint on evaluation and the science of complexity by Ben Ramalingam. Ben notes that for many organisations, evaluations are at the centre of a vicious circle that includes pressure to show results and impacts, and poor learning and accountability amongst others.

Further, Ben notes that “Evaluations are still largely focused on reports as opposed to changed behaviours, ways of thinking and attitudes”. This seems very true, and I am sure many of us would recollect knowing of reports that have been produced for the report’s sake, and not what is in it.

A nice slide from Ben Ramalingam's powerpoint

Image source: http://www.outcomemapping.ca/resource/resource.php?id=183


The image, taken from one of Ben’s slides, encapsulates well the idea that existing process and culture can overshadow the ability to undertake more effective evaluation.

In another post, Ben notes “Some of the issues for evaluation include the tension between learning and accountability, the limits of attribution, how evaluations are or are not used, equality and power, and ideological debates about methodologies, such as the dominance of randomised controlled trials (RCTs)……Although there is a wealth of evaluation methods in theory, in practice they are largely required to conform to scientific management principles……..In contrast, complexity theory (theories) talks about systems that are interconnected, driven by feedback, where the properties of the system are not predictable but emerge from the relationships within that system……..It may be that we need to stop focusing on projects, and look more broadly at the societies that we work in and across sectors and institutions rather than within them. Evaluations may need to be more centred on real-time learning and helping managers adapt what they do.”

This is what Show me the Change is about- discussing how as a community of practice we can overcome the real or imagined culture that can negatively impact on more novel, experimental, and altogether better evaluation practice and processes.

So if you are interested in evaluation, behaviour change and sustainability in a complex workd, take part in the conversations that matter, on 4-6 May in Melbourne.

Tags: , ,
Posted in Evaluation | No Comments »

SLAH Sustainable Living At Home

Saturday, March 27th, 2010

SLAH was a behaviour change program for City of Port Philip residents, continuously improved over 9 rounds from 2001 to late 2008.

The format of SLAH was a series of 5 program workshops run over 3 months, attracting between 60 and 80 households participating in each round. The workshops focussed on practical actions around energy, water, waste, travel and spending. In addition to expert presenter information, householders inspired each other via group activities and discussion to implement actions relating to each theme.

Incentive products were supplied at each workshop (e.g. CFLs, worm farms) to encourage retrofit or behaviour change action.

Posted in Case Study | No Comments »

Pathways beyond Show Me The Change

Friday, March 26th, 2010

Show Me The Change is linked to a range of conferences in Australia and around the world. It seems like it is ‘the time’ for people to explore complexity and rethink the way we have been approaching behaviour change and it’s evaluation.

One such gathering is in Melbourne. Hosted by the folks at ALARA (Action Learning Action Research Association), this event is sure to attract people from our’s and visa-versa!

Ross Colliver was kind enough to provide some words to describe the conference …

Eighth ALARA World Congress 2010
Participatory Action Research and Action Learning
Appreciating our Pasts, Comprehending our Presents
Prefiguring our Futures

Eighth ALARA World Congress 2010 on Participatory Action Research and Action Learning, September 6-9th 2010, Melbourne, Australia. Plus two pre-congress skill-sharing days (4th-5th)

Meet people actively engaged in Action Research / Action Learning and its applied fields. Be stimulated by critical dialogues and reflections the the Praxis Streams of Social Ecology, Community Development, International Development, Health and Wellbeing, Systems /Business / Organisational Development, Education and Learning, and Decolonising Practice, including cross-cultural learning with Indigenous Peoples.

Meet practitioners of the ‘new’ generation. Talk and listen to ‘Elders’ who have contributed to the early growth periods of these methodologies and who continue to be initiators and innovators. Become part of local, regional and global networks practicing PAR /AR / AL.

Engage with like-minded people in conversations about philosophies underpinning action research, debate issues of power, deepen your understanding of methodologies. Brush up on the basics, gather materials, shared-knowledge and contacts to take Action Research / Action Learning with you into workplaces, associations, organisations and lives.

Ross Colliver
TTDG
0411 226519


Posted in Uncategorized | No Comments »

Which indicators are important?

Wednesday, March 24th, 2010

Thanks to Glenn Elliot at LaTrobe Uni for sending me this link!

Here is a 10 minute, narrated slideshow that gets into the nitty-gritty detail of confidence stats, confidence intervals and p-values.

The thing to note here is the ‘key message’ – in the psychology field (and probably many others), decisions are often made about the ‘success’ of research experiments based on a single number … the “P-Value”. This video highlights the randomness, variability and complexity of research.

It also uses a nice musical theme to reinforce it’s message. I wonder what numbers we get too focused on when running our behaviour change programs AND are they really measures of our success and brilliance … or are they simply an example of randomness and chance?

Posted in Uncategorized | No Comments »

Using fun for behaviour change

Monday, March 22nd, 2010

Here are three resources which have recently crossed my path that involve using fun and games for social change.  Some of these work with groups and some work across social spaces – demographics, communities or organizations.  What I like about these games is that they provide a built in set of measureables that can be used to gauge progress and evaluate behaviour change.  Sesms like combining fun, visible change and simple yet powerful standards for noticing shift is the holy grail in this kind of work.

Games for Change: Games for Change (G4C) is a non-profit which seeks to harness the extraordinary power of video games to address the most pressing issues of our day, including poverty, education, human rights, global conflict and climate change. G4C acts as a voice for the transformative power of games, bringing together organizations and individuals from the nonprofit sector, government, journalism, academia, industry and the arts, to grow the sector and provide a platform for the exchange of ideas and resources.


The Fun Theory:  We’ve blogged this before, but The Fun Theory is “dedicated to the thought that something as simple as fun is the easiest way to change people’s behaviour for the better. Be it for yourself, for the environment, or for something entirely different, the only thing that matters is that it’s change for the better.”

The FreeChild Project: Lots of games and resources at this website dedictaed to youth engagement around social change.  FreeChild has been working for almost eight years to promote the idea that when engaged in meaningful ways throughout society, the knowledge, action and wisdom of children and youth can make the world more democratic, more non-violent and engaging for everyone. By working with adults as allies young people learn, teach and lead democracy throughout society!

Posted in Behaviour Change | No Comments »

Action & change challenge

Thursday, March 18th, 2010

The US National Coalition for Dialogue and Deliberation has just released a report on democratic governance. The action and change challenge is of particular interest:

More and more people are coming to realize that addressing the major challenges of our time is dependent on our ability to collectively move to a new level of thinking about those challenges, and that dialogic and deliberative processes help people make this leap.  Yet we continually struggle with how best to link dialogue and deliberation with action and change, and with the misperception that dialogue and deliberation are “just talk.”

You can download the full report here, or download a 3-page overview here.

Sandy Heierbacher, author of the report, also highlights a couple of promising frameworks.

Maggie Herzig’s “Virtuous and Vicious Cycles” model is presented, which acknowledges the systemic and cyclical nature of dialogue and deliberation (as opposed to a linear progression of steps or stages).  And Philip Thomas’ integral theory of dialogue seeks to reconcile the seemingly incompatible views of dialogue he came across while working on the Handbook on Dialogue published by the United Nations Program on Development and its partners.  Thomas interviewed some practitioners who felt, for example, that personal transformation among dialogue participants was a critical outcome to emphasize in the Handbook, while others he interviewed wanted to de-emphasize and even eliminate such concepts from the book and focus primarily on political processes and outcomes.

Viv McWaters

Posted in Behaviour Change | No Comments »

To evaluate or not to evaluate?

Wednesday, March 17th, 2010

This is a question that we don’t really ask ourselves, as we are often made to evaluate (to account for the money we received and spent), and to is also in our nature to demonstrate success (and hide failure..??).

A recent post by by Chris Brogan,   that we should pursue the goal, not the method (also picked up by David Gurteen, made me think about how this applies to evaluation. Often, we consume ourselves on developing and implementing a methodology to evaluate change, as opposed to actually focussing on achieving the change itself.

This made me recall a conversation with Greg Bruce, from Townsville City Council, where he and his team are achieving some transformative change, across the community, but also throughout the Council and partnering organisations.

Greg wondered whether it was of any use to focus so much effort on evaluation, especially considering the short time frames in which traditional project evaluation is undertaken (that is, during and immediately post-project). Why spend so much time, effort and money, when you could re-direct the focus towards on-ground action and transformation. For Greg, evaluation “needs long timeframes to evaluate impact to effort – in system dynamics in order to show effect”. Greg proposed that equally good indicators of success would include the buy-in of other organisations (whether through official or unofficial partnerships), receiving further grants, as well as recognition from local, national and international delegations (in in Greg’s humorous ways, “they will come and go as equally mystified as when they arrived“).

So, how much effort should we put into evaluating something that will likely occur over a long time frame, way beyond the scope of the project, or interest of funding agencies? Let us know what you think? Post a response, ask the question amongst your colleagues and network partners, and share your conversations at Show me the Change.

Posted by Damien Sweeney

Posted in Behaviour Change, Evaluation | No Comments »

Agility and Change

Wednesday, March 17th, 2010

Neil Perkin penned this terrific post Agile Planning.

6a00d8341d4dc653ef01310f98bf5a970c-500wi

Image Source

This post resonated loudly with me because of the recent work I have been doing as a facilitator with Government agencies. Also, the upcoming Show Me The Change conference in Melbourne this May is also linked.

Here’s a snapshot of Neil’s Agile post:

We are all bombarded by change and it’s accelerating (and will not slow down) … the enterprise of the future will be hungry for change … but our current business/government are stuck in linear systems that are slow, top-down & inflexible …

“Our structures need to be more speedy. Speed used to kill now lack of speed kills. Lets have organizations that can iterate quickly and empower its folks to make decisions. Percolating decisions up and down an organization makes little sense” Rishad Tobaccowala

… being ‘agile’ is not a process or panacea – it’s a philosophy that you either have or you don’t … Agile is:

… welcoming of changing requirements, even late in development, because it is an opportunity to harness change for competitive advantage. Big business creates big projects that take a long time to confirm, implement, and complete. Projects are often stalled by hierarchical management processes.

… focused on frequent deliverables, with a preference for shippable product and shorter cycles, and implemented at a constant pace which is measured and transparent.

… centred around the belief that the best results come from self-organising teams. Teams that reflect regularly on how to become more effective, then adjust behaviours accordingly.

… projects are built around trusted, motivated individuals who are given the environment and support they need.

… documentation is kept to a minimum, with face-to-face communication preferred, and a focus on simplicity – maximising the amount of work not done.

And I really like this paragraph as it sums up my observations of the planning processes that so many of our large institutions are stuck in …

“I don’t claim here that agile development processes are some kind of cure-all. But I do think that business processes in many industries and organisations are woefully out-of-date and hopelessly rigid. Businesses increasingly operate in complex adaptive systems which, as Bud Caddell rightly says, are “characterized by perpetual novelty – talking of equilibrium is pointless, equilibrium in a complex adaptive system is essentially a dead system”. Inflexible, long-term strategic plans are increasingly irrelevant.” Neil Perkin

Show Me The Change

Show me the Change_eCard_Final

Ok, so what’s this conference got to do with Agility? Everything!!!!

Participants who are coming to Show Me The Change (deep down) know that our current approach to the design, staging and evaluation of behaviour change projects needs a re-think. We need to take stock and challenge our assumptions about behaviour change and the goals that we set. We have tinkered with ‘behaviour change’ tools and tactics for long enough. It’s time to go deeper.

In the context of ‘Behaviour Change projects’, agility is a mindset and NOT a set of processes, strategies and tools. It’s a philosophy that everyone involved adopts.

In practice, I think it involves these types of tangible features (some adapted from Dave Snowden’s post here):

  1. Do lots of little things rather than 1 big thing – Don’t put all your resources into 1 strategy … now matter how much ‘planning’ and research you have done
  2. Don’t be afraid to experiment and some things will fail – We often learn more from failure than success anyway
  3. Design projects that can be ‘monitored’ – By monitor I mean we can quickly notice if things are working or not. It’s no good if you need to wait 12 months for the data you need!
  4. Beware of ‘Best Practice’ and the success of others – Why? Because your context is different. Just becasue a series of home assessments lead to a reduction in household energy use in Town A, it doesn’t mean the same will apply in Town B
  5. Challenge the ‘Goals’ and ‘Evaluation Measures’ imposed on you by funders – Often we get asked to report a whole heap of indicators that are useless. At times, even the underlying goal of a project is unattainable … it’s aspirational at best. Don’t get sucked in to being ‘measured’ against such goals/aims. Have these conversations early.
  6. Challenge the direction and priorities of your project as it emerges – If you pay attention and monitor your project, new insights will emerge. Many will be unexpected and point to new priorities. Don’t be afraid to challenge your project plan and re work it along the way. Again, have these conersations early on.
  7. Collect and share Stories – Stories of failure and success are critical. Stories are memorable and they stick. Analysis of stories can uncover patterns that data analysis misses.
  8. Don’t be fearful of failure – Did I say that already?

Geoff Brown

Tags:
Posted in Behaviour Change, Evaluation, Fear | No Comments »

Exploding cylinders, complexity and evaluating behaviour change

Wednesday, March 10th, 2010

Last July, and an oxygen cylinder aboard a Qantas plane suffered what could be considered a relatively rare “behaviour change” in that it exploded, ripping a large hole in the fuselage. Following a stringent follow-up investigation (or evaluation), the cause of the exploding cylinder still remains a mystery. The news item reported that investigators even pressure tested the remaining gas bottles and none failed.
What this shows is that no matter how much knowledge we have about something, or no matter how many tests we replicate, some things remain a mystery. The reason behind the exploding cylinder are complex, and understanding the reason cannot be through attempting to replicate the problem by studying other cylinders, but through understanding the emergent properties related to that particular exploding cylinder. If only cylinders could tell a story? Except for the fact that this one is lost somewhere in the ocean!
So, what does that have to do with evaluating behaviour change? Well, people, like oxygen cylinders, are often considered similar and predictable in that what works for one is considered to work for others. But in reality, we are more like the “exploding cylinder” in that we often react unpredictably, or in a complex and unique manners, when placed in different situations. As such, it is hard to know what parameters to evaluate in behaviour change programs, as we cannot necessarily predict the outcome.
In understanding the reason for a change, what is important is asking the “one that changed”. If only investigators could get the story from the most significant (or exploding) cylinder? In evaluating behaviour change programs, we need to be more open to emergent properties, some which may be instantaneous and highly observable (like the exploding cylinder) and others that may occur over a longer term, and less visible.

Posted in Behaviour Change, complexity, Evaluation | No Comments »

Resources for evaluation in complexity

Saturday, March 6th, 2010

A group of Canadians are working with the public health system in Nova Scotia to shift the way health care is delivered to a more collaborative model. Along the way they are running capacity building workingshops exploring the Art of Social Innovation. In response to a recent call from these hosts, several resources have been posted on the Art of Hosting email list relating to complexity and evaluation. Here are a few of them:

All of these methods have applications for participatory initiatives in complex systems.

Posted in Evaluation | No Comments »

« Older Entries