SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Tani

Participants:

Key Points:

Two year program, community garden, start mid end evaluation.

•    They used nickname to provide stats for confidentiality
•    Completed mid project workshop for evaluation and sharing
•    Interviews at end, one on one
•    30 participants
•    Used consultancy for survey / evaluation
•    Easy to get feedback due to already formed group.

“Living 4 our future”

City of Boroondara – sustainability, water, food, waste program pre & post evaluation, measured footprint, 3 workshops, incentives provided saw concrete outcomes by com parity water bills etc.

Good tool – global footprint tool, EU states most creditable tool.

A reduction was seen – it resource use.

“Most significant change” – method = story telling methodology.

On-going indepth long-term interviews highly regarded.

Key = acting, reflecting, learning.

“Participationary approach” – google for websites.

“Action Research”

Outcome mapping.

Spider diagrams – set criteria to evaluate work – shape changes over time.

Radar graph – participatory method.

It is about the people involved creating the data to be measured – visual immediate.

Role modelling method – stakeholders take role of group / individual.  Empowering as it goes.

Assists in seeing how the individual is effected.

Only suitable on a small scale.

Also suitable on a small scale.

Also suitable for team members in acting out the issues.

Useful to ask “critical” friends to look at your evaluation tech.

Valuable way to progress knowledge is to keep talking, meeting, discussing – challenges research to stay relevant.

Make evaluation embedded in process and project plan.

“Program logic” design tool for planning clarification evaluation.

You can complete an internal evaluation for your team for learning and internal use only.  Then when completing the external evaluation this should be an easier task.

Very valuable to verbally have a conversation with participate.

QSR – Doncaster.

“MVIVO” program – can record conversations have them typed up then placed in “MVIVO” program to make it rigorous = code issues to sort collected data.

Two questions to ask:
1.    What would you say to others to attend XYZ?
2.    What would you say to others as to reasons not to attend?

All good evaluations should use a variety of tools to get different angles.

•    Book – the trianglisation design
•    Have / use multiple evaluation tools to ensure you have the right data / answers somewhere along the way
•    “Plan-do-act-check” – around since 30’s
•    The risk analysis of your project plan will guide you on where you need to focus and guide your evaluation
•    If evaluating your own program – a diary is handy to keep
•    Debrief with others to get learnings.

Allow all learning styles to communicate in their style:
1.    amiable
2.    kinaesthetic
3.    verbal
4.    auditory

Don’t just evaluate in the style you prefer when collecting evaluation !!

Tool – photo – learning cards.

“Gallery technique” – people work on walls and present / flipchart.

SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Tricia Hiley
tricia@potentialspace.com

Participants:

Key Points:

We had a rich and flowing sharing of stories, people working with projects in regional or rural areas.

Some of the themes included:

1.    Lack of access to services increases isolation.
For instance –

•    No local insulation installer and no local relationship with one from “away”
•    Home sustainability assessors not funded to travel to rural locations (ie travel costs paid from post office closest to clients house)
•    Don’t have local trades people
•    Not eligible for water tank rebate because they are not on town waters
•    Regional project staff may only be able to do one visit a day because of distance compared to 6 or 8 in the city

How do we take this “systemic” consequence into account when planning and evaluating ∆ projects?

2.    Enthusiasm and involvement of whole communities.
Examples include:

•    Heyfield, where the whole of the community has embraced a sustainable town project.  It includes a “game” where people can, through their sustainability efforts, “attain” a white then green then blue flag to place prominently on their roof for all to see and celebrate.

•    Mildura, which has benefited by being a small, discrete community which has an inherent focus such as a city community may find it difficult as residents are members of so many different “communities” (ie living, working, school, shops etc.).

3.    There was a reasonable amount of sharing of the experience of vulnerability in farm families, particularly where there is a single income in which leads to a perceived lack of robustness – which can lead to resistance to change.  One group member introduced “crunch theory” which considers a community’s vulnerability with respect to natural hazards.  The example of the consequences of a similar sized earthquake to Haiti and Chile was given as an example.  In Haiti over 200,000 died.  In Chile it was in low hundreds.

4.    A couple of images that linger for me are  (I NEED THE ARTIST HERE)
a)    “the value of one tree in the paddock”
b)    a small town with a white, green or blue flag on the roof of every building
c)    “the country wave” as a young woman waves to or says hi to everyone she passes
d)    a set of scales with “natural hazard” on one side and “community vulnerability” on the other
e)    a project worker in the city – a project worker in the country

SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Chad Foulkes

Participants: Many

Key Points:

Liminal space is where I discover what is and what could become

•    Teaching people to juggle (let go)

•    Questioning

•    Reflection – silence – structured – personal

•    Case study – see others experience & relate it to ones own experience

•    Loosen patterns & allow new / other things to connect to it

•    Simulations

•    Make a statement, stop talking & wait for others to contribute

•    When there is no stated “activity” yet the group “agrees” to do what it does – this can bring about change

•    Disrupting habits is our challenge

•    Humour as a way to get to limited space

•    Facilitate self evaluation rather than evaluate what happened

•    The report is the artefact, the work has been done then

SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Candyce Presland

Participants:    Many

Key Points:

•    Making the information relevant to the audience know the audience

•    Using interim evaluation data to evaluate throughout the program

•    Collaboration

•    Consistent messaging (not having too many messages or mixing them)

•    Sharing information about network

•    If a program doesn’t get “buy-in” learn from it and move onto something that will

•    Name and shame if necessary (as a last resort)

•    Look at the ithree methodology

•    Sell it – media – photo opportunity

•    Have a champion or influence

SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Candice Feuerring

Participants:

Key Points:

•    Indigenous Sustainability Festival – St Kilda

•    WTC – could seek to tie in sustainability messages

•    Strength of faith communities – focus on volunteering

•    Linking sustainability messages into religion

•    Reconnecting the 1st generation back to their culture

•    Issues with engaging older people from CALD more likely to engage

•    Issues with translating into languages and whether 1st generation use their language to extend their ……..  ?? / older generation

•    Difference between tailoring messages v’s generic communications

•    Difference between how you approach sustainability for different groups

•    Accessing community leaders from CALD groups to inform programs and provide advice on messages = these community leaders are generally happy to be active and are already doing this

•    Remembering that CALD groups are diverse within themselves, not all of one community has same priorities

•    Visiting community groups and going to them and where they are already, meeting = connect with existing networks.  Outreach is vital!  “Don’t ask communities to come to you, go to them”.

•    Talking stick / message stick = idea of having conversation and that being passed onto others

•    Issues:  involving diverse groups

•    Need for a process of enquiry = we should be teaching each other

•    Mainstream needs to learn and embark on a process of enquiry and provide space for everyone to be teachers

•    1st generation creates community then the 2nd generation lives bi-culturally

•    2nd generation may have more choices because of their ability to live bi-culturally

•    The knowledge of the 1st generation in creating community is something that should be learn by mainstream

•    Existing networks spread the message much quicker than recreating new networks = building on existing networks / structures is undervalued

•    Ethics of asking people to do something that you will step away from = how do you go in and work with a community as a paid person and then ask others to do it voluntarily

•    How do we turn a specific non-mainstream message into mainstream and why do we need to do it?

•    Why do we focus on difference?

•    What has happened in our lives that was difficult and how we respond to it?

•    People from other cultures come with experience and adversity that brings insight and resilience

•    Vulnerability brings strength

•    Last 15 years = problem with identity / politics

SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Stephanie Madeley

Participants: Danielle Kennedy    Victoria Johnson
Merryl Whyte    Simon Kreeure
Kerry Edgecombe    Jon Wapainakefe
Felicity Wardlaw    Viv Beufon
Karen Parissien

Key Points:

Inability, disinterest, not usually have a correlation – its more that (low socio-economic) groups want to change but there are too many barrier to change.

Access – go out, don’t expect them to com to you.  Organisational flexibility is very important and ask them what they want.  Community often has the answer.  The role of schools in a community can be profound.  This is part of the importance of relationships.

Should we place greater importance on social change at a societal level?  Would this make things easier?

What is the relevance to people (eg economic hardship)?

SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Fay Valcane
fvalcane@museum.vic.gov.au

Participants: Kathryn McCullum    Deb Orb
Stephanie Madeley    Jane Lieman
Viv Benton    Linda Duffy
Greg Bruce

Key Points:

•    Telling stories around climate change in an engaging (immersive way).  Creating experiences that put people in “changed environment”.

•    Prof Sam Hom US Thematic Communication & Tourism Industry Int Physiologist was quoted as a good contact to make.  He will be based at Latrobe Uni, …………?? Oct / Nov 2010.

•    Working in partnerships with key organisations, where the museum can be the neutral trusted authority.  Provides platform to engage with experts in the field under one collective point – and the museum not re-inventing great work being done, eg Deakin Uni and Wheeler Centre lectures.

•    Targeting mums with young kids – museum can provide safe place to bring mums together and equip them with strategies to change behaviours, creating safe spaces for discussions to occur.

•    Work with something that goes to where people are, not wait for them to visit you (at the museum).

•    Museums are places that provide opportunities for brave discussions, provide balance …….?? Sceptic views & science non-sceptic views.

•    Follow up with Monteray Bay Aquarium in UK who are also looking at ways of addressing climate change issues.

•    Use people and their stories to help communicate issues to engage audiences rather than tackling it by a “we are experts approach”.  Sited Jane (surname ?????) a Professor at Flinders University, for eg of great approach on thematic communications.  Example was given where Jane talks about her passion for geology – rocks via taking people on a journey that they are interested (eg their migration history) and finding a creative link to a story on the geology of the area (this sounded very intriguing)!

SHOW  ME  THE  CHANGE

Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Rebecca Petit

Participants: ZandyTibballs    Martin Hausenblas
Fay Valcanis    Darlene Pertland
Jonathan Day    Megan Hughes
Merryl Whyte    Felix Acher
Debbie Megyar    Michael Baranovic
Helga Svendsen    Michelle Lambert
Millicent Burke    Anjali Pal
Bridge Wetherral    Ian Blair
Brian Hardy    Marcia Hewitt
Greg Campbell    Julia Davies
Danielle Kennedy

Key Points:

•    CBSM – use of tools, prompts etc. case studies work, may not go deep enough with values – moving from energy – water – www.cbsm.com

•    Future perfect – green home – way of visioning a utopia

•    YVW behaviour change barrier breakdown process, levels of engagement, water smart – based on social data methodology, 2 yrs of data before 9000 – based on.  Produced data around self reported change

•    Internal and external barriers eg can’t affect my water bill – cost

•    Board – aware of change, energy knowing its usage and cost eg usually swimming pools.  Black balloons campaign marketing not tangible.  Was successful in some visual aspects.

•    People are so different – amount of positive thinking – 3 important parts of it – what will be the outcome of the action for me?  2 – social pressure – is there a group of people around me who care – males, gov, neighbours etc. and 3 – barriers – what are they – money, time etc?   ……………………..????? these barriers there are people who are willing and people who are not willing.  Survey – what are those beliefs?  Eg Townsville example, people will do it for environment etc.  I…………………………………………….????Positive thinking to support this target. (name of the model) Author fishbeinicekajzen – predicting action and change behaviour, the reasoned approach.

•    Need a multi-prong approach eg legislation, cultural norms, beliefs, etc.

•    Household relationship & habits how to effect

•    Transition towns – read the transition handbook by Rob Hopkins.  Connecting communities and building resilience in peak oil, conversations, relationships then transactions.  The sharehood – www.thesharehood.org   CSA – community sustainable agriculture – similar views

•    Princes Hill High School & Lauriston on the school building – solar panels, water tanks etc.  Assi – Australian Sustainable Schools – resource smart is the Victorian Section – SEC – SV-website

•    Jika – Jika – growing greening neighbourhoods house

•    ANU – does network analysis look at how connected the communities are and what model will you use to target group eg aids example on unit address large group and disconnected groups.

•    Transition book self funded from day one – to do your own and have active people – steering committee to be Rachel Lowery – palm oil.  Act for nature campaign – commitment chosen

•    Solar critics – Townsville – magnet and follow – 6 months later

•    Increase your own commitment to remind yourself

Themes
•    Behaviour change models and theories CBSM

•    Community grass roots organisations that work eg transitions towns.  The sharehood etc. examples of demonstrations that work eg Princes Hill high – solar panels on school, raintanks etc – growing greener neighbourhoods etc.

Examples of programs, other councils, sustainable homes, greenhouse ACF etc SLAH – sustainable living at home – evaluation on website.

Topic:    Storytelling as a form of evaluating change behaviour

Key Points:
•    Storytelling can be in narrative or anecdotal process
•    DSE fire have used it as a debrief for recent fire season – same participants had trouble “storytelling” (adapting to change in debrief method)
•    To help use stories as data, sifting process allow for “themes” to come out of process for evaluation
•    Storytelling can be used as a tool and measure in behaviour change
•    Is storytelling really that different from one on one interview or focus groups?
•    How can we place storytelling with statistics?
•    Need to have storytelling throughout project – ongoing evaluation

•    Obama campaign used narrative / conversation
– self         )
– group     )    used by local people to spread
– now        )    questions / stories

•    Need to record failures and successes as part evaluation
•    Storytelling is significant to tellers of their story – individual learning and recognition can come from storytelling
•    Self story / narrative – Terese Riley – Penny Hall – Prof Stuart Hill – purposeful lying ??
•    Narrative analysis –
–    anecdotal
–    used debrief DSE – begin / middle end – challenge, choice, outcome
–    personal stories have / give credibility
–    anecdotal circle
–    digital storytelling, ………………..    ?????  children

•    How is it different from focus group – case study
•    Stories resonate with people
•    Sift stories – the story that resonates the greatest in the group
•    Significant – significant for you? – for the people for the storyteller?
•    Conversation – interview – action tool – Barack Obama / self / grip / now
–    measure of change
–    measure of of tools / behaviour change
–    could develop measures

•    Use community to help
•    Social element
•    Not the only way to evaluate change
•    Start process at beginning
•    Stats v’s story ??
•    Recording failures and successes
•    How can we allow for change / wave of change
•    Evaluation and transference of story experience