By Professor Brydie-Leigh Bartleet
Impact is something that cuts across the lives of artists both outside and inside the academy. Considerations of impact have always been core to creative artists as they have activated the power of the arts to influence people, places, practices, and politics across many diverse cultural contexts and time periods. What is far newer is a bureaucratic turn towards measuring these impacts for the sake of accountability and measuring return on public investment.
Recently we have seen major shifts in our national research landscape driven by the Government’s innovation and science agenda, which presents both possibilities but also risks to the arts ecology here in Australia. The government’s new impact evaluation exercise has emerged as a significant driver in this landscape. As the government says: “While the success of university research can be viewed in measures of excellence, it can also be found in its economic, social, and environmental impacts. Assessing and reporting on how our investments in university research translate to tangible benefits for Australia will help show where collaboration with industry and other partners could bolster and more quickly deliver these benefits.”
The kind of impact accountability that is pervading higher education is not new to artists outside the academy. For many decades arts funding bodies have had various manifestations of criteria around impact or broader societal contribution, and/or required evaluations that speak to the impact of the project and their return on investment. As the director of a research centre, I frequently get requests from artists and arts organisations to evaluate the impact of their projects to meet the requirements of these funding bodies. What they are asking for is not always in-depth research, but rather proof that what they have done has had some kind of impact. Here evidence gets confused with advocacy, where arts organisations often want metrics that prove their desired impact rather than trying to understand it.
In these recent developments in higher education and the arts sector, we have also seen a tendency towards quantifying and measuring the instrumental impact of the arts. By their very nature, these exercises seem to focus more on measuring the ways that the arts serve social, economic and environmental agendas, rather than trying to understand the intrinsic, conceptual, and indeed aesthetic processes that are central to them. I have certainly felt these tensions in explaining the multidimensional impact of my community-engaged, relationship-focused, arts-based work in the one-dimensional text-based form required by the Government’s Impact and Engagement Exercise. This form uses transactional terminology such as end-users and beneficiaries, and privileges a limited definition of impact. Significantly, it asks for this case to be made in a small number of words and numbers only. The resulting truncated narrative and metrics only provide one part of the picture.
A major risk in quantifying and narrowly measuring the social and economic benefits of the arts in this way is that measurement becomes prized and rewarded over understanding how these benefits actually work. You might say it is a case of “the tail wagging the dog.” As Meyrick, Phiddian and Barnett (2018) argue in their recent book, What Matters? Talking Value in Australian Culture, as culture gets more complex, so must the ways we evaluate it. They suggest that we are going backwards in our obsession with quantification, and turning arts and culture into something to be scaled, measured and benchmarked, rather than trying to better understand its interconnected instrumental and intrinsic value.
Within the field of cultural policy, there has been significant critique of this limited focus on the instrumental impact in the arts, and its resulting metrics and measures. It has pointed out that such an approach to thinking about measuring impact is highly problematic because the arts operate as part of complex ecosystems of forces and factors, where cause and effect explanations are rarely sufficient, and systems of data collection struggle to find effective ways of measuring the preventative role of the arts. Moreover, responses to the arts are always personal and individual (some people are radically transformed by a particular arts experience while others are left unmoved), and capturing the subjective impact is difficult. Current thinking in this field advocates for a much more integrated way of thinking about measuring impact in the arts (see for example, Reframing the Debate About the Value of the Arts.
This raises questions about how we respond to these external pressures to measure our impact when we know that it is a far more complex process than the government (or indeed some funding bodies) sometimes perceive it to be. We could ignore it, and hope it goes away, but unfortunately that seems unlikely and maybe even a missed opportunity. Rather, it might be worth thinking about how we challenge and turn this impact exercise on its head. My sense is that we are not going to find inspiration nor wisdom in government guidelines, but rather rethinking definitions of impact in our field, and how we can develop and implement robust methods and measures that can in turn enhance our practices and how we understand them.
We don’t need to reinvent the wheel. There is a significant amount of work that has already been done on this topic here in Australia. This work focuses on a wide range of methods and approaches to understanding, measuring and communicating impact. MacDowall, Badham, Blomkamp and Dunphy’s (2015) pivotal collection essays explores the expanding field of cultural measurement and presents new approaches and frameworks for accounting for culture in local, national and international contexts. Gattenhoff’s research (2017) explores ways to move beyond narrow, acquittal-focused evaluations of impact and cultural value to far more dynamic and participatory partnerships between the evaluators and arts organisations. Likewise, Meyrick, Phiddian and Barnett’s research (2018) shares a number of examples of cultural practice that are defying the current reliance on metrics. The Australia Council for the Arts’ latest national arts participation survey “Connecting Australians” has focused on the impact of the arts, in both instrumental and intrinsic ways (see table).
A recent report from the AHRC on understanding the value of the arts and culture also suggests speaking to the question of metrics in a way that resonates with more holistic understandings of impact in the arts (Crossick & Kaszynska, 2016). It advocates for more formative and participatory evaluations of impact, as opposed to more summative and non-participatory approaches. It suggests that accounting for human experiences of the arts calls for multi-criteria analyses and a range of approaches from artistic research methods, peer review, case studies, benchmarking, cost-benefit analysis, hindsight studies, and so on. In essence these arguments and methods suggest that we need to find more rounded ways of integrating instrumental and intrinsic impact and value.
As these examples show, within the arts sector we have the know-how to turn the current impact agenda around from its rather narrow focus on quantitative metrics, fragmented measures, and instrumentalised outcomes. Our experience and insights position us perfectly to advocate for more holistic approaches to evaluation and measurement that combat such constricted and oftentimes short-term thinking. However, this know-how seems to be struggling to cut through conceptually in the current political landscape. The pressing challenge for us seems to be how we translate these nuanced and sophisticated ways of engaging with impact in the current environment, and play a leading role in transforming prevailing ways of measuring impact in both higher education and the arts sector more broadly.
Crossick, G. & Kaszynska, P. (2016). Understanding the value of arts & culture. Swindon, UK: Arts and Humanities Research Council.
Gattenhof, S. (2017). Measuring impact: Models for evaluation in the Australian arts and culture landscape. London: Palgrave.
MacDowall, L., Badham, M., Blomkamp, E., & Dunphy, K. (Eds.). (2015). Making culture count: The politics of cultural measurement. London: Palgrave.
Meyrick, J., Phiddian, R., & Barnett, T. (2018). What matters?: Talking value in Australian culture. Melbourne: Monash University Publishing.
Professor Brydie-Leigh Bartleet is Director of the Queensland Conservatorium Research Centre, Griffith University, Australia. She is known worldwide for her research in community music and community engagement and has led many projects that explore the social impact of the arts. She has worked in partnership with a wide range of NGOs, arts and community organizations, and colleagues across Australia and internationally to design, drive and deliver innovative and highly complex projects. This work has led to new and interdisciplinary approaches to music research that intersect with health and wellbeing, corrections and criminology, Indigenous and cultural policy, social justice and regional arts development, and most recently human rights. She has worked on five nationally competitive grants, five consultancies and three prestigious fellowships (totalling over $1.2 million), as well as 140 research outputs in high-level national and international publications, and keynotes in Aotearoa New Zealand, Australia, Canada, Japan, Germany and Ireland. In 2014 she was awarded the Australian University Teacher of the Year and in 2018 she was awarded a highly competitive Art for Good Fellowship from the Singapore Foundation.