Business as usual

By Professor Ross Woodrow

More than ten years of ERA (Excellence of Research in Australia) data gives a clear picture of the trajectory of creative arts research in academe.  For the period from 2006 to 2013, the total number of research outputs in field of research (FoR) 19 Creative Arts and Writing increased by 36%. This makes the 14% increase for the period 2011 to 2016 look far less like a cause for optimism, especially considering the national average discipline growth of 17%. The full-time equivalent number of staff producing research outputs in FoR19 compared to the previous ERA has shown a marginal increase of 4%. This ends the good news, such as it is. 

It suggests that, at best, university-based gatekeepers for ARC grant applications, have not facilitated and encouraged grant applications from researchers producing NTROs in the FoR 19 or, at worst, they have discouraged or blocked them.

The only two areas that matter in relation to ERA measures are quality scores (1 to 5) and grant income. The quality of the research in Creative Arts and Writing, as measured by the gold medal standard of “5” rating, is at the bottom end of the 22 broad disciplines in academe.

As with past ERA exercises the field of Creative Arts and Writing falls last of the twenty-two disciplines in the percentage of Category One grants awarded. However, the percentage of the Australian Competitive Grants research income, that totalled $5.08 billion for the ERA2018 reference period, has now dropped so close to 0% that it isn’t recorded on the published graph.

Does this prove, what some Associate Research Deans (ARDs) in the HASS sector and many PVCs Research have suspected all along, that NTRO outputs in the creative fields are rubbish research and researchers producing them have no chance of getting an ARC grant?

Far from proving any such thing, it simply demonstrates the effect of this deeply-embedded prejudice against any form of creative arts research that does not conform to the humanities and social science research models and methodologies. It suggests that, at best, university-based gatekeepers for ARC grant applications, have not facilitated and encouraged grant applications from researchers producing NTROs in the FoR 19 or, at worst, they have discouraged or blocked them.

The ARC statistical data claims that FoR 19 Creative Arts and Writing have a similar success rate to other humanities and social science disciplines. Creative Arts and Writing ranks in the mid-range of the 22 discipline clusters for ERA volume of outputs submitted and number of staff. It is therefore an absurd and misleading suggestion that it is not a cause for concern that FoR 19 is awarded almost zero percent of ARC grant income because a negligible number of applications are submitted.

What are ARDs and PVCRs doing about ramping up the number of ARC grant applications in Creative Arts and Writing? Very little, especially now, as the ERA “evidence” shows that researchers producing NTROs don’t get category one grants, and this, they believe, is because the quality of their research is stagnant and that’s a disciplinary problem. Giving extra funds to support the production of NTROs by researchers in the creative disciplines gets no institutional return. They are best left to do “business as usual” as one arts and humanities research dean recently wrote in an official university document recommending withdrawing research centre funding for the creative arts.   

...the diagnosis and prognosis for the problems obstructing the development of research culture in the creative arts fields are commingled around the lack of gold medals, leading to the misguided pre-emptive assumption that the symptomatic cause must lie with the peer-reviewers in FoR 19.

Concern over the difficulty in judging NTROs as research, by DASSH Associate Deans Research, came to a head at their 2017 annual meeting when they agreed to commission a pilot study based on a survey of 30 key NTRO decision makers in the 19 field “involved in scoring ERA submissions, assessing ARC applications and other forms of research assessment at the national level”. The reason for a focus on this field is noted in the background to the survey:

“While every other FoR has increased its average score in each of the ERA rounds - as universities better understand what is expected of them, and as the ERA exercise drives changes in practice - FoR19 is the only code in which the average score across the sector has decreased in each ERA round. The ADR network was concerned that this speaks to a continuing uncertainty as to how these judgments are being made in practice.” (1)

For anyone looking at the ERA data, a very selective form of tunnel vision is required to avoid noticing that one of the largest fields in this cluster gets close to zero percentage of ARC grants awarded. Here the diagnosis and prognosis for the problems obstructing the development of research culture in the creative arts fields are commingled around the lack of gold medals, leading to the misguided pre-emptive assumption that the symptomatic cause must lie with the peer-reviewers in FoR 19.

The thirty experts surveyed in August 2018 offered little consensus in rating the importance of nineteen possible criteria they might use in making quality judgements about NTROs. Most often respondents took the op-out option of “it depends” which took comments to a second, qualitative stage of analysis. Nevertheless, it was heartening to see that the majority expert opinion does not support “The presence of a strong academic research question in the research statement” as being very important. The majority also believed that it is relevant “Whether the NTRO has been critically reviewed” but not very important to judgement of its quality. In other words, a majority of expert peer-reviewers of non-discursive research outputs are resisting the pressure to move closer to existing humanities research models and methodologies. 

It is now evident that the ERA, like the HERDC, is a quantitative system but with a far greater focus on grant income. The quantity and quality of publication outputs of any sort will not alone achieve the gold medal of a “5”.

The strong reliance on the starting point of “the research question” in social science and humanities is mostly antithetical to the outcomes of creative practices which hardly ever yield “propositional claims about states of affairs.” (2) Such approaches, by default, deny the primacy of the non-discursive forms that creative arts research takes which are often shot through with complexity, sometimes including contradictory or unresolved dilemmas, yet nonetheless in operation enhances an understanding of the underlying issue, human condition or state of affairs. It is evident from even a cursory scan of the research statements that support NTROs in the ERA that artists and other creative arts researchers rarely even mention the term “research question” and statistically are five times more likely to use the terms “metaphor” or metaphoric” when giving the background to the development of a particular output. (3)

The thirty experts found consensus on only one of the nineteen criteria in making quality judgements as being very important: “Demonstrated familiarity in the research statement with the current state of knowledge in the relevant academic disciplines.” Peer-reviewers in Creative Arts and Writing are meticulously doing their job by taking each work they review on its individual merits, measured against the research statement that demonstrates the researcher is aware of the operational, historical, contemporary and conceptual context in which their work operates.  I fear that the ARDs were attempting to refract NTROs through the lens of their own research methodology with this survey and they missed the obvious; that they might be part of the problem. For inconsistent judgement by expert peer-reviewers of NTROs is not the reason for the inertia in gaining higher ERA scores in FoR19.

It is now evident that the ERA, like the HERDC, is a quantitative system but with a far greater focus on grant income. The quantity and quality of publication outputs of any sort will not alone achieve the gold medal of a “5” or silver medal “4” ranking in the ERA.  The number of “5” ratings given at the sub-discipline level within the 22 broad research fields is correlated categorically with grant income.  For example, in:

  • FoR11 Medical and Health Sciences, where 36% of the Category One grants were awarded, 170 disciplines across the 36 institutions measured were ranked at “5”

  • FoR09 Engineering, with 9% Cat One grants, 76 disciplines ranked at 5

  • FoR17 Psychology and Cognitive Sciences  3% grant income with 20 disciplinary 5 rankings.

In FoR19 Creative Arts and Writing, only one 5 rating was given in ERA 2018 and this was achieved by an institution with the two individual researchers who rank first and second in terms of number and value of ARC grants awarded in this 19 FoR over the past ten years. As already noted, this one “5” ranking placed Creative Arts and Writing second to last place above FoR12 Built Environment and Design on the list of two-digit discipline codes for the number of “5” ratings. Four of the sub-disciplines in FoR19 were ranked at “5” and three of these were from the institution that ranked at “5” overall. 

Creative arts research is very different to conventional forms of research and if the quality of the insights and understandings achieved can be captured through processes of peer review offered by the ERA there is no further justification or argument needed for awarding ARC category one grants for non-discursive research outcomes such as a novel, exhibition, film or any other creative arts work that has no identifiable instrumental value. The lack of progress in such acceptance over the past ten years suggests this will not happen with the current ERA system and ARC grant distribution protocols.

The situation is currently no different to the twenty years prior to the introduction of the ERA in 2010 when researchers producing NTROs were specifically excluded by the ARC from grant applications – back to business as usual.

Notes

1. Alan McKee “Analysis of the criteria used by key decision makers in Australia to judge the academic quality of NTROs” an unpublished report commissioned by DASSH, 2018 2

2. Tom Barone and Eliot Eisner Arts Based Research, Sage, 2012

3. In 2010 I conducted a small concordance analysis of research statements for outputs submitted in ERA 2010. https://www.researchgate.net/publication/332249472_Studio_Research_'Shrunk_to_this_Little_Measure'

The indications then, as with a follow up larger concordance study of research statements submitted for ERA2015, suggested that the exegetical framing of creative-based research is specifically distinctive and does not correlate strongly with traditional science writing or discursive research practices in the humanities and social sciences.


Professor Ross Woodrow was the founding Director of the Griffith Centre for Creative Arts Research. His involvement with creative arts research and research education spans more than twenty years. He was one of the lucky few to receive an ARC small project grant for an exhibition, Bush Burial in 1996 during the brief period when the “J” category allowed artists in academe to submit NTROs for ARC grants.