A Book Without Peer

In the third installment of Quadrant Online’s series on the curious institution that is Melbourne University’s Sustainable Society Institute, Tony Thomas takes up one of its publications and finds not enlightenment but bafflement as to the definition of “peer review”, not to mention federal funding guidelines

empty bookLast October, Melbourne University’s Sustainable Society Institute (MSSI), with the usual fanfare, launched a free, 140-page, peer-reviewed e-book called “Melbourne: What Next – A discussion on creating a better future for Melbourne”. Editors were Professor of  Urban Planning Carolyn Whitzman (lead author), Professor Brendan Gleeson (Director of the MSSI) and Alexander Sheko (research assistant).

The e-book is a nice riff on ideas to make Melbourne nicer, but it seems a bit lightweight for an academic production.

I ran a check and it was still being promoted by MSSI as “peer reviewed” on December 8.

In fact, the book is mainly transcribed talks delivered at Fed Square public meetings, plus “ideas shared by the public on the Future Melbourne Network [blogsite] and at the seminars.” Hence the peer-reviewed book includes Lord Mayor Robert Doyle’s familiar shtick on council initiatives, and something about Post-It Notes in an essay by The Age editor-in-chief Andrew Holden.

Part I: Melbourne University’s Watermelon Patch
Part 2: The Joy of Yurts and Jam-Jar Glassware

What really took my interest was an essay by dark-green junior activist Kirsty Albion, a national co-director of the Youth Climate Coalition. I’d previously heard her speak at a Moonee Valley City Council pre-election function.  Her topic then was how to undermine the Australian coal industry.

In the MSSI peer-reviewed book, her essay was “People Power for a Renewable Energy Future”. In it, she described how she and her team went to Lend Lease’s annual meeting dressed as fish to dissuade the company from financing coal loading at Abbot Point, near Bowen.[i]

I imagine the Age editor and Kirsty were chuffed at the recognition by Melbourne University of their scholarly ways.

But my b/s detector went off and last month I sent the following query to Melbourne University Press Office on December 8:

I’m a bit puzzled. “Melbourne: What Next?” is described as a “peer reviewed” book but seems to bear no resemblance to any peer reviewed work I have seen. It is a collection  of talks put into book form. 

My questions are:

1. Is this book a peer-reviewed work, as described?

2. If not, will Melbourne University issue appropriate corrections?

3. If yes, can you please provide details of the peer review process, e.g. number of peer reviewers, their fields of expertise… how   speeches by lay persons, journalists, activists, bureaucrats etc can be peer reviewed given that they make no claims to participating in the scientific method; who took responsibility for assessing the peer reviewers’ comments and publishing the book.

 With commendable speed, the press office obtained a response by Professor Whitzman next day:

“The following five research chapters were peer reviewed, using the criteria developed by the Commonwealth Department of Education…(HERDC)  specifically: “an acceptable peer review process is one that involves impartial and independent assessment or review of the [individual chapters in this case] before publication, conducted by independent, qualified experts.” (p. 13)

(1) A Vision for Metropolitan Melbourne (Carolyn Whitzman and Chris Ryan)

(2) Jobs and Housing: making ends meet (Kate Shaw and Bryn Davies)

(3) Cool Melbourne: towards a sustainable and resilient zero carbon city in a hotter world (John Wiseman, David Karoly and Alexander Sheko)

(4) Building Confidence in the Future of Melbourne’s Public Transport Systems (John Stone and Jan Schuerer)

(5) Implementation: Getting Our Act Together (Crystal Legacy, Brendan Gleeson and Jago Dodson).”

The rest of the book was not peer reviewed, she said:  “It is not uncommon for a book or an academic journal to include both peer-reviewed and non-peer-reviewed contributions. I hope this answers Mr. Thomas’ questions.”

No, it obviously didn’t. The university still seemed to be defending its marketing of the entire book as “peer reviewed”. Dr Whitzman made no mention of retracting the claim, as I had proposed.

In the commercial sector, the ACCC reacts fiercely to misleading advertising – hence those groveling apologies by supermarkets over tiny mis-statements about pricing  and labeling. They risk fines of up to $1.1m. Even newspapers volunteer to correct their errors. So  I emailed back on the same day:

1.     Would you concede that MSSI describing the book as “peer reviewed” is not accurate, given that only 5 of about 25 articles in it were peer reviewed?

2.     I can find no reference to “peer review” anywhere in “Melbourne: What Next?”  Would it be useful to amend the e-book to distinguish the peer-reviewed content from non-peer-reviewed content, so that readers are better able to evaluate the contents?

Three days after my initial inquiry, the university replied:

“The authors have taken on your suggestion and have made an edit to the promotional wording about the e-book.”

However, there is no correction (they did what’s called a ‘stealth edit’), and the new wording strikes me as ambiguous verging on the misleading:  “An e-book, featuring peer-reviewed chapters written by leading academics…”

In fact, none of the five “chapters” are peer reviewed. They all contain a big majority of non-peer-reviewed essays. Only the first essay in each chapter is peer reviewed. And no essay in the entire book is labelled as peer reviewed or otherwise.

According to the university’s own guidelines, non-commercially-published research books require clear statements about which of a book’s elements involved peer review. The MSSI itself doesn’t seem to be a commercial publisher, given that of its five books to date, two are free and three are actually published by Routledge.

Moving on to a related issue, I was perplexed by the non-academic tone of the peer-reviewed research essays. They seemed a cross between advocacy and description. Take the conclusion in the final peer-reviewed essay, by authors Legacy, Gleeson (MSSI director) and Dodson:

“The MPA [Metropolitan Planning Authority] has the potential, if it decides to embrace an open, deliberative model, to facilitate a similar kind of ‘new’ thinking about Melbourne’s future. We commit to working with it, in both critical and constructive conversation, to realising better planning for Melbourne.”

How can such a ‘commitment’ be audited via peer review? As the book’s subtitle suggests, this essay seems more like “A Discussion”.

In the peer reviewed essay in the Transport Chapter, two authors conclude,

“The next big public transport investment will need to be more than just a piece of headline-grabbing infrastructure. It will need to be a total package…  After generations of decline and failed promises, successful pursuit of these new directions will require … professionals and politicians willing to work with the community to re-build public confidence in public transport.”

In style and content, this is most unlike any peer-reviewed literature with which I am familiar. Again, it seems like “a discussion”.

Now try Chapter 3, “Climate + Design” and its opening peer-reviewed chapter by Wiseman, Karoly and Sheko. The last  of its opening “Key Points” reads:

“A variety of policies relevant to climate change exist at the local and state government levels in Victoria. While state government policies such as Plan Melbourne appear unlikely to ensure effective action on climate change, more promising directions are taken by local governments such as the City of Melbourne. This chapter provides some recommendations for how Victoria can transition towards a resilient, zero emissions city.”

And at their conclusion, we find that

“…This chapter concludes with suggestions on how governments at the state and local levels can take action to plan for a city capable of delivering both emergency speed emissions reductions, and resilient, adaptive social and technological systems.”

Again, can peer reviewers audit political recommendations and ‘suggestions’?  In the university’s own definition of  “research activity”, the essential characteristic is “that it leads to publicly verifiable outcomes which are open to peer appraisal.”

I referred the peer reviewed material in the book to an academic at a different large Melbourne university  (who has no political/climate axe to grind), and asked her to peer review it. She responded that the peer-reviewed pieces were descriptive and lacked rigor, sometimes using two-sentence paragraphs which failed to display critical thinking. Some of the pieces would in her view fall below “C”-journal standards and more resembled graduate-student essays, she said. She found it very unusual that peer reviewed and lay material was mixed together in the book (remember, lead author Whitzman describes such mixtures as ‘not uncommon’).

I also referred the MSSI’s output of “Voluntary Simplicity” (i.e. back -to -Nature) tracts to another arm’s length academic, who responded,

“Voluntary simplicity is getting some traction in the literature, I have referred to it myself in an old publication. I would agree with you that the arguments used are weak and exaggerated and unlikely to be evidence based. They do not belong in academic publications but should be presented as opinion pieces or discussion papers.”

Academics live or die by their output of peer-reviewed research. Such output is also a significant input into university funding and ranking against rival universities. As Melbourne University puts it:

“The [research output] data provides the university with valuable information on the research activity of its staff and uses the publication data as a key performance indicator in the analysis of research performance across departments and faculties and in benchmarking with other research intensive universities. Research publication data is also used in strategic planning and the allocation of internal and external funding resources.”

The research outcomes “are very important to the university” partly because they influence funding that arrives from the federal Department of Education (HERDC).  The Department of course is alert to the risk of a university gaming the system and sets up quality-control mechanisms.

For Education Department purposes, “research publications” need to be authored research books, chapters in research books, refereed journal articles or refereed conference publications. I can’t work out which category is the right one for MSSI’s free e-book including transcripts of    public meetings and ideas from blog posts.

The guidelines also say,

“For the purposes of the HERDC, an acceptable peer review process is one that involves impartial and independent assessment or review of the research publication in its entirety before publication, conducted by independent, qualified experts. Independent in this context means independent of the author.”

MSSI and the university then declined to answer a Quadrant question about whether the parts of the book that had in fact been peer reviewed, had been peer reviewed by any  reviewers staffed at Melbourne University.  (Such reviewers would have a common commercial interest with the authors in boosting the university’s research tally). Perhaps the university was correctly protecting its peer reviewers’ anonymity.

A few days earlier, MSSI put another erroneous claim on its website, that Al Gore is a “Nobel Prize winner”. Gore won only half of the 2007  Nobel Peace Prize, previously shared by the likes of Yasser Arafat and the European Union.

Summing up these three Quadrant essays (see the featured links above), MSSI is a possible case study in what is described as the Left’s “long march through the institutions”.

The university has its own Fairness and Diversity Unit which polices the policy of “maintaining and strengthening a vibrant and inclusive institutional environment in which cultural diversity is recognised, valued and celebrated.”

Perhaps the unit could look into the under-representation at MSSI of the minority group comprising conservatives.

Tony Thomas blogs at tthomas061.wordpress.com


[i] Terrified by the anthropomorphic fish and other activist ploys, Lend Lease caved in last February.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s