Leagan Gaeilge   Text Version   Display: Zoom (-) (+)   View ( Normal High contrast ) Printer Friendly  A to Z  Sitemap

University Rankings Explained - Why Ireland cannot win at the rankings game.

In recent years various university ranking tables have emerged. Possibly the best known is the THES or Times Higher Education University Ranking.  The THES World Ranking – and others - engage policy makers and, some claim, influence employers and potential students. However, many people who understand universities doubt the validity and value of these rankings.

One might question the volatility of the results; does it really make sense that a university could go from 484^th to 368^th, to 243^rd and then to 299^th and back to 367^th over a period of five successive years? The ’outputs’ of universities are graduates and postgraduates; the impact and outcome of research is measured over many years; it is therefore difficult to explain this short term volatility.  Universities are not premier league football clubs, who invest in star players to deliver a ’quick bounce’.

So are the World Rankings a valid measure of university performance?  To answer, we need to understand what it is they measure and consider if the measures are appropriate to universities in general and to the Irish universities system in particular. 

The THES seeks to measure research and teaching performance. Research is the dominant element, consuming 73% of the score, while teaching accounts for the remaining 27%

The biggest single measure of research, accounting for 30% of the full ranking is what is termed ’citation impact’, which measures the impact of a research paper by the number of times that paper is cited by other researchers.  The measure is restricted to a sub set of all journals and is a very poor indicator or performance in some disciplines, in particular the humanities and social sciences.  The assumption is that a significant paper is one which is cited by peers.  There are those who would argue with this; but there are few who would disagree that it works best in the scientific, medical and some technology disciplines. 

The second major element in the research assessment, accounting for 18% of the total ranking, is a 'reputational' survey.  Here research is assessed  as part of an ’academic reputation survey’, involving over 17,500 randomly selected experienced scholars,  who offer views on the perceived prestige of institutions in both teaching and research.  A further 15% of the total score is assigned to the perceptions of those surveyed on the teaching quality of universities.  It is difficult to understand how a randomly chosen scholar, in say Japan, could comment sensibly on the teaching quality of an Irish university.

In short 33% of the total ranking is based on a reputation survey. Some might argue that this is effectively a ’beauty contest’ where branding and heritage count more than performance.   The subjectivity of this element of the ranking is clearly open to criticism. It is also a criterion on which all Irish institutions, even Ireland’s oldest university, perform poorly.

Overall, reputation (at 33%) and citations impact (at 30%) account for the majority of any institution’s score and therefore ranking. The remaining indicators, all of which weighted at 6% or less and the majority at 2.5% are listed in the table below. They include evidence based metrics such as the number of PhDs awarded per academic, research income per academic, papers per academic, undergraduates admitted per academic, the ratio of bachelor degrees to PhDs awarded, ratios of international to national staff, ratio of international to national students, and research income from industry per academic, weighted at 2.5%. 

Are these measures appropriate?  Is the weighting attached to each reasonable?  Given that rankings drive behaviour, what type of rational behaviour should an Irish university engage in if it were to chase the rankings? 

Clearly research, measured in terms of its academic impact is the main driver of performance. This suggests a strategy of encouraging colleagues to focus on research publications (the ’publish or perish’ syndrome ) in recognised academic journals which enhances citation impact and academic reputation; ignore translational research; and disengage from industrially oriented or applied research which tends to be time consuming and certainly not citation friendly.

Such a university would probably down play investment in teaching. Employing more academics increases the staff student ratio (regarded as a measure of teaching performance), but if those academics are encouraged to engage in academic research and the production of PhDs, it is not clear how the teaching of undergraduates improves. And what about staff who are committed to a better balance of teaching and research; staff who are engaging with the new pedagogy, the new computer-based learning tools which are doing so much to enhance the student learning experience?  How are they to respond?

Should such a university allocate scarce resources to support access for students from socio-economically disadvantaged backgrounds? What about special programmes for 2^nd chance or mature students?  Programmes which meet short term industrial needs?  Staff employed to support such initiatives or existing staff who allocate their time to these activities, at the expense of research with potential for publication, are ’off message’ from the perspective of international ranking systems.

An Irish university seeking to achieve a high placing on the THES ranking might be advised to ignore national priorities in areas like access to third level and the translation of research into economic and social developments. Any incremental investments or efforts to serve undergraduate teaching, to develop student feedback systems for example, are unlikely to bring returns. What about the employability of graduates?  The THES ranking does not reward any efforts in this area.  

Is this the type of university our country needs? Is this what industry needs to help us innovate our way to economic stability? Is this what employers want from our universities? Is this what students themselves want?

Or should our universities focus on a more balanced agenda? Our economy is in transition; we need to move from a situation where we manufactured and exported products designed by others to where we export innovative products, processes and services designed, marketed and manufactured here in Ireland. We need graduates with the skills to effect this transition. We have a duty to translate our research into outputs that benefit Irish industry.

That is not to say that academic research is unimportant or that our ability to compete worldwide on research metrics is not important. It most certainly is. Universities compete in a global market for staff and students. There is no place for ’national’ or ’All Ireland’ champions! The only game is excellence on an international scale.   

But we must ensure balance. It is tempting to draw analogies with the credit ratings agencies and their assessments of bank and sovereign debt in the mid 2000s.  The THES rankings reflect a particular view of university performance which if followed to its logical conclusion lead to dysfunctional behaviour. 

So what’s the alternative? Ireland’s universities cannot opt out of the World Rankings. But neither do we have any hope of competing on a level playing field with the reputations of Harvard, Oxford or Cambridge.

We need to find other ways of establishing our international credentials. One option is to pursue a strategy of developing a strong network of collaborating universities each of which achieves excellence in particular and complementary areas, appropriate to our short, medium and long term needs as a society and an economy.

Public policy might encourage the creation of a network of universities, each of which develops excellence in selected and complementary areas that are appropriate to the needs of Irish society.  In such an ecosystem, each institution or group of institutions develops excellence in selected and complementary areas. And the level of that excellence must be on a global scale. Anything less is a disservice to Irish society and to industry which competes in a global market.

Irish universities can – and are – establishing international reputations in select and appropriate fields of expertise. We can’t be good at everything and we haven’t the resources or the reputations to reach the lofty heights of what are frankly, inappropriate league tables. We can and will achieve world-class status in selected areas if we invest and manage strategically.

Dr Jim Browne

President, NUI Galway

October 2011

 

 

• Disclaimer
• Privacy
• Copyright
• Accessibility