Annotated documentation on league tables and rankings

Ben Wilbrink

As an appetizer, be sure to read the critical article by a researcher knowing almost everything that is methodologically relevant in the rankings games played everywhere, nowadays:

Anthony F. J. van Raan (2005). Challenges in Ranking of Universities. http://www.cwts.nl/cwts/AvR-ShanghaiConf.pdf" [dead link? 1-2009]

This page, of course, is related to the ones on indicators and competion between educational institutions [those pages are - should be - in English as far as English literature is mentioned]. The general problem, of which ranking of schools and universities is just a specific example, is that of assessment of achievement, the subject of most of the professional part of my website.

My mission statement regarding this webpage? Behold the folly of management high and low. On a more serious note: ranking is a medieval habit that seemingly was re-introduced in Western society in the field of sports. It might even be possible that a certain competitive spirit among the poorer residents of Cambridge University in the later eighteenth century spilled over in English sports, that however is purely speculation on my part. Sportive competition has become a metaphor abused by scores of politicians and other authoritarians outside sports proper. One such abuse is to hold educational institutions accountable on their local or global ranking between comparable? institutions.

medieval

In the later Middle Ages students traveled long distances to attend a university of their parents' or clerical order's choice, in Western Europe: Bologna, Paris, Oxford, Cologne, Bourg, Leuven, and others in Italy, Spain, Portugal, France, Britain, the Low Countries, Germany. I am not aware of any publication that addresses the issue directly. It is evidently the case, however, that Bologna, Salamanca, or Paris were very great names, attracting students from Britain and Ireland also. Scores of priviliges determined the attractiveness of the particular university, as well as the city of location, or its tradition in particular academic fields or in serving the children of the elite, as for example Bologna, on both accounts mentioned last.

1854

1990

Richard Farnum: Prestige in the IVY League: Democratization and discrimination at Penn and Columbia, 1890-1970. In Paul William Kingston and Lionel S. Lewis (Eds) (1990). The high-status track: studies of elite schools and stratification. Albany: State University of New York Press.

1998

Paul Black (1998). Learning, League Tables and National Assessment: opportunity lost or hope deferred? Oxford Review of Education, 24

• abstract In 1988, the UK government accepted most of a set of radical proposals which their Task Group on Assessment and Testing had set out as the basis for a new scheme of national assessment. This paper outlines the formulation, the reception and the eventual abandonment of the scheme, illustrating how this outcome emerged from the interplay of political groups who approached the problems with fundamentally different beliefs about assessment and Learning. It then reviews some of the basic problems involved, calling in particular on evidence produced over the last ten years about the importance and effectiveness of formative assessment in classrooms. It is argued that such assessment requires, and could justify, a new and substantial investment in development and training. It is further argued that current national testing in England and Wales falls far short of acceptable requirements of reliability and validity, and that the place of teachers' assessments in such testing has to be enhanced to meet the shortcomings. Overall, however, manv public prejudices and misunderstandings about testing will have to be overcome if such programmes are to command support.
• from the introduction This paper looks in two directions. In the first part it surveys the story of the 1987/88 Task Group on Assessment and Testing (TGAT) and the subsequent development of national assessment policy. The reflection is selective, aiming to highlight issues and lessons which are fundamental to any consideration of the future. The second part attempts to look ahead, by re-examining some of the issues in the light of evidence and experience which has developed since 1987 and which should enrich our understanding and feed new hopes.

Fisher Dilke (September 8, 1998). Education league tables 'worthless.' BBC. html

• League tables of school performance are popular with government, parents, the media. But do they really have any value? Statisticians think not. Fisher Dilke reports for the Living By Numbers series on BBC Radio 4.

2004

The Times Higher Education Supplement World University Rankings 2004

2005

Anthony F. J. van Raan (2005). Challenges in Ranking of Universities. Invited paper for the First International Conference on World Class Universities, Shanghai Jaio Tong University, Shanghai, June 16-18, 2005. http://www.cwts.nl/cwts/AvR-ShanghaiConf.pdf [dead link? 1-2009]

• abstract In this paper we discuss recent developments in rankings of universities and the impact of these rankings on academia in the context of international benchmarking and evaluation. We focus on technical and methodological problems behind these rankings, particularly those based on bibliometric methods, with special attention to the social sciences and humanities. We criticize the recent expert-based rankings by showing that the correlation of expert score and bibliometric outcomes is practically zero. This finding casts severe doubts on the reliability of these expert-based rankings. New approaches are proposed on the basis of advanced bibliometric methods. It is argued on the basis of preliminary results that, probably due to 'finite size' considerations, a league of outstanding universities worldwide will not have much more than around 200 members. Finally, we discuss the challenges for further research and practical applications.
• If you happen to have wet dreams on the subject of your institution's world ranking, you should read this article at least twice. Subsequently, be sure to bar this methodological nightmare from your brain cells. Van Raan and his co-workers might do the job for you. My own conviction, strengthened by the work of Brewer et aliis (2003) (see below), is that world rankings - just like Carthago in Cicero's opinion - should be demolished while such is still possible to do.

The Times Higher Education Supplement (October 28, 2005). Extra supplement: The Times Higher World University Rankings 2005 - the best guide to the world's top universities. pdf

• "This second edition, compiled by QS, has been refined to provide greater accuracy and even more information for 2005. So, which are the world's best universities? Readers from around the globe can now view our results and rekindle the debate... "
  "Familiar names fill the top ranks, but our tables, which have been refined to provide even more detail this year, show that those at the summit cannot afford to rest on their laurels, John O'Leary writes."
  The report is not in an online version for sale. Subscribers can access the report, and so can visitors using the free 14-day trial [the next year you'll have to buy the paper version, or subscribe].

http://www.weltklasse-uni.at/

Robert Tijssen Hugo Hollanders Thed van Leeuwen and Anton Nederhof (2005). Science and technology indicators 2005. Summary. Netherlands Observatory of Science and Technology. site

2006

The Academic Ranking of World Universities (ARWU) (2006). The Institute of Higher Education, Shanghai Jiao Tong University (IHE-SJTU). html

• The list in rank order pdf
• Methodology. Mix: quality of education 10%, quality of faculty (awards, citations) 40%, research output 40%, size of institution 10%.
• Nobel and field prizes are the main criterion, for example quality of education is defined here as "The total number of the alumni of an institution winning Nobel Prizes and Fields Medals. Alumni are defined as those who obtain bachelor, Master's or doctoral degrees from the institution. Different weights are set according to the periods of obtaining degrees. The weight is 100% for alumni obtaining degrees in 1991-2000, 90% for alumni obtaining degrees in 1981-1990, 80% for alumni obtaining degrees in 1971-1980, and so on, and finally 10% for alumni obtaining degrees in 1901-1910. If a person obtains more than one degrees from an institution, the institution is considered once only."
• The data come from http://nobelprize.org/ (Nobel prizes), http://www.mathunion.org/medals/ [dead link? 1-2009] (Fields medals), http://www.isihighlycited.com/ (highly cited researchers), http://sub3.isiknowledge.com/ (articles published)
• "Size. The weighted scores of the above five indicators divided by the number of full-time equivalent academic staff. If the number of academic staff for institutions of a country cannot be obtained, the weighted scores of the above five indicators is used."

The Times Higher Education Supplement, October 6, 2006 World University Rankings http://www.thes.co.uk. No, the rankings are not available for free, unless you take a free 14-day trial

• The trend here is for universities to compete with each other globally. Beijing University in this hitlist is on place 14. Ah, yes, Harvard is #1, Cambridge #2, Oxford #3. The peer review score weights heavily in the total scores institutions get.
• The THES Editorial: 'Global vision ensures healthy competition.' Do not believe them, or your own gut feeling. Read instead the recent books by Derek Bok (2003) (He is now temporarily Harvard's Big Boss, after having been President before in the '70s and '80s; he is a man with a mission: the quality of Harvard's educational process, and Harvard's admissions of socially disadvantaged students), and especially Dominic J. Brewer, Susan M. Gates and Charles A. Goldman (2002). In pursuit of prestige. Strategy and competition in U.S. higher education. New Brunswick: Transaction Publishers. Read all about how selectivity is a weapon in the hands of Harvard's etc. adminsitrations to secure high places on hit lists (this, incidentally, is one of the best kept secrets in the U.S. Read the volume on admissions in the 21st century, edited by Camara and Kimmel, and see how it is completely silent on this abuse of selectivity (or read my review of the book's chapters here)).
• October 13, 2006: p. 8 World's top 100 in science, p. 9 World's top 100 in technology.
• The rankings are a joint venture between the THES and GS Quacquarelli Symonds; the slim ranking 200 itself is outranked by the book publication containing the full ranking 500: Top Universities Guide. I must confess I have not seen the book. If you are interested, look at www.topuniversities.com, a sick name for a website, isn't it? Then use the Blackwell site. The book has been authored by John O'Leary, Nunzio Quacquarelli and Martin Ince, and is advertised as "A must-have book for anyone seeking a quality university education at home or abroad" (Amazed? The book is a guide to students. Lots of advice, and the - in this context - definitely misleading rankings of 200, not 500, universities).

U.S. News & World Report. America's best colleges 2006. pagina. Geeft de instellingen naar hun rank in de lijst, meteen doorklikbaar naar de website van de betreffende instelling. Voor de gedetalleerde data moet je de online version kopen voor ca 15 dollar. Dit is wel de ongelooflijk invloedrijke ranglijst waarvoor iedereen siddert en bereid is met zijn in te sturen gegevens te sjoemelen. Zie Ehrenberg (2001) http://www.ilr.cornell.edu/cheri/wp/cheri_wp17.pdf [dead link? 1-2009] (hierboven) voor een inside visie op USNWR' ranking.

U.S. News & World Report. America's best graduate schools 2006. pagina. Idem (zie boven). En je kunt meteen door naar de beste e-learning instellingen, ziektekostenverzekeringen, ziekenhuizen, en politici.Wat een wereld. De kiosk op de hoek, of die op Schiphol, heeft wel een exemplaar van de gedrukte versie, als die recent uit is.

Alexandra Blair (September 12, 2006). Britain slips to ninth on university league table. Times Online html

• "The Organisation for Economic Cooperation and Development (OECD) gave warning that the high numbers of young people leaving school without five good GCSEs is a serious block to further progress, and that the situation will only improve if the United Kingdom embarks on a radical overhaul of its education system."

OECD (2006). Education at a Glance. OECD Indicators - 2006 Edition. e-book is available for € 45. Shame on the OECD. Is there someone willing to send me the e-book? site

John O'Leary (2006). The Times Good University Guide. site

• "The Times Good University Guide is the definitive guide for students, parents, business and academia. Read articles on this year's developments, the top universities league table and subject rankings. Plus, the editor of this year's guide, John O'Leary, answers your questions."
• Among others the Top universities league table 2007, a scripted table online available

Martin Birchall (2006). The Times Top 100 Graduate Employers 2006-2007.

Tom Halpin (June 5, 2006). In the scrum for places, is your university a winner? The Times html

The Sunday Times University League Table pdf, also student satisfaction pdf

• The world ranking figure lives on the Sunday Times website

The Guardian University Guide 2006 html. How the tables are computed html

2007

The Leiden Ranking

• http://www.cwts.nl/cwts/Leiden_Ranking_1_C_blue_list.pdf [dead link? 1-2009] "ranking by the 'simple' citations-per-publication indicator (CPP)"
• pdf pink list "CPP-ranking for the top-50"
• http://www.cwts.nl/cwts/Leiden_Ranking_1_A_yellow_list.pdf [dead link? 1-2009] "ranking by size, i.e. number of publications (P)"
• http://www.cwts.nl/cwts/Leiden_Ranking_1_B_green_list.pdf [dead link? 1-2009]"ranking by the size-independent, field-normalized average impact (our 'crown indicator' CPP/FCSm)"
• http://www.cwts.nl/cwts/Leiden_Ranking_1_D_orange_list.pdf [dead link? 1-2009] ranking by the size-dependent 'brute force' impact indicator, the multiplication of P with the university's field-normalized average impact (P * CPP/FCSm)

2010

THE World University Rankings 2010. The Times Higher Education Supplement. html

• “For even richer and deeper analysis of more than 400 institutions, download our Times Higher Education World University Rankings iPhone application, which allows you to create your own methodology, filter results by country and region, and match your personalised rankings against cost-of-living and tuition-fees data.”

Links

The Leiden Ranking

ARWU site presents a list of sites of rankings, ranking guides, etcetera. http://ed.sjtu.edu.cn/rank/2006/ARWU2006Resources.htm

The Centre for Science and Technology Studies CWTS at Leiden University "http://www.cwts.nl/scripts/index.pl [dead link? 1-2009]

• "The Centre for Science and Technology Studies (CWTS) specializes in advanced quantitative analysis of science and technology performance and the cognitive and organizational structure of science and technology. Research in short- and long-term programmes is carried out for governments, European Union (EU), national and international research organizations, universities and companies."

OECD (2006). Education at a Glance: OESO-indicatoren - uitgave 2006. Samenvatting in het Nederlandspdf

OCenW (12 september 2006). Samnevatting van Education at a Glance 2006. pdf

• Mooie tabellen, hoor.

Ellen Hazelkorn (2015 2nd) Rankings and the reshaping of higher education. The battle for world-class excellence. Palgrave Macmillan. [PEDAG 40-g-196] [als eBook in KB] info

Ellen Hazelkorn was member of the Veerman Committee (Commissie-Veerman)

Jung Cheol Shin; Robert K Toutkoushian; Ulrich Teichler (2011). University Rankings. Theoretcal basis, methodology and impacts on global higher education. Springer. [eBook KB]

Björn Hammarfelt, Sarah de Rijcke, Paul Wouters (2017). From Eminent Men to Excellent Universities: University Rankings as Calculative Devices. Minerva DOI 10.1007/s11024-017-9329-x pdf

Mark Deuze (8 maart 2018). Rankings bedreigen alles wat goed en mooi is in de academie Folia blog

Jerry Z. Muller (2018). The Tyranny of Metrics. Princeton University Press. 9780691174952 reviewed in Science .

March, 2018

  www.benwilbrink.nl/literature/rankings.htm