A new methodology for ranking scientific institutions

Transcription

1 A ew metodology for rakg setf sttutos J.. Molar 1 A. Molar 2* 1 Laboratory of Meas ad Teology Eole ormale Supéreure de Caa Pars 6 rae 2 Laboratory of Pyss ad Meas of Materals Uversté Paul Verlae Metz rae * Correspodg autor: molar@lpmm.uv-metz.fr Abstrat: We exted te poeerg work of J.E. Hrs te vetor of te -dex by proposg a smple ad seemgly robust approa for omparg te setf produtvty ad vsblty of sttutos. Our ma fdgs are tat wle te -dex s a sesble rtero for omparg setsts wt a gve feld t does ot dretly exted to rak sttutos of dsparate szes ad jourals owever te -dex w always reases wt paper populato as a uversal growt rate for large umbers of papers; tus te -dex of a large populato of papers a be deomposed to te produt of a mpat dex ad a fator depedg o te populato sze v as a omplemet to te -dex ts ew mpat dex provdes a terestg way to ompare te setf produto of sttutos uverstes laboratores or jourals. 1

2 Itroduto Te evaluato of te produtvty of dvdual setsts as tradtoally reled o te opo of experts. Wt te veue of te Iteret ad aess to large databases ludg te Web of See [1] omparg te mpat of setf otrbutos as gaed automato ad smplty. Reetly a ovel ad smple-to-use dex was proposed by J.E. Hrs [2]. Ts dex referred to as or -dex s teded to replae te tradtoal metod of paper outg w advatages setsts wo produe a large quatty of weakly ted papers as well as te total tato out w may put too mu empass o few gly ted papers. Te -dex s defed as te umber of papers wt tato umber ger or equal to [2]. s tus a dato of bot quatty ad qualty. Te -dex a oly rease a setst s areer ad terefore oe sould ompare setsts of a gve age group or ompare te slope of we plotted as a futo of tme. Ts ovel dator as gaed fast ad broad reogto [3456] ad t s atural to try to exted ts use for rakg sttutos be tey uverstes sttutes laboratores setf jourals et as was suggested te orgal paper [2]. However te -dex s a futo of te umber of papers osdered ad tus large resear sttutos produg a great umber of papers may ave a larger -dex ta smaller sttutos. Tus smply osderg may ot be a far way to evaluate te trs resear performae of sttutos. I order to geeralze te use of te -dex t s mportat to uderstad te effet of te populato sze. I addto wle te -dex s a useful dator for omparg 2

3 dvduals produtvty a gve feld t sould be oted tat strog varatos exst betwee dsples ad eve betwee resear tops wt a gve feld. Ts was oted reetly by MG Baks [7] wo exteded te terpretato of te -dex to elp studets judge te popularty of a gve resear top. Clearly as stated above tere s a dager of msterpretg te -dex as a g does ot reflet oly qualty but also populato sze. Top setsts Bology are more ted ta top setsts te Pyss ommuty wo are temselves more ted ta top setsts te Egeerg ommuty [8]. It s lear tat tese dfferees do ot oly reflet te qualty of te work of dvduals questos but also reflet prates of a gve feld or te populato sze of a setf ommuty. Wt tese wargs md we odut feld expermets usg te ISI Web of Kowledge database [1]. Ts database mposed a upper lmt o te sample populato of 10 5 papers. Results g. 1 sows te -dex terms of umber of papers for tree reowed jourals See geeral terest Ata Materala Materals See joural w was prevously kow as Ata Metallurga ad te Joural of te Meas ad Pyss of Solds Meas. Ea pot ts log-log dagram orrespods to a gve outry or a group of outres. or stae te USA as publsed a total umber of papers See ad te -dex at te date of ovember of ts set of paper s

4 g. 1: master urves of setf jourals: See dots Ata Materala rosses te Joural of te Meas ad Pyss of Solds astersks. Tese master urves sow te evoluto of te -dex terms of umber of papers. Ea pot ts log-log dagram orrespods to a gve outry. Te t ad bold stragt les are a ft to te data ad dstgus two dfferet growt regmes. Te square symbol represetg te terseto of te lowest bold le wt te vertal axs araterzes m for Ata Materala or for te Joural of te Meas ad Pyss of Solds. If for a gve joural say See oe osders varous outres t s remarkable to observe tat all pots form a le. Ts le w wll be sow to be smlar sape for all jourals ad for all set of papers publsed by resear sttutos s alled 4

5 eefort te master urve m-urve of te joural or sttuto osdered. Ideed t a be sow tat to a gve set of papers S oe a assoate a uque way a teoretal referee m-urve defed by statstal treatmet of te -dex of subsets radomly sorted out of S matematal developmets are gve Appedx ad [9]. Te m-urve of te Joural of te Meas ad Pyss of Solds s sow g.2. Here S s te set of te 2638 papers publsed ts joural utl ovember It appears tat te astersks g.1 referrg to papers publsed by partular outres are dstrbuted alog te referee m-urve of ts joural. Zoe 2 Zoe 1 g. 2: teoretal master urve m-urve of te Joural of te Meas ad Pyss of Solds. Te teoretal m-urve s made of te assembly of dsrete pots see Appedx but s represeted ere for larty as a otuous bold le. Te dased les represet two dfferet growt rates zoes 1 ad 2. Astersks are dvdual outres setf produto for ts spef joural ad are dstrbuted aroud te m-urve. 5

6 Ea m-urve a be deomposed two zoes te log-log dagram of gs 1 ad 2. I g.1 te t stragt le zoe 1 for small umber of papers ad bold stragt le zoe 2 for large umber of papers reflet two dfferet growt rates. Zoes 1 ad 2 are also sow g.2. or a large umber of papers te growt rate of te -dex s govered by a power law wt expoet β slope of te bold stragt les g.1: β = m 1 were s te umber of papers. Te value of β s about 0.4. A key observato we ave made s tat ts growt rate apples for all te sttutos ad jourals we ave osdered ludg tose tat are ot sow ts paper ad as terefore a geeral arater at least for te rage tested ere.e. <10 5. ote tat te level of zoe 2 g.1 s araterzed by te terseto log m wt te vertal axs square symbol. All te results obtaed for See Ata Materala ad te Joural of te Meas ad Pyss of Solds merge wt zoe 1 g.1. Wle te last two jourals almost ode wt zoe 2 smlar m See s araterzed by a ger level of zoe 2 ger m. Te value of m a be vewed as a measure of te mpat or vsblty of te papers publsed by a gve joural. A pot above te teoretal referee m-urve of a gve joural s a dato of a better qualty of te setf produto publsed te joural by te outry osdered. Se for ea joural te dsperso fgure 1 s ot too large t s see tat all tree jourals osdered ere proeed to a far seleto of 6

7 papers based o qualty depedetly from te outres from w te papers orgated. g. 3: -dex as futo of populato sze for varous teratoal resear sttutos; ote tat for te large umbers of papers osdered te -dex les wt zoe 2. Tese results exemplfy te uversalty of te power law expoet aroud 0.4. Te vertal sft betwee sttutos s a dato of ter respetve vsblty. Te sample szes were bult by assemblg papers publsed eve years startg 1980 up to 1998 more reet papers were ot luded as tey are geerally ot suffetly ted to mpat te -dex. Te two otuous les wt slope 0.4 are added o te fgure to sow te geeral growt rate but are o meas uversal upper or lower bouds. Ideed most uverstes would fall uder te lower stragt le. 7

8 Aoter llustrato of te depedee of o umber of papers s gve g. 3 w we ompare aadem sttutos. We osder Amera ad Europea sttutos of exellet stadg. Tese uverstes ed up osstetly o top of Amera ad teratoal rakgs [10] ote tat teratoal rakgs are strogly based o resear produtvty. If we sgle out Calte ad Harvard oe would experee dffulty at dfferetatg te qualty of tese sttutos yet Harvard s - dex for ay gve year s larger ta Calte s. Clearly populato sze s ere a strog fator. To uderstad ts effet we ave supermposed te -dex for reasg populato szes for Harvard ad Calte o g.3. Te database was olleted by measurg at te date of Otober te -dex for all papers ssued 1980 te for te ombed years 1980 ad 1982 up to ludg all eve years from 1980 to Ts proedure permtted te assembly of large set of papers. Amazgly te - des as futo of umber of papers for tese two uverstes alg almost perfetly w we dare to terpret as dato of equvalet resear mpat. O te same fgure we ave added te data obtaed troug te same proedure for oter teratoal sttutos. Cambrdge resear output seems of omparable vsblty wereas te mpat dex m emaatg from rae top uversty te Saga Uversty rakg [10] s slgtly lower. It sould be added tat re atoal resear sttuto CRS as a output omparable to Pars 6. As a smple matematal osequee of te power growt law ombg te paper populatos of Pars 6 ad 8

9 CRS would rease te legt of te stragt le g.1 but would ot sft t vertally. rom tese feld observatos major ommets a be draw. rst te master urve assoated to a gve populato of papers a be vewed as a sgature of te vsblty of tese papers. or a large populato te orrespodg -dex a be fatorzed as te produt of a mpat-dex m wt a sze depedet fator 0.4 were s te populato sze. Ts deomposto llustrates te major feature of te -dex of beg a metr aoutg for bot qualty ad quatty. Te power law expoet of te order of 0.4 w govers te ultmate growt rate of te master urves s amazgly osstet betwee dfferet sttutos ad dsples as represeted by dfferet jourals g. 1. Ts expoet must ovey a rate of dffuso of formato ad s tself a terestg top to aalyse. Seod a assessmet of te trs vsblty of a sttuto a be ferred from te vertal sfts of te urves gs. 1 ad 2 w are araterzed by m. A mportat feature of te dex m assoated to a gve set of papers S s tat t a be farly well estmated by usg ay represetatve subset of S otag a suffetly large umber of papers. Ts property may be qute useful se t s sometmes dffult to ollet te totalty of te papers publsed by some sttutos due to a lak of systemat lassfato or labellg. I order to furter establs te robustess of te m dator for assessg resear mpat we aalyse g. 4 ts evoluto wt tme for te tree prevous jourals Ata Materala Joural of te Meas ad Pyss of Solds ad See. Here m s 9

10 alulated for dfferet years wt for example m 1980 orrespodg to te set of papers publsed by a gve joural durg te year m s a dereasg futo of tme for reet years w reflets tat reet papers ave ot ad yet eoug tme to aumulate tatos. However m qukly reaes a plateau for Ata Materala ad for te Joural of te Meas ad Pyss of Solds dato tat te vsblty of tese jourals s stable alog te years osdered. Te evoluto of m wt tme for See as a dfferet tred. Ideed m appears to be reasg over te years w we may terpret as a sg of reforemet of te joural mpat wt tme peraps partly due to te reasg fluee of expadg dsples. Clearly smlar observatos ould be made for uverstes wose resear s emergg ad reversely for sttutos or departmets tat are losg groud tat ase m would ot plateau but would derease steadly over te years. 10

11 g. 4: Evoluto of m wt publato date for See stars Ata Materala squares ad te Joural of te Meas ad Pyss of Solds damods. Te dex m may be put perspetve wt te Hgly-Cted HC rtero tat eters may rakgs. A ew lassfato tat as gaed g vsblty s te Saga s uversty 2006 teratoal rakg [10]. Oe of te rtera eterg [10] s te umber of Hgly-Cted researers a gve sttuto w may advatage large uverstes. Our omputato of m for a group of setf domas see Table 1 ad for seleted uverstes eterg te top 100 postos [10] sows ow t orrelates wt HC. or stae Harvard Uversty remas top posto. However small yet exellet sttutos su as Calte ad Preto appear refored. Tus m appears to orret te sze effet ad ts use s more geeral ta H-C as t a be exteded to ompare setf jourals or sttutos ad laboratores of small szes. As a addtoal test we osder te groupg of two sttutos a soluto sometmes proposed to rease te vsblty ad rakg by reasg te sze. Te ase study we osder ere s te mergg of Berkeley USA ad Cambrdge Eglad two uverstes wt almost detal m. Te uo of tese sttutos yelds for te resear domas osdered Table 1 a total of papers. ote tat ts umber s slgtly lower ta te dret sum of te papers produed by bot sttutos w s a dato of te volume of ollaboratve work. Te -dex of te uo s 152 a umber larger ta te -dex of ea sttuto 115 ad 116. However m s left uaged at A matematal formalsm for mergg rules s proposed a follow up work [9]. 11

12 Ts example learly llustrates te terest of usg a sze-orreted dex su as m to evaluate te trs performae of a resear sttuto or joural. Colusos Te sgature of te vsblty of a gve populato of papers a be araterzed by a sgle urve alled master-urve or m-urve. Ts urve sows te growt of te - dex terms of te umber of papers. or a large populato more ta some udreds te level of te master urve s defed by a sgle dator m araterzg te mpat of te papers osdered. Ts dator m as te mportat feature of beg sze-orreted. Ts was exemplfed by te ase study of te mergg of two uverstes of same m wt te effet of leavg te ombed m uaged. Te -dex of a large populato of papers a be deomposed to te produt of te mpat dex m wt a fator depedg o te populato sze. Due to ts smplty omplemet to exstg dators we advoate usg te mpat dex m for omparg jourals wt a gve feld ad geeral resear sttutos of dfferet szes. It ould be used as well to ompare relatvely small-sze resear groups su as departmets laboratores ad sttutes. Te matematal foudato of te results preseted ts paper s developed a follow up paper [9]. 12

13 Isttuto ame HC : umber -dex m =/ 0.4 sore of papers Table rak Harvard Uversty Staford Uv Uv Calfora Berkeley Massausetts Ist Te Preto Uv Calfora Ist Te Columba Uv Uv Cambrdge Uv Cago Uv Oxford Duke Uv Jos Hopks Uv Mga State Uv Uv Utret Osaka Uversty Pars Case Wester Reserve Uv Uv. Brmgam Uv Mu

14 Uv Roma La Sapeza Table 1: Seleted uverstes from te 2006 Saga s uversty rakg [10] ere raked by ter H-C sore related to te umber of gly ted researers wt te sttuto. Te ose uverstes were te frst te raks [10] plus te addtoal sttutos separated by about te postos up to rak 100. Te umber of papers was obtaed troug a Web Of See sear for te tme perod ad for te followg groupg of dsples: Pyss Matemats Cemstry Materals Egeerg Meas. ote tat all papers may ot ave bee seleted for ea uverstes beause of a lak of systemat lassfato; owever ea obtaed set s larger ta some udreds w yelds a safe determato of m. ote also tat te results of fg.2 were obtaed by osderg all dsples wt te effet of produg a ger dex m as a osequee of te ger level of tatos for papers mede ad bology. Referees ad otes: 1. ISI Web Of Kowledge ttp://portal.skowledge.om 2. J.E. Hrs Pro. at. Aad. See P. Ball ature G. Saad Setometrs W. Glazel Setometrs T. Brau W. Glazel A. Subert Setometrs M.G. Baks Setometrs ttp://e.wkpeda.org/wk/hrs_umber 14

15 9. J.. Molar A. Molar Matematal foudato of a ew rtero for rakg setf sttutos based o te -dex to be submtted at Setometrs Saga s 2006 Iteratoal uversty rakg: ttp://ed.sjtu.edu./rakg.tm 15

16 Appedx: defto of te m-urve I ts appedx te relatosp betwee te -dex ad umber of papers s aalysed. Cosder a set S of papers. Te -dex of S s te umber of papers of S w are ted at least tmes. Te -dex of S s deoted by S. A.1 Ctato law Let us deote by T = ards te total umber of papers S ad by p =12 a permutato of te papers of S; s te umber of tatos of te paper p. T Te permutato p. s ose su tat s dereasg wt. I oter words te papers are sorted desedg umber of tatos ote tat T some of te may ave detal values. Te set of papers S s araterzed te preset aalyss by te futo. : defed o w s alled eefort te tato-law of S. T Te umber d of dstt values of s smaller or equal to 1. Let us sort te dstt values of desedg order G1 G2.. G wt G 1 d 1 = G 1 < G2 <.. < ad G Gj for j. or ea m 1 m d m s defed as G d 16

17 te largest teger su tat = G ; m s also te umber of papers avg m m a umber of tatos larger or equal to G. m A.2 Depedee of te -dex terms of te umber of papers: te master urve Cosder two tegers ad su tat ad. Deote by S a ombato T of te papers of S at a tme. Te total umber of tese ombatos s T. Te otato p! q p = q! p q! s used to represet te umber of ombatos of p elemets q at a tme. ~ Deotg by te umber of ombatos S w ave a -dex equal to S = we ave: T = = 0 ~ A1 Te probablty P r to ave a ombato S su tat S = s: P = r ~ T A2 rom A1 t follows tat = 1. = 0 P r 17

18 Clearly we ave 0 ~ = = P r for 1 > or for >. or 1 tere s a uque teger su tat G = or 1 G G > >. Cosderg tese two possbltes t a be sow tat for ay teger su tat T ~ s gve by: + = = = k j b j k T a k b k j 0 1 ~ for G = A3 wt: f b a = 1 = b A4 ad by = T 1 1 ~ for 1 G G > > A5 ote tat A3 a also be wrtte as: + + = = = k j b j k T a k b k j T 1 1 ~ f G = A6 Wt Equs. A3-A5 oe a ow determe all relevat statstal formato ludg te mea -dex for te ombatos S : = = r P 0 A7 ad te varae ad stadard devato: = = r P 0 2 var var = σ A8 18

19 Te master urve or m-urve of te set of papers S s by defto te urve 1. T 19