This page has some summaries of the raw data that was used/generated for our citation analysis research, which led to the following publications
- Jill Freyne, Lorcan Coyle, Barry Smyth, Pádraig Cunningham (2010). Relative Status of Journal and Conference Publications in Computer Science. Communications of the ACM Vol 53(10). To Appear.
- Lorcan Coyle, Jill Freyne, Barry Smyth, Pádraig Cunningham (2008). A Quantitative Evaluation of the Relative Status of Journal and Conference Publications in Computer Science. Technical report number UCD-CSI-2008-08, University College Dublin, Dublin.
This page was (mostly) automatically generated from our Citation Analysis Scripts (FYI: the original automatically generated page is here). Our study gathered data from a total of 8764 papers (15 conferences with 3258 papers and 17 journals with 5506 articles).
Please get in contact with the authors if you want more information on this work.
Correlation between ISI Impact Factor and Google Scholar Impact Factor for Selected Journals
There is a strong correlation between citation counts computed from Google Scholar data and comparable data from the ISI Web of Knowledge index, validating the use of our new Google Scholar Impact Factor as an alternative citation-based evaluation metric applicable to both journals and conferences. The Pearson correlation coefficient between the two is 0.86. The table below shows the raw values.
|Category||Journal Name||ISI Impact Factor||GS Impact Factor|
|A*||Journal of the ACM||2.9||35|
|B||Pattern Analysis Applications||0.4||8|
|A*||Transactions on Knowledge and Data Engineering||2.1||15|
|A*||IEEE Trans. on Pattern Analysis and Machine Intelligence||4.3||34|
|A||AI in Medicine||1.6||12|
|B||International Journal of Pattern Recognition||0.5||4|
|A||Decision Support Systems||1.2||17.5|
|A*||International Journal of Computer Vision||6||27|
|A||Data and Knowledge Engineering||1.4||8|
Below is a scatter plot of GS impact factor against conference rejection rates for 23 conferences across the 15 conference series we evaluated; note the data points reflect a subset of the full set of conferences, namely those for which we were able to obtain. Reliable rejection rates could not be obtained for all of the conferences for which we had data. More details on the rejection rates we used are below. While our study indicate some correlation between GS impact factor and rejection rates, the related Pearson correlation score of 0.53 was not very convincing and reflects considerable variation when it comes to the relationship between conference rejection rates and a conference paper’s ability to attract citations. There are conferences with similar rejection rates but very different Google Scholar impact factors. There are also conferences with very different rejection rates that still manage to achieve similar GS impact factors. For instance, AAAI achieves a Google Scholar impact factor of 20, with a rejection rate of 65%–75%. Its European counterpart, ECAI the European Conference on Artificial Intelligence, is just as selective but achieves a median GS impact factor of only 7 (see the section on regional bias below).
Another example of a lack of correlation between paper acceptance rate and citation count is the 2002 European Conference on Case-Based Reasoning (ECCBR), which had a rejection rate of approximately 0.33 33% and Google Scholar impact factor of 7, achieving a citation rate better than the European Conference on Machine Learning?? 2000 and ECAI 2002, which had twice its rejection rate (see the Section below on open-world versus closed-world bias).
To further test the strength of the correlation between GS impact factor and rejection rate we examined the available data for three conference series that took place between 2000-2003; UAI, ICML, and ECML were the only conferences that occurred and had published rejection rates available in every year of the study. Table 5 outlines the GS impact factors and rejection rates for each year in the study. What is interesting here is there is no significant correlation between rejection rate and GS impact factor. The Pearson score for UAI is was?? only 0.27, for ICML 0.24, and for ECML 0.42. This suggests that, at least for these conferences, the yearly changes in rejection rates had little bearing on expected citation count.
These results highlight the assumption?? the fact?? that any assumed relationship between conference rejection rates and a conference’s ability to attract citations is at best weak, so other factors play a more critical role when it comes to influencing future citations.
|Conference||Rejection Rate||Google Scholar Impact Factor|
We found evidence of strong regional bias between similar conferences, with international (mainly U.S.-centric) conferences attracting much higher citation counts that their similarly selective though less-well-cited European counterparts. Both the AAAI and ECAI conferences target the same research area and attract submissions from a similar community of researchers in a way that is equally selective. Yet the U.S.-centric AAAI enjoys an expected citation count (computed from the product of the median citation count and the rejection rate of the conference) more than twice that of ECAI.
This apparent regional bias is also evident in another pair of related conferences: the International Conference on Machine Learning (ICML) and ECML. Once again, a more U.S.-centric ICML conference series attracts far more citations—twice as many—than a similarly selective Euro-centric ECML conference series. There are several possible explanations: One is that non-European researchers are likely to miss publications at European venues (such as ECAI and ECML), so papers at these conferences pick up references only from European researchers. Another is that a pre-selection process allows researchers to keep their best work for the international conference.
There does not appear to be a case for bias between the International and European Conferences on Case Based Reasoning, although these conferences are held on alternative years, and in ICCBR was held in Norway in 2003. This table is here only because we have the data to show it.
The table below shows a breakdown of the GS Impact Factor count of ECCBR (formally known as EWCBR) versus ECAI. The European Conference on Case Based Reasoning (CBR) ranked almost as highly in 2000 and 2002 as the European Conference on AI conference, which has a much higher rejection rate. This suggests that there is some merit in targeting one's own community and publishing in niche conferences rather than the larger conferences with a more general audience. However, there is not enough evidence here to draw strong conclusions.
Google Scholar Impact Factors
This table shows a breakdown of the Google Scholar Impact factor that were calculated for each venue in each available year.
|Venue||2000||2001||2002||2003||Google Scholar Impact Factor|
These are the available Acceptance rates and their sources. These acceptance rates were used to generate the scatterplot above. The rejection rate is calculated as 100% - acceptance rate. If you have further acceptance rates please send them to us and we will add new points to our Scatterplot.
|Conference||Acceptance Rate||Number Of Submissions||Number of Accepted Papers||Notes||Source|
|UAI 2000||0.36||84.0||30||The Computational Intelligence Repository's Conference Acceptance Ratio Statistics|
|UAI 2001||0.4||84.0||0||no figures for the number of accepted papers quoted||The Computational Intelligence Repository's Conference Acceptance Ratio Statistics|
|UAI 2002||0.34||192.0||66||The Computational Intelligence Repository's Conference Acceptance Ratio Statistics|
|UAI 2003||0.336||229.0||77||25 papers accepted for oral presentation (10.9% acceptance rate) & 52 accepted for poster presentation||The Computational Intelligence Repository's Conference Acceptance Ratio Statistics|
|ECAI 2002||0.277||505.0||140||from Pádraig Cunningham|
|NIPS 2001||0.302||650.0||196||The Computational Intelligence Repository's Conference Acceptance Ratio Statistics|
|ECAI 2000||0.314||443.0||139||from Pádraig Cunningham|
|NIPS 2003||0.276||717.0||198||The Computational Intelligence Repository's Conference Acceptance Ratio Statistics|
|AAAI 2000||0.332||431.0||143||AAAI Website|
|AAAI 2002||0.258||469.0||121||AAAI Website|
|ECCV 2002||0.379||600.0||226||45 papers accepted for oral presentation (7.5% acceptance rate) & 181 accepted for poster presentation (30.2% poster acceptance rate)||The Computational Intelligence Repository's Conference Acceptance Ratio Statistics|
|ICML 2001||0.321||249.0||80||The Computational Intelligence Repository's Conference Acceptance Ratio Statistics|
|ICML 2000||0.443||349.0||151||The Computational Intelligence Repository's Conference Acceptance Ratio Statistics|
|ICML 2003||0.321||371.0||119||The Computational Intelligence Repository's Conference Acceptance Ratio Statistics|
|ICML 2002||0.33||261.0||86||The Computational Intelligence Repository's Conference Acceptance Ratio Statistics|
|IJCAI 2001||0.25||796.0||197||The Computational Intelligence Repository's Conference Acceptance Ratio Statistics|
|IJCAI 2003||0.207||913.0||189||189(regular= 20.7%) **63(poster)**||The Computational Intelligence Repository's Conference Acceptance Ratio Statistics|
|ECML 2001||0.375||240.0||50||The acceptance rate quoted here is calculated on pkdd/ecml where 90 papers were submitted. However our database only contains 50 papers. This acceptance rate is therefore an approximation (90 from 240) from the combined figures for ECML/PKDD||from Pádraig Cunningham|
|ECML 2000||0.43||100.0||43||43 papers accepted (20 + 23) from 100||from Pádraig Cunningham|
|NIPS 2002||0.311||710.0||221||The Computational Intelligence Repository's Conference Acceptance Ratio Statistics|
|ECCBR 2002||0.703||64.0||45||64 submissions (18+5+14+8) accepted||from a mail from Susan Craw to Pádraig Cunningham|
|GECCO 2003||0.686||417.0||286||194 papers accepted for oral presentation (46.5% acceptance rate) & 92 accepted for poster presentation||The Computational Intelligence Repository's Conference Acceptance Ratio Statistics|
|ICCBR 2003||0.554||92.0||51||51(19+32) from 92||from Pádraig Cunningham|
Paper conference/type Breakdown
This table shows a breakdown of how many papers of each type are in the database for each conference. If you have more up-to-date values for any of these conferences we will update the table. We used only full papers in our calculations, when that information was available.
|Conference||Type||Number Of Papers||Google Scholar Impact Factor|
|AAAI 2000||Application Paper||18||9|
|AAAI 2000||DC Paper||10||0|
|AAAI 2000||Demo Paper||12||2.5|
|AAAI 2000||Invited Paper||8||4|
|AAAI 2000||Robot Competition and Exibition Paper||1||None|
|AAAI 2000||Student Abstract Paper||36||0|
|AAAI 2000||Technical Paper||143||24.5|
|AAAI 2002||Application Paper||18||10.5|
|AAAI 2002||DC Paper||13||0|
|AAAI 2002||Demo Paper||10||0|
|AAAI 2002||Invited Paper||2||0|
|AAAI 2002||Student Abstract Paper||17||4|
|AAAI 2002||Technical Paper||120||14|
|AH 2000||DC Paper||4||3.5|
|AH 2000||Full Paper||22||19.5|
|AH 2000||Invited Paper||1||3|
|AH 2000||Short Paper||31||5|
|AH 2002||Full Paper||33||16|
|AH 2002||Invited Paper||3||33|
|AH 2002||Poster Paper||35||3|
|AH 2002||Short Paper||23||3|
|COLING 2000||Not Specified||174||10|
|COLING 2002||Not Specified||198||8|
|ECAI 2000||Not Specified||141||9|
|ECAI 2002||Not Specified||140||6|
|ECCV 2000||Not Specified||116||11.5|
|ECCV 2002||Not Specified||226||16|
|ECML 2000||Full Paper||43||6|
|ECML 2000||Invited Paper||2||18|
|ECML 2001||Full Paper||50||10.5|
|ECML 2001||Invited Paper||5||2|
|ECML 2002||Full Paper||41||7|
|ECML 2002||Invited Paper||4||19|
|ECML 2003||Full Paper||40||5|
|ECML 2003||Invited Paper||4||1|
|ECCBR 2000||Application Paper||16||6.5|
|ECCBR 2000||Invited Paper||2||4|
|ECCBR 2000||Research Paper||26||11.5|
|ECCBR 2002||Application Paper||14||4.5|
|ECCBR 2002||Invited Paper||2||8.5|
|ECCBR 2002||Research Paper||31||8|
|GECCO 2000||Full Paper||119||9|
|GECCO 2000||Poster Paper||63||3|
|GECCO 2002||Not Specified||230||6|
|GECCO 2003||Full Paper||200||5|
|GECCO 2003||Poster Paper||85||2|
|ICANN 2001||Invited Paper||3||6.5|
|ICANN 2001||Not Specified||171||3|
|ICANN 2002||Not Specified||221||3|
|ICANN 2003||Not Specified||140||2|
|ICCBR 2001||Application Paper||14||4.5|
|ICCBR 2001||Invited Paper||3||14|
|ICCBR 2001||Research Paper||36||13.5|
|ICCBR 2003||Full Paper||51||4|
|ICCBR 2003||Invited Paper||3||0.5|
|ICML 2000||Not Specified||150||15|
|ICML 2001||Not Specified||80||16|
|ICML 2002||Not Specified||87||11|
|ICML 2003||Not Specified||116||14|
|IJCAI 2001||Invited Paper||3||5|
|IJCAI 2001||Not Specified||196||20|
|IJCAI 2003||Computers and Thought Award Paper||1||29|
|IJCAI 2003||Full Paper||189||15|
|IJCAI 2003||Intelligent Systems Demonstrations||9||0|
|IJCAI 2003||Invited Speakers||10||16.5|
|IJCAI 2003||Poster Paper||87||3|
|NIPS 2000||Not Specified||153||14|
|NIPS 2001||Not Specified||196||13|
|NIPS 2002||Not Specified||207||13|
|NIPS 2003||Not Specified||198||9.5|
|TREC 2000||Not Specified||74||10|
|TREC 2001||Not Specified||84||10|
|TREC 2002||Not Specified||100||6|
|TREC 2003||Not Specified||100||6|
|UAI 2000||Not Specified||75||17|
|UAI 2001||Not Specified||71||11|
|UAI 2002||Not Specified||69||16|
|UAI 2003||Not Specified||77||10|
This work has been was supported by Science Foundation Ireland through grants 07/CE/I1147, 04/RPI/1544, 03/CE2/I303 1, and 05/IN.1/I24.