This page has some summaries of the raw data that was used/generated for our citation analysis research, which led to the following publications This page was (mostly) automatically generated from our Citation Analysis Scripts (FYI: the original automatically generated page is here). Our study gathered data from a total of 8764 papers (15 conferences with 3258 papers and 17 journals with 5506 articles). Please get in contact with the authors if you want more information on this work.

Correlation between ISI Impact Factor and Google Scholar Impact Factor for Selected Journals

There is a strong correlation between citation counts computed from Google Scholar data and comparable data from the ISI Web of Knowledge index, validating the use of our new Google Scholar Impact Factor as an alternative citation-based evaluation metric applicable to both journals and conferences. The Pearson correlation coefficient between the two is 0.86. The table below shows the raw values.
ISI and Google Scholar Impact Factors for Selected Journals
CategoryJournal NameISI Impact FactorGS Impact Factor
A* Journal of the ACM2.935
BPattern Analysis Applications0.48
A*Transactions on Knowledge and Data Engineering2.115
A Information Retrieal1.79.5
A* Artificial Intelligence2.322
B AI EDAM0.45
A Computational Intelligence1.49
A* IEEE Trans. on Pattern Analysis and Machine Intelligence4.334
B AI Communications0.54
A AI in Medicine1.612
B International Journal of Pattern Recognition0.54
A Decision Support Systems1.217.5
A* Machine Learning2.632.5
A* International Journal of Computer Vision627
A Data and Knowledge Engineering1.48

Scatterplot of Available Rejection Rates versus Google Scholar Impact Factors

Below is a scatter plot of GS impact factor against conference rejection rates for 23 conferences across the 15 conference series we evaluated; note the data points reflect a subset of the full set of conferences, namely those for which we were able to obtain. Reliable rejection rates could not be obtained for all of the conferences for which we had data. More details on the rejection rates we used are below. While our study indicate some correlation between GS impact factor and rejection rates, the related Pearson correlation score of 0.53 was not very convincing and reflects considerable variation when it comes to the relationship between conference rejection rates and a conference paper’s ability to attract citations. There are conferences with similar rejection rates but very different Google Scholar impact factors. There are also conferences with very different rejection rates that still manage to achieve similar GS impact factors. For instance, AAAI achieves a Google Scholar impact factor of 20, with a rejection rate of 65%–75%. Its European counterpart, ECAI the European Conference on Artificial Intelligence, is just as selective but achieves a median GS impact factor of only 7 (see the section on regional bias below). Another example of a lack of correlation between paper acceptance rate and citation count is the 2002 European Conference on Case-Based Reasoning (ECCBR), which had a rejection rate of approximately 0.33 33% and Google Scholar impact factor of 7, achieving a citation rate better than the European Conference on Machine Learning?? 2000 and ECAI 2002, which had twice its rejection rate (see the Section below on open-world versus closed-world bias). To further test the strength of the correlation between GS impact factor and rejection rate we examined the available data for three conference series that took place between 2000-2003; UAI, ICML, and ECML were the only conferences that occurred and had published rejection rates available in every year of the study. Table 5 outlines the GS impact factors and rejection rates for each year in the study. What is interesting here is there is no significant correlation between rejection rate and GS impact factor. The Pearson score for UAI is was?? only 0.27, for ICML 0.24, and for ECML 0.42. This suggests that, at least for these conferences, the yearly changes in rejection rates had little bearing on expected citation count. These results highlight the assumption?? the fact?? that any assumed relationship between conference rejection rates and a conference’s ability to attract citations is at best weak, so other factors play a more critical role when it comes to influencing future citations. Scatterplot of Available Rejection Rates versus Google Scholar Impact Factors
Scatter Plot Values
ConferenceRejection RateGoogle Scholar Impact Factor
UAI 20000.6417.00
UAI 20010.6011.00
UAI 20020.6616.00
UAI 20030.6610.00
ECAI 20020.726.00
NIPS 20010.7013.00
ECAI 20000.699.00
NIPS 20030.729.50
AAAI 20000.6724.50
AAAI 20020.7414.00
ECCV 20020.6216.00
ICML 20010.6816.00
ICML 20000.5615.00
ICML 20030.6814.00
ICML 20020.6711.00
IJCAI 20010.7520.00
IJCAI 20030.7915.00
ECML 20010.6210.50
ECML 20000.576.00
NIPS 20020.6913.00
ECCBR 20020.307.00
GECCO 20030.314.00
ICCBR 20030.454.00

International versus Local Venues

We found evidence of strong regional bias between similar conferences, with international (mainly U.S.-centric) conferences attracting much higher citation counts that their similarly selective though less-well-cited European counterparts. Both the AAAI and ECAI conferences target the same research area and attract submissions from a similar community of researchers in a way that is equally selective. Yet the U.S.-centric AAAI enjoys an expected citation count (computed from the product of the median citation count and the rejection rate of the conference) more than twice that of ECAI.
Google Scholar Rates for the AAAI Conference on Artificial Intelligence versus the European Conference on Artificial Intelligence between 2000-2003
YearAAAIECAI
200024.59
2001NoneNone
2002146
2003NoneNone
This apparent regional bias is also evident in another pair of related conferences: the International Conference on Machine Learning (ICML) and ECML. Once again, a more U.S.-centric ICML conference series attracts far more citations—twice as many—than a similarly selective Euro-centric ECML conference series. There are several possible explanations: One is that non-European researchers are likely to miss publications at European venues (such as ECAI and ECML), so papers at these conferences pick up references only from European researchers. Another is that a pre-selection process allows researchers to keep their best work for the international conference.
Google Scholar Rates for the International Conference on Machine Learning versus the European Conference on Machine Learning between 2000-2003
YearICMLECML
2000156
20011610.5
2002117
2003145
There does not appear to be a case for bias between the International and European Conferences on Case Based Reasoning, although these conferences are held on alternative years, and in ICCBR was held in Norway in 2003. This table is here only because we have the data to show it.
Google Scholar Rates for the International Conference on Case Based Reasoning versus the European Conference on Case Based Reasoning between 2000-2003
YearICCBRECCBR
2000None8
20019None
2002None7
20034None

Open versus Closed World Venues

The table below shows a breakdown of the GS Impact Factor count of ECCBR (formally known as EWCBR) versus ECAI. The European Conference on Case Based Reasoning (CBR) ranked almost as highly in 2000 and 2002 as the European Conference on AI conference, which has a much higher rejection rate. This suggests that there is some merit in targeting one's own community and publishing in niche conferences rather than the larger conferences with a more general audience. However, there is not enough evidence here to draw strong conclusions.
Google Scholar Rates for the European Conference on Case Based Reasoning versus the European Conference on Artificial Intelligence between 2000-2003
YearECCBRECAI
200089
2001NoneNone
200276
2003NoneNone

Google Scholar Impact Factors

This table shows a breakdown of the Google Scholar Impact factor that were calculated for each venue in each available year.
Google Scholar Impact Factors for each available year
Venue2000200120022003Google Scholar Impact Factor
TKDE2314151415
PE911958.5
PAMI4332362634
PAA118978
ML35.532362932.5
JACM32.55951.524.535
IR1586.559.5
INFFUS1010.5106.59
IJPRAI45.5534
IJCV36.513.5272127
DSS1718181717.5
DKE1010.5868
CI7.58179
ARTMED159.51012.512
AIEDAM54635
AICOM1.547.52.54
AI272222.51722
UAI1711161013
TREC1010668
NIPS1413139.512
IJCAI201517
ICML1516111414
ICCBR946
ICANN3323
GECCO7645
ECCBR878
ECML610.5757
ECCV11.51613.5
ECAI967
COLING1089
AH19.51617
AAAI24.51420

Table of Available Acceptance Rates

These are the available Acceptance rates and their sources. These acceptance rates were used to generate the scatterplot above. The rejection rate is calculated as 100% - acceptance rate. If you have further acceptance rates please send them to us and we will add new points to our Scatterplot.
Table of available acceptance rates and the source of each value
ConferenceAcceptance RateNumber Of SubmissionsNumber of Accepted PapersNotesSource
UAI 20000.3684.030The Computational Intelligence Repository's Conference Acceptance Ratio Statistics
UAI 20010.484.00no figures for the number of accepted papers quotedThe Computational Intelligence Repository's Conference Acceptance Ratio Statistics
UAI 20020.34192.066The Computational Intelligence Repository's Conference Acceptance Ratio Statistics
UAI 20030.336229.07725 papers accepted for oral presentation (10.9% acceptance rate) & 52 accepted for poster presentationThe Computational Intelligence Repository's Conference Acceptance Ratio Statistics
ECAI 20020.277505.0140from Pádraig Cunningham
NIPS 20010.302650.0196The Computational Intelligence Repository's Conference Acceptance Ratio Statistics
ECAI 20000.314443.0139from Pádraig Cunningham
NIPS 20030.276717.0198The Computational Intelligence Repository's Conference Acceptance Ratio Statistics
AAAI 20000.332431.0143AAAI Website
AAAI 20020.258469.0121AAAI Website
ECCV 20020.379600.022645 papers accepted for oral presentation (7.5% acceptance rate) & 181 accepted for poster presentation (30.2% poster acceptance rate)The Computational Intelligence Repository's Conference Acceptance Ratio Statistics
ICML 20010.321249.080The Computational Intelligence Repository's Conference Acceptance Ratio Statistics
ICML 20000.443349.0151The Computational Intelligence Repository's Conference Acceptance Ratio Statistics
ICML 20030.321371.0119The Computational Intelligence Repository's Conference Acceptance Ratio Statistics
ICML 20020.33261.086The Computational Intelligence Repository's Conference Acceptance Ratio Statistics
IJCAI 20010.25796.0197The Computational Intelligence Repository's Conference Acceptance Ratio Statistics
IJCAI 20030.207913.0189189(regular= 20.7%) **63(poster)**The Computational Intelligence Repository's Conference Acceptance Ratio Statistics
ECML 20010.375240.050The acceptance rate quoted here is calculated on pkdd/ecml where 90 papers were submitted. However our database only contains 50 papers. This acceptance rate is therefore an approximation (90 from 240) from the combined figures for ECML/PKDD from Pádraig Cunningham
ECML 20000.43100.04343 papers accepted (20 + 23) from 100 from Pádraig Cunningham
NIPS 20020.311710.0221The Computational Intelligence Repository's Conference Acceptance Ratio Statistics
ECCBR 20020.70364.04564 submissions (18+5+14+8) acceptedfrom a mail from Susan Craw to Pádraig Cunningham
GECCO 20030.686417.0286194 papers accepted for oral presentation (46.5% acceptance rate) & 92 accepted for poster presentationThe Computational Intelligence Repository's Conference Acceptance Ratio Statistics
ICCBR 20030.55492.05151(19+32) from 92from Pádraig Cunningham

Paper conference/type Breakdown

This table shows a breakdown of how many papers of each type are in the database for each conference. If you have more up-to-date values for any of these conferences we will update the table. We used only full papers in our calculations, when that information was available.
Breakdown of the numbers of different types of paper for each conference, where available
ConferenceTypeNumber Of PapersGoogle Scholar Impact Factor
AAAI 2000Application Paper189
AAAI 2000DC Paper100
AAAI 2000Demo Paper122.5
AAAI 2000Invited Paper84
AAAI 2000Robot Competition and Exibition Paper1None
AAAI 2000Student Abstract Paper360
AAAI 2000Technical Paper14324.5
AAAI 2002Application Paper1810.5
AAAI 2002DC Paper130
AAAI 2002Demo Paper100
AAAI 2002Invited Paper20
AAAI 2002Student Abstract Paper174
AAAI 2002Technical Paper12014
AH 2000DC Paper43.5
AH 2000Full Paper2219.5
AH 2000Invited Paper13
AH 2000Short Paper315
AH 2002Full Paper3316
AH 2002Invited Paper333
AH 2002Poster Paper353
AH 2002Short Paper233
COLING 2000Not Specified17410
COLING 2002Not Specified1988
ECAI 2000Not Specified1419
ECAI 2002Not Specified1406
ECCV 2000Not Specified11611.5
ECCV 2002Not Specified22616
ECML 2000Full Paper436
ECML 2000Invited Paper218
ECML 2001Full Paper5010.5
ECML 2001Invited Paper52
ECML 2002Full Paper417
ECML 2002Invited Paper419
ECML 2003Full Paper405
ECML 2003Invited Paper41
ECCBR 2000Application Paper166.5
ECCBR 2000Invited Paper24
ECCBR 2000Research Paper2611.5
ECCBR 2002Application Paper144.5
ECCBR 2002Invited Paper28.5
ECCBR 2002Research Paper318
GECCO 2000Full Paper1199
GECCO 2000Poster Paper633
GECCO 2002Not Specified2306
GECCO 2003Full Paper2005
GECCO 2003Poster Paper852
ICANN 2001Invited Paper36.5
ICANN 2001Not Specified1713
ICANN 2002Not Specified2213
ICANN 2003Not Specified1402
ICCBR 2001Application Paper144.5
ICCBR 2001Invited Paper314
ICCBR 2001Research Paper3613.5
ICCBR 2003Full Paper514
ICCBR 2003Invited Paper30.5
ICML 2000Not Specified15015
ICML 2001Not Specified8016
ICML 2002Not Specified8711
ICML 2003Not Specified11614
IJCAI 2001Invited Paper35
IJCAI 2001Not Specified19620
IJCAI 2003Computers and Thought Award Paper129
IJCAI 2003Full Paper18915
IJCAI 2003Intelligent Systems Demonstrations90
IJCAI 2003Invited Speakers1016.5
IJCAI 2003Poster Paper873
NIPS 2000Not Specified15314
NIPS 2001Not Specified19613
NIPS 2002Not Specified20713
NIPS 2003Not Specified1989.5
TREC 2000Not Specified7410
TREC 2001Not Specified8410
TREC 2002Not Specified1006
TREC 2003Not Specified1006
UAI 2000Not Specified7517
UAI 2001Not Specified7111
UAI 2002Not Specified6916
UAI 2003Not Specified7710

Acknowledgements

This work has been was supported by Science Foundation Ireland through grants 07/CE/I1147, 04/RPI/1544, 03/CE2/I303 1, and 05/IN.1/I24. We gathered and generated data using Python scripts, taking the details of accepted papers from DBLP and citation figures from Google Scholar. Matlab was used to generate the scatterplots.