Short communicationComparing impact factors from two different citation databases: The case of Computer Science
Introduction
The evaluation of research quality is of paramount importance for individual scientists, institutions assessing promotion and funding organizations and investors in general. The impact factor (IF) provided by Thompson Scientific (or Thompson Reuters) is a metric widely used for research evaluation purposes and included in their Journal Citation Reports (JCR™) database. The IF was proposed in by Garfield (1955) and reflects the frequency with which the journal's articles are cited in scientific literature. A journal's IF is the average of the number of citations in the current year to items published in the previous two years in that journal.
There has been considerable debate and criticism around the IF, especially considering that it does not capture all the important aspects required for the complex task of assessing the research output of individuals and groups (Opthof, 1997). Indeed, the IF covers only impact, and does it by considering a single, specific measure. Aspects potentially important that are not covered by the IF include, for example, usage metrics (Bollen, Van de Sompel, Smith, & Luce, 2005) or prestige (Habibzadeh & Yadollahie, 2008). Even though they do not provide a perfect tool as recognized by its proposer (Garfield, 2006), impact factor rankings are still widely used worldwide for research evaluation, probably due to the fact that they provide a quantitative and cost-effective assessment method. Recent research has advanced in developing an axiomatic analysis of impact factors when used as tools for ranking journals (Bouyssou & Marchant, 2011). In consequence, it is worth the effort to continue analyzing and contrasting its validity.
For years, Thompson Reuters has been in a position of dominance as provider of impact factors, based on an established journal selection process and the regular maintenance of a considerably large citation database. But recently, an open alternative to the JCR has appeared (Butler, 2008a). The SJR (SCImago Journal Rank1) site offers journal rankings based on several measures of impact including the classical IF, but using Scopus™2 as its data source rather than Thompson Reuters’ citation databases. There are other open alternative citation databases as CiteSeer and Google Scholar, however these are very different in that they attempt to automatically index resources, following a very different approach to data collection and thus resulting in divergent quality and coverage figures (Jacsó, 2005).
The SJR indicator (Gonzalez-Pereira et al., 2010, SCImago, 2007) provided in the SJR site is a metric alternative to the IF which weights citations based on the impact of the citing journal. There have been some attempts to contrast the IF provided in the JCR with the SJR indicator (Lopez-Illescas et al., 2008, Schopfel and Prost, 2009). However, the SJR indicator takes into account other elements that are not considered in the JCR IF, so that it can be hypothesized that it is measuring a different aspect of impact than the IF. The SJR site (not to be confused with the indicator of the same name) provides also the classical Garfield's IF metric available also in the JCR database, so that it is possible to contrast the same indicator as computed from two different citation databases. In consequence, we focus on classical IF for the comparison. This contrast is important for several reasons. From a theoretical perspective, it can be hypothesized that the same indicator computed separately from two different but homogeneous citation databases should yield similar rankings. If this is not the case in general, we can draw the conclusion that journal selection policies (or how these policies are applied) are significantly divergent, and the two rankings should not be used interchangeably. A similarity in the rankings also provides evidence on the lack of errors and bias in each of the databases, and citing based on that, the relative position of a journal in both listings when available can be considered as complementary evidence about the publication's impact. From a practical perspective, if the rankings result to be similar, any of the rankings can be used by researchers and evaluators as evidence of impact. This paper reports in a concrete study addressing the discipline of Computer Science, complementing existing related studies (Kimura, 2008, Lopez-Illescas et al., 2008, Schopfel and Prost, 2009). This can be extended to a contrast of different categories and disciplines, thus evaluating the coverage of the databases piecewise.
In order to compare both impact factor rankings, a number of issues need to be considered. The most important one is that journal coverage in JCR and SJR (Scopus) are not identical, as Scopus database is significantly larger in number of journals included. But ideally, similar rankings considering journal relative position would arise from the two, as both are using the same measures with two different samples and similar general journal selection criteria (Bühringer, Metzner, Lämmle, & Künzel, 2006). If the differences are not significant enough to affect potential assessment situations, then both computed rankings could be considered equally valid for research evaluation. This would represent the great benefit of having available a larger coverage of journals, considering both sources as complementary. A concern has been raised in the coverage of certain disciplines in either JCR or Scopus. For example, Togia and Tsigilis (2006) raised such concern for the field of education. Such divergences may result in significantly different IF distributions, so that coverage analysis is required on a discipline basis. However, in other disciplines, these concerns have not been raised to date and citation database contrast can be done under the assumption that journal selection have resulted in representative samples. A recent study by Lopez-Illescas et al. (2008) showed that for the field of oncology, the Web of Science (WoS) database (the citation database underlying Thompson Reuters JCR) is a genuine subset of Scopus, and tends to cover the best journals from it in terms of citation impact per paper.
This paper reports a study comparing impact factor rankings for the disciplines related to Computer Science in JCR and SJR. The study was carried out by extracting impact factor rankings of the last five years from both listings, and then evaluating their similarity, especially considering the common evaluation situation in which journal papers are assessed based on their relative position (e.g. terciles or quartiles). The results show that JCR and SJR rankings for the discipline of Computer Science show high significant linear correlations. Relative position analysis shows also high degrees of similarity with some exceptions that deserve further inquiry. Concretely, in a few cases, if quartiles are used to evaluate journal relative position, JCR and SJR may lead to different results, and this needs to be carefully considered.
The rest of this paper is structured as follows. Section 2 reports the data collection, preparation and basic descriptive statistics. Then, data analysis and hypotheses testing together with results discussion are provided in Section 3. Section 4 discusses limitations of the study and Section 5 provides conclusions and outlook.
Section snippets
Data gathering and descriptive analysis
Two-year impact factors were obtained from Computer Science subject categories from 2004 onwards, both from the JCR and from the SJR Web interfaces3 and a relational database was developed following a similar design as Mallig (2010). The journals considered were those using “Computer Science” as a prefix in the name of subject categories (in the case of the JCR), and those classified under “Computer
Contrast and discussion
The relation of the impact factors of the journals covered by both listings was analyzed using the non-parametric Spearman's rank correlation coefficient, and also performing linear regression. Results are provided in Table 3.
Results show a strong significant correlation between the impact factors in JCR and SJR, again with similar figures as those reported by Lopez-Illescas et al. (2008) for oncology. This can be considered significant evidence in favour of considering both citation databases
Limitations
Subject categories in both JCR and SJR are broad categories without a clear definition of the contents covered. Indeed, many journals are cross-cutting several of the categories listed, and classification is often challenging (Leydesdorff & Rafols, 2009). A broader definition of computing could have covered for example journals in the category of “Medical informatics”. The discrimination of the “Computer Science” discipline by using the top level breakdown of both rankings was selected as a
Conclusions and outlook
The results of the contrast of classical impact factors in JCR and SJR listings for the discipline of Computer Science confirm that in spite of the differences in the underlying citation databases, they are highly correlated and yield comparable rankings. This can be used as evidence supporting the hypotheses that both rankings are similarly valid for research assessment, as they will point to similar relative journal impact assessments when approaching research output evaluation. It should be
References (21)
- et al.
Toward alternative metrics of journal impact: A comparison of download and citation data
Information Processing & Management
(2005) - et al.
Bibliometric rankings of journals based on impact factors: An axiomatic approach
Journal of Informetrics
(2011) The skewness of computer science
Information Processing & Management
(2011)- et al.
A new approach to the metric of journals’ scientific prestige: The SJR indicator
Journal of Informetrics
(2010) - et al.
Journal weighted impact factor: A proposal
Journal of Informetrics
(2008) - et al.
Coverage and citation impact of oncological journals in the Web of Science and Scopus
Journal of Informetrics
(2008) A relational database for bibliometric analysis
Journal of Informetrics
(2010)- et al.
Le JCR facteur d’impact (IF) et le SCImago Journal Rank Indicator (SJR) des revues francaises: une etude comparative
Psychologie Francaise
(2009) The development of an AI journal ranking based on the revealed preference approach
Journal of Informetrics
(2010)- et al.
Impact factor and education journals: a critical examination and analysis
International Journal of Educational Research
(2006)
Cited by (21)
Sustainable agrifood supply chains: Bibliometric, network and content analyses
2022, Science of the Total EnvironmentCitation Excerpt :Table 4 also shows information on the quality of these journals, i.e., their Impact Factor (IF) (for the reference year 2019) and their best rank in the indexing Scopus categories. All the top ten leading journals are positioned in the first quartile (Q1), as per Scimago Journal Rank (SJR), which assigns weights to bibliographic citations based on the journal relevance, so that citations issued by the most important journals are more valuable than those issued by less important ones (Sicilia et al., 2011). Table 5 shows instead the top six most productive authors in terms of published documents within the analyzed field of research, as well as in terms of productivity, through their H-index and i-10 index.
Conference presentations and academic publishing
2021, Economic ModellingWind energy research in Mexico
2018, Renewable EnergyCitation Excerpt :In Table 4 is shown the top 30 journals where Mexican researchers have written during the period 1969–2016, this information contains the number of items of each journal; the impact factor of the JCR and the SCImago Journal Rank from Scopus, SJR, as well as, h-index and country. The journals Astrophysical Journal, Journal od Geophysical Research Atmospheres, Atmospheric Environment and Atmospheric Chemistry and Physics have the highest amount of references, their impact factor from JCR and SRJ has a correlation coefficient of 0.81 that indicates, that they have good relationship, as founded by Refs. [68–75], another important information obtained is that USA has most of the principal journals publishing about wind, followed by England, Netherland, France and Germany. h-index apparently presents some variation with respect to the impact factor of JCR and SRJ as show [53,76–78] [1–3], but in this case the correlation coefficient between h-index versus JCR and SRJ are 0.64 and 0.59 respectively.
How Did Journals in Water Sciences Survive the COVID-19 Pandemic? A Scientometric Study
2023, Limnological ReviewPublication Activity as a Research Efficiency Indicator by the Example of Chemical Organizations
2020, Herald of the Russian Academy of SciencesExpatriate management and national culture: a bibliometric study of prolific, productive, and most cited authors and institutions
2020, International Journal of Human Resource Management