Elsevier

Journal of Informetrics

Volume 5, Issue 4, October 2011, Pages 698-704
Journal of Informetrics

Short communication
Comparing impact factors from two different citation databases: The case of Computer Science

https://doi.org/10.1016/j.joi.2011.01.007Get rights and content

Abstract

Journal impact factors continue to play an important role in research output assessment, in spite of the criticisms and debates around them. The impact factor rankings provided in the Journal Citation Reports (JCR™) database by Thompson Reuters have enjoyed a position of monopoly for many years. But this has recently changed with the availability of the Scopus™ database and its associated journal ranking published in the Scimago Journal Rank (SJR) Web page, as the former provides a citation database with similar inclusion criteria to those used in the JCR and the latter and openly accessible impact factor-based ranking. The availability of alternatives to the JCR impact factor listings using a different citation database raises the question of the extent to which the two rankings can be considered equally valid for research evaluation purposes. This paper reports the results of a contrast of both listings in Computer Science-related topics. It attempts to answer the validity question by comparing the impact factors of journals ranked in both listings and their relative position. The results show that impact factors for journals included in both rankings are strongly correlated, with SJR impact factors in general slightly higher, confirming previous studies related to other disciplines. Nonetheless, the consideration of tercile and quartile position of journal yields some divergences for journals appearing in both rankings that need to be accounted for in research evaluation procedures.

Introduction

The evaluation of research quality is of paramount importance for individual scientists, institutions assessing promotion and funding organizations and investors in general. The impact factor (IF) provided by Thompson Scientific (or Thompson Reuters) is a metric widely used for research evaluation purposes and included in their Journal Citation Reports (JCR™) database. The IF was proposed in by Garfield (1955) and reflects the frequency with which the journal's articles are cited in scientific literature. A journal's IF is the average of the number of citations in the current year to items published in the previous two years in that journal.

There has been considerable debate and criticism around the IF, especially considering that it does not capture all the important aspects required for the complex task of assessing the research output of individuals and groups (Opthof, 1997). Indeed, the IF covers only impact, and does it by considering a single, specific measure. Aspects potentially important that are not covered by the IF include, for example, usage metrics (Bollen, Van de Sompel, Smith, & Luce, 2005) or prestige (Habibzadeh & Yadollahie, 2008). Even though they do not provide a perfect tool as recognized by its proposer (Garfield, 2006), impact factor rankings are still widely used worldwide for research evaluation, probably due to the fact that they provide a quantitative and cost-effective assessment method. Recent research has advanced in developing an axiomatic analysis of impact factors when used as tools for ranking journals (Bouyssou & Marchant, 2011). In consequence, it is worth the effort to continue analyzing and contrasting its validity.

For years, Thompson Reuters has been in a position of dominance as provider of impact factors, based on an established journal selection process and the regular maintenance of a considerably large citation database. But recently, an open alternative to the JCR has appeared (Butler, 2008a). The SJR (SCImago Journal Rank1) site offers journal rankings based on several measures of impact including the classical IF, but using Scopus™2 as its data source rather than Thompson Reuters’ citation databases. There are other open alternative citation databases as CiteSeer and Google Scholar, however these are very different in that they attempt to automatically index resources, following a very different approach to data collection and thus resulting in divergent quality and coverage figures (Jacsó, 2005).

The SJR indicator (Gonzalez-Pereira et al., 2010, SCImago, 2007) provided in the SJR site is a metric alternative to the IF which weights citations based on the impact of the citing journal. There have been some attempts to contrast the IF provided in the JCR with the SJR indicator (Lopez-Illescas et al., 2008, Schopfel and Prost, 2009). However, the SJR indicator takes into account other elements that are not considered in the JCR IF, so that it can be hypothesized that it is measuring a different aspect of impact than the IF. The SJR site (not to be confused with the indicator of the same name) provides also the classical Garfield's IF metric available also in the JCR database, so that it is possible to contrast the same indicator as computed from two different citation databases. In consequence, we focus on classical IF for the comparison. This contrast is important for several reasons. From a theoretical perspective, it can be hypothesized that the same indicator computed separately from two different but homogeneous citation databases should yield similar rankings. If this is not the case in general, we can draw the conclusion that journal selection policies (or how these policies are applied) are significantly divergent, and the two rankings should not be used interchangeably. A similarity in the rankings also provides evidence on the lack of errors and bias in each of the databases, and citing based on that, the relative position of a journal in both listings when available can be considered as complementary evidence about the publication's impact. From a practical perspective, if the rankings result to be similar, any of the rankings can be used by researchers and evaluators as evidence of impact. This paper reports in a concrete study addressing the discipline of Computer Science, complementing existing related studies (Kimura, 2008, Lopez-Illescas et al., 2008, Schopfel and Prost, 2009). This can be extended to a contrast of different categories and disciplines, thus evaluating the coverage of the databases piecewise.

In order to compare both impact factor rankings, a number of issues need to be considered. The most important one is that journal coverage in JCR and SJR (Scopus) are not identical, as Scopus database is significantly larger in number of journals included. But ideally, similar rankings considering journal relative position would arise from the two, as both are using the same measures with two different samples and similar general journal selection criteria (Bühringer, Metzner, Lämmle, & Künzel, 2006). If the differences are not significant enough to affect potential assessment situations, then both computed rankings could be considered equally valid for research evaluation. This would represent the great benefit of having available a larger coverage of journals, considering both sources as complementary. A concern has been raised in the coverage of certain disciplines in either JCR or Scopus. For example, Togia and Tsigilis (2006) raised such concern for the field of education. Such divergences may result in significantly different IF distributions, so that coverage analysis is required on a discipline basis. However, in other disciplines, these concerns have not been raised to date and citation database contrast can be done under the assumption that journal selection have resulted in representative samples. A recent study by Lopez-Illescas et al. (2008) showed that for the field of oncology, the Web of Science (WoS) database (the citation database underlying Thompson Reuters JCR) is a genuine subset of Scopus, and tends to cover the best journals from it in terms of citation impact per paper.

This paper reports a study comparing impact factor rankings for the disciplines related to Computer Science in JCR and SJR. The study was carried out by extracting impact factor rankings of the last five years from both listings, and then evaluating their similarity, especially considering the common evaluation situation in which journal papers are assessed based on their relative position (e.g. terciles or quartiles). The results show that JCR and SJR rankings for the discipline of Computer Science show high significant linear correlations. Relative position analysis shows also high degrees of similarity with some exceptions that deserve further inquiry. Concretely, in a few cases, if quartiles are used to evaluate journal relative position, JCR and SJR may lead to different results, and this needs to be carefully considered.

The rest of this paper is structured as follows. Section 2 reports the data collection, preparation and basic descriptive statistics. Then, data analysis and hypotheses testing together with results discussion are provided in Section 3. Section 4 discusses limitations of the study and Section 5 provides conclusions and outlook.

Section snippets

Data gathering and descriptive analysis

Two-year impact factors were obtained from Computer Science subject categories from 2004 onwards, both from the JCR and from the SJR Web interfaces3 and a relational database was developed following a similar design as Mallig (2010). The journals considered were those using “Computer Science” as a prefix in the name of subject categories (in the case of the JCR), and those classified under “Computer

Contrast and discussion

The relation of the impact factors of the journals covered by both listings was analyzed using the non-parametric Spearman's rank correlation coefficient, and also performing linear regression. Results are provided in Table 3.

Results show a strong significant correlation between the impact factors in JCR and SJR, again with similar figures as those reported by Lopez-Illescas et al. (2008) for oncology. This can be considered significant evidence in favour of considering both citation databases

Limitations

Subject categories in both JCR and SJR are broad categories without a clear definition of the contents covered. Indeed, many journals are cross-cutting several of the categories listed, and classification is often challenging (Leydesdorff & Rafols, 2009). A broader definition of computing could have covered for example journals in the category of “Medical informatics”. The discrimination of the “Computer Science” discipline by using the top level breakdown of both rankings was selected as a

Conclusions and outlook

The results of the contrast of classical impact factors in JCR and SJR listings for the discipline of Computer Science confirm that in spite of the differences in the underlying citation databases, they are highly correlated and yield comparable rankings. This can be used as evidence supporting the hypotheses that both rankings are similarly valid for research assessment, as they will point to similar relative journal impact assessments when approaching research output evaluation. It should be

References (21)

There are more references available in the full text version of this article.

Cited by (21)

  • Sustainable agrifood supply chains: Bibliometric, network and content analyses

    2022, Science of the Total Environment
    Citation Excerpt :

    Table 4 also shows information on the quality of these journals, i.e., their Impact Factor (IF) (for the reference year 2019) and their best rank in the indexing Scopus categories. All the top ten leading journals are positioned in the first quartile (Q1), as per Scimago Journal Rank (SJR), which assigns weights to bibliographic citations based on the journal relevance, so that citations issued by the most important journals are more valuable than those issued by less important ones (Sicilia et al., 2011). Table 5 shows instead the top six most productive authors in terms of published documents within the analyzed field of research, as well as in terms of productivity, through their H-index and i-10 index.

  • Wind energy research in Mexico

    2018, Renewable Energy
    Citation Excerpt :

    In Table 4 is shown the top 30 journals where Mexican researchers have written during the period 1969–2016, this information contains the number of items of each journal; the impact factor of the JCR and the SCImago Journal Rank from Scopus, SJR, as well as, h-index and country. The journals Astrophysical Journal, Journal od Geophysical Research Atmospheres, Atmospheric Environment and Atmospheric Chemistry and Physics have the highest amount of references, their impact factor from JCR and SRJ has a correlation coefficient of 0.81 that indicates, that they have good relationship, as founded by Refs. [68–75], another important information obtained is that USA has most of the principal journals publishing about wind, followed by England, Netherland, France and Germany. h-index apparently presents some variation with respect to the impact factor of JCR and SRJ as show [53,76–78] [1–3], but in this case the correlation coefficient between h-index versus JCR and SRJ are 0.64 and 0.59 respectively.

View all citing articles on Scopus
View full text