Challenges of scholarly publications on the Web to the evaluation of science—A comparison of author visibility on the Web and in print journals

https://doi.org/10.1016/j.ipm.2005.03.013Get rights and content

Abstract

This article reveals different patterns of scholarly communication in the XML research field on the Web and in print journals in terms of author visibility, and challenges the common practice of exclusively using the ISI’s databases to obtain citation counts as scientific performance indicators. Results from this study demonstrate both the importance and the feasibility of the use of multiple citation data sources in citation analysis studies of scholarly communication, and provide evidence for a developing “two tier” scholarly communication system.

Introduction

As the accelerated development of information technology, especially the rapid growth of the Web, is changing the circumstances and consequently the structures and processes of scholarly communication, there is renewed interest in the study of scholarly communication to see how it is being transformed, what the similarities or differences between the new formats of communication and the traditional ones might be, and how the new formats facilitate or inhibit the scholarly communication process (Borgman and Furner, 2002, Cronin, 2001, Zhao, 2003).

Citation analysis and other informetric techniques have been applied successfully to the study of this new phenomenon in scholarly communication. As Zhao and Logan (2002) point out, some studies apply, often with modifications, informetric principles and techniques to study the characteristics and link structures of the Web. Examples include studies on search engines that make use of hyperlink structure (Clever Project, 1999), and so-called “Webometrics” studies (Almind and Ingwersen, 1997, Cronin et al., 1998, Egghe, 2000, Larson, 1996, Rousseau, 1997, Thelwall and Harries, 2004, Turnbull, 2000, Wilkinson et al., 2003). Other studies consider “electronic ingredients” in journal articles—either in reference lists or in abstracts—to gauge the impact of electronic publications on traditional print journal-based scholarly communication (Harter, 1992, Harter and Kim, 1996, ISI, 2004a, Lu, 1999, McCain, 2000, Youngen, 1997). Still others examine scholarly communication patterns demonstrated in research papers published on the Web, and study the differences from, and similarities to, what we have come to expect from print journal-based communication (Goodrum et al., 2001, Lawrence, 2001, The Open Citation Project, 2001, Zhao and Logan, 2002, Zhao, 2003, Zhao, 2004).

As part of a larger research project that aims to systematically compare scholarly communication patterns between the Web and the print world, the present study focuses on the comparison of author visibility between the Web and print journals as revealed from citation analysis, and discusses the challenges of scholarly communication being increasingly conducted over the Internet to traditional scholarly communication system in general and to the common practice of science evaluation based on the databases of the Institute for Scientific Information (ISI) in particular. The present study along with other parts of the project (Zhao, 2004, Zhao and Logan, 2002, Zhao and Strotmann, 2004) may contribute to the understanding of the transition of scholarly communication from print to electronic media, to advancing citation analysis theory and methodology, and to information organization and retrieval on the Web.

Section snippets

Research questions

Within a citation analysis framework, author visibility can either be measured in terms of how frequently authors have been publishing or in terms of how often their published works have been used (cited) by other scholars. Based on this consideration, the research questions to be explored in the present study are as follows.

  • Are there any significant correlations between author rankings by number of publications identified from the Web and those identified from print journals in the field of

Data collection

The ISI’s Science Citation Index (SCI) and The NEC Research Institute’s CiteSeer1 were used in the present study to collect information on research papers published in print journals and on the Web, respectively. To date, the ISI databases including SCI have been used as the single data source for most of

Findings

A search on “XML” or “eXtensible Markup Language” resulted in 312 papers using CiteSeer (after removing duplicates) and in 374 papers with reference lists using SCI, 268 of which from computer science journals. Among these papers, only 26 were common to both CiteSeer and SCI. The papers from CiteSeer made 4578 citations, and those from SCI made 6782 citations. Among the cited papers from CiteSeer, 22% (987) were proceedings, 22% (991) were Web publications, 2.3% (105) were technical reports,

Discussion

The findings presented above can shed some light on issues of both citation analysis and scholarly communication in transition.

Conclusion

Scholarly communication is increasingly being conducted over the Internet, certainly in the core research areas of computer science but increasingly also in other sciences. This has brought some challenges as well as opportunities to the traditional scholarly communication systems and to the associated evaluation approaches. The present study has explored these through a citation analysis of author visibility in one particular research field—the XML research field. It has revealed different

Acknowledgement

The author wishes to thank Dr. Andreas Strotmann of the Center for Applied Computer Science at the University of Cologne, for his many helpful insights.

This work was supported in part by a Fellowship of the School of Computational Science and Information Technology, Florida State University.

Dangzhi Zhao is an Assistant Professor at the School of Library and Information Studies of the University of Alberta, Canada. Her research interests are in the areas of information systems, scholarly communication, and information technology. Recent research concentrates on the comparison of scholarly communication between the Web and the print world.

References (38)

  • A.A. Goodrum et al.

    Scholarly publishing in the Internet age: a citation analysis of computer science literature

    Information Processing and Management

    (2001)
  • T.C. Almind et al.

    Informetric analyses on the World Wide Web: Methodological approaches to “Webometrics”

    Journal of Documentation

    (1997)
  • J. Bar-Ilan

    Data collection methods on the Web for informetric purposes—A review and analysis

    Scientometrics

    (2001)
  • T. Berners-Lee et al.

    Weaving the web

    (1999)
  • T. Berners-Lee et al.

    The semantic web

    Scientific American

    (2001)
  • C.L. Borgman et al.

    Scholarly communication and bibliometrics

  • Bosak, J. (1997). XML, Java and the future of the Web. Retrieved April 8, 2005, Available from...
  • J. Bosak et al.

    XML and the second-generation Web

    Scientific American

    (1999)
  • L. Chan

    Supporting and enhancing scholarship in the digital age: the role of open-access institutional repositories

    Canadian Journal of Communication

    (2004)
  • Clever Project. (1999). Hypersearching the Web. Retrieved 2000, Available from...
  • B. Cronin

    Bibliometrics and beyond: some thoughts on Web-based citation analysis

    Journal of Information Science

    (2001)
  • B. Cronin et al.

    Invoked on the Web

    Journal of the American Society for Information Science

    (1998)
  • Crow, R. (2002). The case for institutional repositories: A SPARC position paper. Retrieved February 8, 2005, from The...
  • L. Egghe

    New informetric aspects of the Internet: some reflections, many problems

    Journal of Information Science

    (2000)
  • L. Egghe et al.
  • S.P. Harter

    Psychological relevance and information science

    Journal of the American Society for Information Science

    (1992)
  • Harter, S. P., & Kim, H. J. (1996). Electronic journals and scholarly communication: A citation and reference study. In...
  • Institute for Scientific Information (2004a). The impact of open access journals: A citation study from Thomson ISI....
  • Institute for Scientific Information (2004b). The ISI database: The journal selection process. Retrieved May 30, 2004,...
  • Cited by (34)

    • The distribution of Web citations

      2012, Information Processing and Management
      Citation Excerpt :

      The rapid development of computer and network technology has resulted in the transformation of the traditional scholarly communication system, which has been in place for more than three centuries, by the Internet (Zhao, 2005).

    • Infgraph: Influential Researcher and Cited Research Analysis Using Citation Network

      2022, International Journal of Decision Support System Technology
    • IMUW-APP: An instrument for measuring the usability of web applications

      2021, Indonesian Journal of Electrical Engineering and Computer Science
    • Becoming Metric-Wise: A Bibliometric Guide for Researchers

      2018, Becoming Metric-Wise: A Bibliometric Guide for Researchers
    View all citing articles on Scopus

    Dangzhi Zhao is an Assistant Professor at the School of Library and Information Studies of the University of Alberta, Canada. Her research interests are in the areas of information systems, scholarly communication, and information technology. Recent research concentrates on the comparison of scholarly communication between the Web and the print world.

    View full text