Citation analysis of orthopaedic literature; 18 major orthopaedic journals compared for Impact Factor and SCImago.

BACKGROUND
One of the disadvantages of the Impact Factor (IF) is self-citation. The SCImago Journal Rank (SJR) indicator excludes self-citations and considers the quality, rather than absolute numbers, of citations of a journal by other journals. The present study re-evaluated the influence of self-citation on the 2007 IF for 18 major orthopaedic journals and investigated the difference in ranking between IF and SJR.


METHODS
The journals were analysed for self-citation both overall and divided into a general group (n = 8) and a specialized group (n = 10). Self-cited and self-citing rates, as well as citation densities and IFs corrected for self-citation (cIF), were calculated. The rankings of the 18 journals by IF and by SJR were compared and the absolute difference between these rankings (DeltaR) was determined.


RESULTS
Specialized journals had higher self-citing rates (p = 0.01, Deltamedian = 9.50, 95%CI -19.42 to 0.42), higher self-cited rates (p = 0.0004, Deltamedian = -10.50, 95%CI -15.28 to -5.72) and greater differences between IF and cIF (p = 0.003, Deltamedian = 3.50, 95%CI -6.1 to 13.1). There was no significant correlation between self-citing rate and IF for both groups (general: r = 0.46, p = 0.27; specialized: r = 0.21, p = 0.56). When the difference in ranking between IF and SJR was compared between both groups, sub-specialist journals were ranked lower compared to their general counterparts (DeltaR: p = 0.006, Deltamedian = 2.0, 95%CI -0.39 to 4.39).


CONCLUSIONS
Citation analysis shows that specialized orthopaedic journals have specific self-citation tendencies. The correlation between self-cited rate and IF in our sample was large but, due to small sample size, not significant. The SJR excludes self-citations in its calculation and therefore enhances the underestimation in ranking of specialized journals.


Background
The IF [1] is considered the best reference utensil for evaluation of scientific journals although its limitations have already been described extensively [2][3][4][5][6][7]. Major points of criticism are the lack of quality assessment for citations [5], poor comparability between different domains of interest per journal [6] and the mainly English language in publications. A major problem with the IF for journals is self-citation, defined in Journal Citation Reports as referring to articles from the same journal [1]. Due to these limitations and the simplicity of calculating the IF, it would be relatively easy for editors to manipulate it. A journal's IF can be increased artificially by using self-citations, publishing relatively many review articles and limiting the number of articles included.
The internet search engine Google™ uses a PageRank algorithm to determine page ranking after a specific search query. Several authors have advocated a similar algorithm for the evaluation of scientific journals. Falagas et al. [5] recently described the application of a PageRank algorithm to the Scopus database to produce the SCImago Journal Rank (SJR) indicator. The SJR represents awarded prestige per article in the analysed year and is calculated using a complicated iterative formula. Self-citations do not contribute to the SJR, since a journal can receive prestige only from other journals, not from itself [8]. The SJR has several other benefits [5], such as the greater number of journals and languages included in its database and the fact that it is open-source software (free of charge).
In the present study we first aimed to evaluate the influence of self-citations on the 2007 IF for 18 major orthopaedic journals, including sub-specialist journals. These journals publish specialized articles concerning a narrow field of interest and as such are expected to receive relatively few citations from other journals. We therefore hypothesized that self-citation rates are higher for specialized journals than for the general orthopaedic literature. Secondly, we investigated ranking of orthopaedic journals with the SJR. This indicator excludes self-citations, therefore we hypothesized that the SJR ranks specialized journals comparatively lower than does the IF.

Methods
We included and analyzed 18 orthopaedic journals, making a distinction between general and specialized orthopaedic journals. Two orthopaedic clinicians participating in this study individually allocated journals to either the general or the specialized group.
The general group included the following journals: Acta Orthopaedica ( For both groups self-citation was analysed for the 2007 IF. The self-citing rate, self-cited rate and citation density, parameters known to influence the IF [2], were calculated for each of these journals as defined by the ISI Web of Science [1]. Also, all IF were corrected for the influence of self-citations (cIF) [1]; for individual journal evaluation, an absolute change of >0.5 in IF (ΔIF = IF -cIF) was considered substantial. To estimate the influence of self-citation on the IF, journals in both groups were correlated for self-citing rate and IF.
The self-citing rate [1] was calculated by dividing the number of self-cited articles in journal X in 2007 by the total amount of citations by that journal in 2007. For example, in 2007 AJSM contained 1542 citations of AJSM articles on a total of 8089 citations; the self-citing rate was 1542/8089 = 19% ( Table 1).
The ratio between the number of self-cited articles in journal X in 2007 and the total amount of citations received in 2007 for articles in journal X is the selfcited rate [1]. For example, AJSM was cited 10,711 times in total, of which 1542 by itself; the self-cited rate was 1542/10711 = 14% ( Table 1).
The citation density [1] was determined by dividing the total number of references in journal X in 2007 by the total number of articles published in that journal in 2007. For example, the 239 articles published in AJSM in 2007 contained a total of 8089 references; the citation density was 8089/239 = 34 citations per article ( Table  1).
We retrieved information regarding ranking by SJR via the SCImago journal-and country-rank website developed by the SCImago research group [8]; the journals were selected by matching international standard serial number (ISSN) found in the JCR. Since the IF ranks 6426 journals and the SJR 15,922, absolute rankings by IF and SJR are not comparable. Therefore only the 18 journals included in our analysis were mutually ranked for IF (R IF ) and SJR (R SJR ). The difference between R IF and R SJR , expressed as ΔR, was calculated to check for a possible difference between both rankings.

Statistical analysis
Statistical analysis was performed using Prism v5.00 for Windows (Graphpad Software Inc, San Diego, CA, USA). Analysing differences between general and specialized journals for the IF, the following specifics were compared using a Mann-Whitney test: self-citing rate, self-cited rate, citation density, ΔIF, R IF , R SJR and ΔR. We determined the medians, the first-and third-quartile values, and the difference between the medians (Δmedian = median general -median specialized ) and their confidence intervals. A Spearman rank correlation coefficient was computed to estimate the correlation between selfciting rates and IF, as well as between the journal rankings by IF and by SJR. For all tests, p values < 0.05 were considered significant.
Comparative Ranking: IF versus SCImago (Table 3) The rankings by IF and SJR for the 18 journals in our analysis are presented in Table 3. For the SJR, an orthopaedic ranking is possible for the sixteen journals represented in the subcategory Medicine: Orthopaedics and Sport Medicine in the SJR database, but not for BMC MD (included in the subcategory Medicine: Miscellaneous) and Spine (included in the subcategory Biochemistry, Genetics and Molecular Biology).

Key findings
Our study revealed the following: (1) Specialized journals receive proportionally more self-citations with a strong influence on the IF. (2) The SJR shows a strong relation with the IF. (3) Correcting for self-citation with the newly introduced SJR results in substantial individual changes for journal ranking. (4) Specialized journals tend to drop, whereas general journals will climb, in rank when the SJR is applied instead of IF.
Group-specific characteristics were clearly visible. Subspecialist journals have higher self-citing (p = 0.01) and self-cited rates (p = 0.0004), also expressed in a greater difference between IF and corrected IF for self-citations (p = 0.003). Despite the elaborate iterative calculation of the SJR, there exists a very strong correlation between IF and SJR ranking (general: r = 0.98, p < 0.0001) (specialized: r = 0.93, p < 0.0001). Comparing the mutual difference in ranking for SJR and IF between both groups showed an increased contrast (p = 0.003).

Strength and weaknesses of our study
Our study is strengthened by the analysis of two separate groups of orthopaedic literature: general and specialized orthopaedic journals. Hakkalamani et al. [9] excluded several journals because of a suspected increased selfcitation because of subject subspecialty. Applying standard definitions for self-citing rate, self-cited rate and citation index from the ISI Web of Science to these journals, we were able to identify specific citation patterns for specialized journals. Comparison of the Impact Factor with the newly introduced SCImago journal rank indicator elucidated whether citation analysis based on a PageRank algorithm provides improved quality assessment of orthopaedic literature over journal indicators.
A limitation of this study is the evaluation of journal citations for 2007 only, and not for other years of publication. Furthermore, the distinction made between general and specialized journals is arbitrary; however, distinguishing journal categories by domain of interest is plausible and is coherent with work published previously [9]. Therefore, we believe that applying widely accepted methods for citation analysis to our analysis groups generated reliable data.

Previous Literature
Self-citation indicators that affect the IF are not limited to the orthopaedic literature [9]. Similar relations have been demonstrated for anaesthesia [10] and radiology [11]. The high self-citation indexes found within these specialized fields of medical science prompted suggestions that self-citations should be eliminated from the calculation of the IF. However for the 2007 IF in orthopaedic literature we did not find a similar distinct influence of self-citations. The high self-citation indexes for specialized journals relative to general journals rather indicate that these journals serve a small and isolated field within the orthopaedic literature [10]. The question can be posed whether high self-citation rates within a specialized domain of interest are indications for lowquality publications or rather reflect overall high quality?   Papers published in these specialized journals are likely to have a high impact in their field of interest [6].

Implications of our study
Postma has stated that the impact of evolutionary papers published in multidisciplinary journals is substantially overestimated by their overall impact factor and, on the other hand, that the impact of papers in more specialized journals is significantly underestimated [6]. We agree that the IF has evolved for many users into an indicator of quality for articles, but one should always be aware of the fact that article citation rates determine the journal impact factor, not vice versa [3]. The differences found in our analysis of two groups of orthopaedic journals do not represent a difference in quality of articles published in these journals. They confirm that these groups publish different types of research, that is, either multidisciplinary or sub-specialized orthopaedic literature. The SJR, based on an PageRank algorithm, provides a more sophisticated alternative for the IF and eliminates the effect of self-citations, which might be desirable if self-citation is seen as a negative aspect. Our analysis showed that the SJR bears a high resemblance with the IF, but further enhances differences between general and specialized literature. As a result of the iterative calculation of the SJR, the contrast between both groups of literature will increase over time.

Conclusions
Even though there was no significant correlation between self-cited rate and IF, there is a strong relation between these both indices. Self-citation indexes tended to be higher for specialized orthopaedic journals. The SJR corrects for self-citations, which increases the gap in overall ranking between specialized and general journals, this reflects a difference in field of interest rather than quality.