Blog

The misuse of citation statistics

"While it is incorrect to say that the impact factor gives no information about individual papers in a journal, the information is surprisingly vague and can be dramatically misleading" says a recent report published by the Joint Committee on Quantitative Assessment of Research {1}.

We loved this highly informative report detailing how statistics such as the impact factor and h-index are misused, not well understood and not well studied. Robert Adler and his colleagues aim to point out the limitations of using citation statistics, as well as how to make better use of them.

This superb article also calls into question the accuracy, objectivity and simplicity of using such statistics as the sole measure of research quality:

"We do not dismiss citation statistics as a tool for assessing the quality of research-citation data and statistics can provide some valuable information ... But citation data provide only a limited and incomplete view of research quality, and the statistics derived from citation data are sometimes poorly understood and misused. Research is too important to measure its value with only a single coarse tool."

We agree - that's why the F1000 Factor is so useful. It provides an alternative measure to the impact factor and is based on the opinion of eminent members of the medical community, irrespective of what journal the article comes from.

The Adler et al paper is receiving a lot of attention and is yet more evidence adding to the ever-growing base that our reliance on, and trust of, such statistics may need to be re-assessed.

Read more about the F1000 Factor here.
 

References:

1. http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf 

Save to del.icio.us | Email this

Posted by Sean Porazinski on Wednesday Jul 02, 2008, Category: Impact factor versus... | Permalink

Worshiping False Idols: The Impact Factor Dilemma

"...although the journal impact factor was born innocently enough ... Garfield's impact factor is now being used by others in ways that threaten to destroy scientific inquiry as we know it" says Roger Brumback in a recent article published in the Journal of Child Neurology {1}.

We think this is yet another interesting piece of literature highlighting the scientific community's dissatisfaction with the continued use of the impact factor to judge the quality of scientists and their work, as well as rank the institutions to which they belong.

The paper looks at the humble beginnings of the impact factor, how it has developed over the last 50 years or so and how its current use is affecting the scientific community. Brumback goes on to write, "Now would seem to be the appropriate time for the academic community to demand valid metrics to assess published scientific material".

He also states, "At a time when both the scientific community and the general public are lamenting the lack of transparency in science (particularly in relation to industry-supported research), it is unconscionable for academicians to deliver their careers into the hands of a for-profit company like Thomson Scientific that secretively derives a number to pigeonhole their research efforts".

The fact that this debate is being discussed in such a lively manner in some of the 'smaller impact' journals speaks volumes for its wide ranging appeal and highlights the need for a satisfactory alternative/compromise to be developed as soon as possible.

References:
{1} Brumback, Journal of Child Neurology 2008, 23:365-367. "Worshiping False Idols: The Impact Factor Dilemma" [PMID:18401031].

Save to del.icio.us | Email this

Posted by Sean Porazinski on Friday Jun 06, 2008, Category: Impact factor versus... | Permalink

The wonderful world of informetrics

Informetrics is an area of information science which is thriving. It provides us with alternative qualitative and quantitative means of measuring scientific output.

Informetrics (a.k.a. infometrics) is the broad term comprising all '-metrics' studies related to information science, including bibliometrics (e.g. bibliographies, libraries), scientometrics (e.g. science policy, citation analysis, research evaluation), webometrics (metrics of the web, the Internet or other social networks, such as citation or collaboration networks), etc. {1}.

An interesting paper within this field, by Judit Bar-Ilan {2}, highlights some of the key literature covering the main issues and difficulties that have been facing informetrics since the beginning of the 21st century. In his review, Bar-Ilan covers developments regarding the advantages and disadvantages of the h-index, citation analyses, debate over the impact factor and other multiple indicators and measures currently used to give a bigger picture of scientific output and quality across the different mediums.

It's great to see these areas being explored in more detail, as these different measures of scientific output provide a more objective means of analysing the ever-increasing amount of medical literature.

References:
1.    Egghe, Information Processing & Management 2005, 41:1311-1316. Expansion of the field of informetrics: origins and consequences.
2.    Bar-Ilan, Journal of Informetrics 2008, 2:1-52. Informetrics at the beginning of the 21st century ? A review.

Save to del.icio.us | Email this

Posted by Sean Porazinski on Thursday May 08, 2008, Category: Impact factor versus... | Permalink

Predicting the impact of a clinical article

A recent article published in the BMJ {1} highlights the importance of being able to predict the relevance and impact of a clinical or evidence-based medicine article shortly after its publication. The study asserts that a definitive method is needed to quickly determine an article's importance other than citation rates, as these are not available until months after publication.

The paper continues by stating that knowing the clinical impact of an article early on will allow it to be highlighted and distributed appropriately to the correct readership. This in turn could improve patient care through effective knowledge translation and continuing medical education, as well as allowing research within that area to advance. The article also proposes an interesting method to predict what the citation rate (and thus popularity) of a paper will be two years down the line.

Here at F1000 Medicine, we are avid believers in the necessity of identifying high impact articles as soon as possible. That's why we have over 2,400 experts worldwide who regularly send us evaluations of new (as well as older) articles that they deem add knowledge to an area of medicine. Thus, we are creating an invaluable online database for clinicians, physicians, researchers, nurses and medical students.

References:
1.    Lokker et al. Prediction of citation counts for clinical articles at two years using data available within three weeks of publication: retrospective cohort study. BMJ February 2008, Epub ahead of print [PMID:18292132].

Save to del.icio.us | Email this

Posted by Sean Porazinski on Thursday Feb 28, 2008, Category: Impact factor versus... | Permalink

H-Value, Strike Rate Index and F1000 factor: Alternatives to the Impact Factor

The Impact Factor (IF) has been around for 50 years as a measure of the influence of a scientific journal. Over the years, the IF has been subject to much criticism as it measures only the overall quality of a journal, not the quality of the individual papers within it. Opinion suggests it may now be time to get a new perspective and look at alternatives to the IF.

One such alternative that has recently been released into the bibliometric community is the H-Value or Hirsch number. This measure was suggested by Jorge Hirsch as a way to 'quantify the cumulative impact and relevance of an individual's scientific research output' {1}. For each individual, this value takes into account many pieces of information, including the number of publications over a particular number of years, the number of citations per publication and the journal that the paper was published in.

Another proposed method of quantifying the impact of scientific research is the Strike Rate Index (SRI). This is based on the 'log relationship of the h-index and the size of the journal and allows journals across diverse fields to be compared to each other' {2}.
At F1000 Medicine, we have a unique article ranking system and rate articles based on their own merits, regardless of the journal they were published in. We use a weighted average with positive bias based on the number of evaluations an article receives, as well as the ratings of those evaluations.

The IF alone is no longer sufficient for quantifying the impact of scientific research. A much more uniform method is needed which can be applied to journals across all fields.

References:

1. Hirsch JE. An index to quantify and individual's scientific research output. Proceedings of the National Academy of Sciences 2005 102: 16569-16572.

2. Barendse W. The strike rate index: a new index for journal quality based on journal size and the h-index of citations. Biomedical Digital Libraries 2007, 4:3

Save to del.icio.us | Email this

Posted by Natalie Davall on Friday Feb 15, 2008, Category: Impact factor versus... | Permalink

The worth of citation rates

Citation rates are often used as a measure of the quality and quantity of research carried out by scientists and institutes. They can determine fund allocation for grant applications, play a part in which journals researchers decide to publish their work in and even influence where scientists choose to work in terms of organisation and country.

However, some are calling into question the use of citation rates as determining factors in the assessment of performance. This is especially pertinent since the Higher Education Funding Council for England recently announced that the assessment and funding of science-based disciplines will now be based on citation rates per paper, aggregated for each subject group at each institution {1}.

In a recent letter published in Nature {2}, the author suggests that the use of citation rates in determining funding hinders research in developing countries. It is argued that in such countries, researchers are often forced to publish their findings in national journals as a result of local legislation. Such journals rarely feature in the most-cited rankings and lack impact and readership on an international level.

A second letter, also published in the same volume of Nature {3}, goes on to say that poor practices by authors reduce the worth of citation rates. This piece highlights that citation rates can be altered by excessive citation of an author's own work, inappropriate citation of irrelevant papers, and geographical and language biases, as well as groups of authors forming 'citation coalitions'.

This is why the F1000 factor is such a useful indicator of a paper's worth - it is a weighted average with positive bias based on the number of evaluations an article receives, as well as the ratings of those evaluations.

References:
1. Research Excellence Framework: Consultation on the assessment and funding of higher education research post-2008. http://www.hefce.ac.uk/Pubs/HEFCE/2007/07_34/07_34.pdf. Accessed 29/1/08
2. Mishra DC. Citations: rankings weigh against developing nations. Nature 2008; 451: 244 [PMID: 18202623]
3. Todd PA, Ladle RJ. Citations: poor practices by authors reduce their value. Nature 2008; 451: 244 [PMID: 18202622]

Save to del.icio.us | Email this

Posted by Sean Porazinski on Thursday Jan 31, 2008, Category: Impact factor versus... | Permalink

The impact of the F1000 Factor

Impact factors are often used to assess the quality of a research article in the journal in which it appears. However, this is a highly flawed system, as not every article in a high impact journal is valuable and, more to the point, not every valuable article is published in a high-impact journal.

In an editorial published in Nature Clinical Practice Rheumatology in April earlier this year {1}, Peter Lipsky, head of our 'Rheumatology & Clinical Immunology' Faculty, writes that "The IF [impact factor] has ...morphed into an institutionalised means of ranking the quality of scientific journals and, by implication, the individual articles published within them ...and has been likened to a popularity contest". He also says " ...journals can manipulate their content to improve their IF" and "The IF is neither a surrogate for quality, nor an estimate of the influence of an article on clinical practice".

This is where F1000 Medicine steps in. As well as giving you expert opinion and clinical applicability, each article also receives a numerical rating - the F1000 Factor - irrespective of what journal it is from.

The F1000 Factor is generated individually for each article and incorporates both the rating it receives and the number of Faculty Members who have evaluated it. Over a third of the articles which are awarded the top 'Exceptional' rating by our members are expected to be from journals other than the NEJM, JAMA, Annals of Internal Medicine, the Lancet and the BMJ. Our 'Hidden Jewels' feature demonstrates this by highlighting valuable articles published only in less widely read journals.

With so many research articles being published, and so many journals to read, F1000 Medicine helps you to focus on only the best.

{1} Lipsky, Nature Clinical Practice Rheumatology 2007, 3:189 [PMID:17396104].

Save to del.icio.us | Email this

Posted by Adie Chan on Friday Dec 21, 2007, Category: Impact factor versus... | Permalink