Evaluating Scholarly Publications for Tenure and Promotion: Quantitative Measures

Many measures of quality pertain to the journal as a whole, not individual articles or researchers.

What:

The average number of citations an article will get over two or five years. Citations counted are from articles indexed in Scopus.

Where:

https://scimagojr.com/

Advantages:

• Visualized data
• Several measures for each journal
• List of similar journals with a similarity rank
• Free to all, no subscription needed

Disadvantages:

• Impact factors of a journal are typically based on a few highly cited articles and are not true measures of any of the articles (either over or under measure the impact).
• Only measures one aspect of an article's impact.

What:

The average number of citations an article will get over the next 2 or 5 years. Citations counted are from articles indexed in Web of Science's citation indexes.

Where:

Advantages:

• Easy to read data
• List of journals in the category
• Indication of percent of the journal that is Open Access Gold

Disadvantages:

• Impact factors of a journal are typically based on a few highly cited articles and are not true measures of any of the articles (either over or under measure the impact).
• Only measures one aspect of an article's impact.
• Web of Science is very selective in the journals indexed so minority of publications have a Journal Impact Factor.

What:

Designed to measure the importance of publications in science. Eigenfactors use a more complex formula than Journal Impact Factors. More weight is given to citations in highly prestigious journals than other journals. Different citation patterns that occur in different disciplines are taken into account.

Where:

Eigenfactor.org

Advantages:

• Can be accessed for free.
• Includes a built-in evaluation period of five years.
• Attempts to give a more accurate representation of the merit of citations than do raw citation counts.

Disadvantages:

• Eigenfactor assigns journals to a single category so it is difficult to compare journals within a discipline.
• Some argue that Eigenfactor score isn't much different than raw citation counts.

What:

Various organizations rank journals using their own criteria.

Where:

• Web of Science, Journal Impact Factors
• SCImago
• Various publications and associations

Advantages:

• Easily understood

Disadvantages:

• Different rankings use different criteria meaning consistency across sources is lacking.

What:

The number of papers accepted for publication in relation to the number of papers submitted. It can be argued that the lower the acceptance rate, the better the journal.

Where:

Journal websites. These can be very difficult to find if a journal does not want their acceptance rate to be known.

Advantages:

• Easily understood

Disadvantages:

• Changeable - this can vary widely year to year.

There are measures that can help show the quality of an individual article or a researcher as a whole.

What:

Compares the citations for one paper against others in the same research area or field. Papers must be two years old.

Where:

https://app.dimensions.ai/discover/publication

Advantages:

• Measure for an individual work
• Claims to include data from many different sources (no details of where)
• Visualized data
• Faculty can search by their name
• Includes publications, awarded grants, clinical trials, patents, and policy papers

Disadvantages:

What:

Various databases and websites measure the number of times an item is downloaded or viewed. A number of databases also count the number of times other papers cite a particular work.

Where:

Advantages:

• Easily available in many cases
• Shows the impact of an individual work

Disadvantages:

• Google Scholar may count a citation to another work multiple times.
• Web of Science is very selective so citations counted are only in the journals indexed in the "best" journals.
• Citation counts are not available until an article has been published for a while because it takes time for an article to be used in other published works.

What:

Designed to measure the impact of a scientific researcher based on both the productivity and the impact of the scientist.

Where:

If an author creates a profile in Google Scholar, they can get their h-index based on publications found in that index.

Web of Science includes the h-index using ResearcherID and OrcidID.

Publish and perish software.

Advantages:

• Measures the impact of the entirety of a researcher's work together.
• Grows over time.

Disadvantages:

• Does not take into account the number of authors of articles - a single author is given the same value as one in twenty authors.
• The amount of a cited article that is used is not taken into consideration - one sentence is the same as a whole page.
• Self-citations (citing one's own works) increases one's h-index
• Cannot be used to compare one person to another because the length of time as a researcher.