What Makes ACP JournalWise Different?

More than 120 journals are read (see Journals Reviewed List) to identify articles with the strongest methods for clinical attention. Journals are selected based on suggestions by librarians, clinicians, editors, and editorial staff; Science Citation Index (SCI) impact factors; systematic examination of the contents of each selected journal for at least 6 months; and by ongoing yield of articles that meet basic Inclusion Criteria for assessing the quality of studies concerning the cause, course, prediction, diagnosis, prognosis, prevention, and treatment of medical disorders. Access to foreign-language journals is provided through the systematic reviews we read, especially those in the Cochrane Database of Systematic Reviews, which summarizes articles taken from over 800 journals in many languages.

More than 400 journal titles have been assessed since 1991 and, based on the number of articles meeting criteria, the top journals form the core set reviewed. Journal yield is reviewed annually and journals with very low yield (<1-2 articles/year) are dropped. New journals are added if nominated or if needed to expand the scope of disciplines covered and if they contain articles that meet our criteria based on review of at least 6 journal issues.

Research associates trained in health research methodology read all original and review articles in the journals on the Journals Reviewed List and apply the Inclusion Criteria relevant to the purpose category (i.e., therapeutics, diagnosis, prognosis, etc). Articles read include a broad range of medical problems of interest to internal medicine and its subspecialties. Articles that pass criteria are tagged for entry in the Critical Appraisal Process (CAP) system.

The process of applying criteria to identify high-quality articles yields relatively few that make it through the methods filter. For example, in 2009 of about 25,000 original and review articles published in 120 journals read, we identified 3728 articles (15%) for clinical attention.

After articles are identified as passing criteria, they are entered into the Critical Appraisal Process (CAP) system. PubMed numbers are used as unique identifiers to ensure items are entered into the process only once. Using the PubMed number, the CAP system automatically pulls the article title, authors, and journal citation.

Research associates add the Article Type (i.e., Original or Review), one or more of ten Purpose Categories (i.e., primary prevention, treatment, diagnosis, prognosis, etiology, quality improvement, economics, clinical prediction guide, differential diagnosis, and qualitative), and one or more Patient Populations. For specific purpose categories, additional data is required. For example, for therapy articles, the follow-up rate is included. Clinical disciplines/specialties are added depending on the target audience. Most articles are of interest to two or more disciplines/specialties.

Indexers add index terms to the article, which are used by various end products.

After the data in Step 3 are added, the articles are then passed to the editor stage where one or more clinicians checks the data added by the research associates and indexers.

Articles that pass the filter process (Step 2) and the checking process (Step 4) are transferred to the McMaster Online Rating of Evidence (MORE) system. Using the PubMed number, the MORE system pulls in additional article information from PubMed such as the abstract and major and minor MeSH headings.

For a detailed description of the MORE step, see the MORE website at http://hiru.mcmaster.ca/more_new.

The MORE system has a database of > 4000 physician raters. Physician raters in MORE are MDs or equivalent, have completed formal training, and are in independent clinical practice at least part time in internal medicine or its subspecialties.

MORE raters complete a profile page indicating their Patient Populations, Discipline/Specialty selections, and the number of articles they would like to rate each month.

We continue to recruit raters for all disciplines/specialties for all the MORE systems. If you would like to join us as a MORE rater, please contact us at more@mcmaster.ca (physicians), moreebn@mcmasterhkr.com (nurses), or rehab@mcmasterhkr.com(rehab specialists).

Articles in MORE are matched to rater selections and rating requests are sent to at least four raters for each discipline/specialty. For example, the article "Primary angioplasty vs. early routine post-fibrinolysis angioplasty for acute myocardial infarction with ST-segment elevation: the Gracia 2 non-inferiority, randomized, controlled trial," was sent to 4 raters each for Cardiology, Internal Medicine, and Emergency Medicine.

Ratings in MORE are on two 7-point scales (highest score, 7). The first scale, for Relevance, is for the extent to which the article was pertinent to practice in the rater’s clinical discipline/specialty. If Relevance is rated at least 3, the rater completes a second 7-point scale, on the extent to which the article’s content represented news or something that clinicians in the rater’s discipline were unlikely to know (which we labeled newsworthiness). Optionally, raters also provide up to 1000 characters of freetext comments (sample Rating Form).

When at least three ratings are received for a discipline, the totals are averaged and articles scoring ≥ 3 of 7 for both Relevance and Newsworthiness for at least one discipline/specialty are transferred to a permanent database, Premium LiteratUre Service (PLUS). Articles not scoring at least 3 of 7 for any discipline/specialty are transferred to a quarantine database.

PLUS ratings are used to know when to send alerts to ACP JournalWise users, and to select content for ACP Journal Club*

*Terms used in ACP Journal Club can be found at Glossary.