Using standard studies to interpret the scores from scales for assessing the internal validity of randomized controlled trials

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Objective: A large number of tools for assessing the quality of randomized controlled trials are available; however, users have little guidance as to whether a given score represents high or low validity. The purpose of this study is to explore the use of studies identified as having high-internal validity, referred to as the standard studies, to interpret internal validity scores from studies with unknown internal validity. Methods: The standard studies were identified by locating 6 candidate studies reporting the findings of randomized controlled trials from the Journal of American Medicine Association or the New England Journal of Medicine and scoring the studies using 2 scales, the Jadad scale (high score = 5; low = 0) and an internal validity information scale (IVI; high score = 70; low = 0). The 2 studies with the highest average rank were chosen as the standard studies. To determine if the standard studies facilitate interpretation of internal validity scores, 11 randomized controlled trials of garlic for the treatment of hypercholesterolemia were scored using both the Jadad and IVI scales and differences in scores resolved by consensus. The total mean scores were compared with the standard-study mean using a 1-sample t test; the interpretation was based on the comparison. Results: The mean-internal validity score for the 2 standard studies was 4.5(90%) for the Jadad scale and 64 (91%) for the IVI scale. For the garlic studies, the Jadad scores ranged from 3 to 5 (60% to 80%), the IVI scores from 15 to 59 (21% to 84%); the mean scores were 4 (standard deviation = 0.9; 80%) and 37(53%), respectively. On the IVI scale, the garlic studies had lower internal validity (P < 0.001), a finding substantiated by the low scores for addressing withdrawals, blinding, and adherence. On the Jadad scale, the scores were comparable with the score for the standard studies (P = 0.093). Conclusions: The strategy of using the standard studies to interpret the scores for the IVI scale worked well; the comparison indicated that, overall, the garlic studies were of low-internal validity. With the Jadad scale, the strategy resulted in erroneous conclusions that the garlic studies had relatively high-internal validity; however, it did indicate that the Jadad scale is limited and may not identify studies with very low-internal validity. The results of this study suggest that scales used to assess internal validity should be tested on studies with a range of validity, so the functional characteristics of the scale can be identified.

Original languageEnglish (US)
JournalHospital Pharmacy
Volume42
Issue number11
StatePublished - Nov 2007

Fingerprint

Randomized Controlled Trials
Garlic
Medicine
New England
Hypercholesterolemia

Keywords

  • Internal validity scale
  • Jadad scale
  • Randomized controlled trials
  • Score interpretation

ASJC Scopus subject areas

  • Pharmaceutical Science

Cite this

@article{86b37ef819334411aa9db5c325b490ea,
title = "Using standard studies to interpret the scores from scales for assessing the internal validity of randomized controlled trials",
abstract = "Objective: A large number of tools for assessing the quality of randomized controlled trials are available; however, users have little guidance as to whether a given score represents high or low validity. The purpose of this study is to explore the use of studies identified as having high-internal validity, referred to as the standard studies, to interpret internal validity scores from studies with unknown internal validity. Methods: The standard studies were identified by locating 6 candidate studies reporting the findings of randomized controlled trials from the Journal of American Medicine Association or the New England Journal of Medicine and scoring the studies using 2 scales, the Jadad scale (high score = 5; low = 0) and an internal validity information scale (IVI; high score = 70; low = 0). The 2 studies with the highest average rank were chosen as the standard studies. To determine if the standard studies facilitate interpretation of internal validity scores, 11 randomized controlled trials of garlic for the treatment of hypercholesterolemia were scored using both the Jadad and IVI scales and differences in scores resolved by consensus. The total mean scores were compared with the standard-study mean using a 1-sample t test; the interpretation was based on the comparison. Results: The mean-internal validity score for the 2 standard studies was 4.5(90{\%}) for the Jadad scale and 64 (91{\%}) for the IVI scale. For the garlic studies, the Jadad scores ranged from 3 to 5 (60{\%} to 80{\%}), the IVI scores from 15 to 59 (21{\%} to 84{\%}); the mean scores were 4 (standard deviation = 0.9; 80{\%}) and 37(53{\%}), respectively. On the IVI scale, the garlic studies had lower internal validity (P < 0.001), a finding substantiated by the low scores for addressing withdrawals, blinding, and adherence. On the Jadad scale, the scores were comparable with the score for the standard studies (P = 0.093). Conclusions: The strategy of using the standard studies to interpret the scores for the IVI scale worked well; the comparison indicated that, overall, the garlic studies were of low-internal validity. With the Jadad scale, the strategy resulted in erroneous conclusions that the garlic studies had relatively high-internal validity; however, it did indicate that the Jadad scale is limited and may not identify studies with very low-internal validity. The results of this study suggest that scales used to assess internal validity should be tested on studies with a range of validity, so the functional characteristics of the scale can be identified.",
keywords = "Internal validity scale, Jadad scale, Randomized controlled trials, Score interpretation",
author = "Slack, {Marion K}",
year = "2007",
month = "11",
language = "English (US)",
volume = "42",
journal = "Hospital Pharmacy",
issn = "0018-5787",
publisher = "Facts and Comparisons",
number = "11",

}

TY - JOUR

T1 - Using standard studies to interpret the scores from scales for assessing the internal validity of randomized controlled trials

AU - Slack, Marion K

PY - 2007/11

Y1 - 2007/11

N2 - Objective: A large number of tools for assessing the quality of randomized controlled trials are available; however, users have little guidance as to whether a given score represents high or low validity. The purpose of this study is to explore the use of studies identified as having high-internal validity, referred to as the standard studies, to interpret internal validity scores from studies with unknown internal validity. Methods: The standard studies were identified by locating 6 candidate studies reporting the findings of randomized controlled trials from the Journal of American Medicine Association or the New England Journal of Medicine and scoring the studies using 2 scales, the Jadad scale (high score = 5; low = 0) and an internal validity information scale (IVI; high score = 70; low = 0). The 2 studies with the highest average rank were chosen as the standard studies. To determine if the standard studies facilitate interpretation of internal validity scores, 11 randomized controlled trials of garlic for the treatment of hypercholesterolemia were scored using both the Jadad and IVI scales and differences in scores resolved by consensus. The total mean scores were compared with the standard-study mean using a 1-sample t test; the interpretation was based on the comparison. Results: The mean-internal validity score for the 2 standard studies was 4.5(90%) for the Jadad scale and 64 (91%) for the IVI scale. For the garlic studies, the Jadad scores ranged from 3 to 5 (60% to 80%), the IVI scores from 15 to 59 (21% to 84%); the mean scores were 4 (standard deviation = 0.9; 80%) and 37(53%), respectively. On the IVI scale, the garlic studies had lower internal validity (P < 0.001), a finding substantiated by the low scores for addressing withdrawals, blinding, and adherence. On the Jadad scale, the scores were comparable with the score for the standard studies (P = 0.093). Conclusions: The strategy of using the standard studies to interpret the scores for the IVI scale worked well; the comparison indicated that, overall, the garlic studies were of low-internal validity. With the Jadad scale, the strategy resulted in erroneous conclusions that the garlic studies had relatively high-internal validity; however, it did indicate that the Jadad scale is limited and may not identify studies with very low-internal validity. The results of this study suggest that scales used to assess internal validity should be tested on studies with a range of validity, so the functional characteristics of the scale can be identified.

AB - Objective: A large number of tools for assessing the quality of randomized controlled trials are available; however, users have little guidance as to whether a given score represents high or low validity. The purpose of this study is to explore the use of studies identified as having high-internal validity, referred to as the standard studies, to interpret internal validity scores from studies with unknown internal validity. Methods: The standard studies were identified by locating 6 candidate studies reporting the findings of randomized controlled trials from the Journal of American Medicine Association or the New England Journal of Medicine and scoring the studies using 2 scales, the Jadad scale (high score = 5; low = 0) and an internal validity information scale (IVI; high score = 70; low = 0). The 2 studies with the highest average rank were chosen as the standard studies. To determine if the standard studies facilitate interpretation of internal validity scores, 11 randomized controlled trials of garlic for the treatment of hypercholesterolemia were scored using both the Jadad and IVI scales and differences in scores resolved by consensus. The total mean scores were compared with the standard-study mean using a 1-sample t test; the interpretation was based on the comparison. Results: The mean-internal validity score for the 2 standard studies was 4.5(90%) for the Jadad scale and 64 (91%) for the IVI scale. For the garlic studies, the Jadad scores ranged from 3 to 5 (60% to 80%), the IVI scores from 15 to 59 (21% to 84%); the mean scores were 4 (standard deviation = 0.9; 80%) and 37(53%), respectively. On the IVI scale, the garlic studies had lower internal validity (P < 0.001), a finding substantiated by the low scores for addressing withdrawals, blinding, and adherence. On the Jadad scale, the scores were comparable with the score for the standard studies (P = 0.093). Conclusions: The strategy of using the standard studies to interpret the scores for the IVI scale worked well; the comparison indicated that, overall, the garlic studies were of low-internal validity. With the Jadad scale, the strategy resulted in erroneous conclusions that the garlic studies had relatively high-internal validity; however, it did indicate that the Jadad scale is limited and may not identify studies with very low-internal validity. The results of this study suggest that scales used to assess internal validity should be tested on studies with a range of validity, so the functional characteristics of the scale can be identified.

KW - Internal validity scale

KW - Jadad scale

KW - Randomized controlled trials

KW - Score interpretation

UR - http://www.scopus.com/inward/record.url?scp=36348953183&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=36348953183&partnerID=8YFLogxK

M3 - Article

VL - 42

JO - Hospital Pharmacy

JF - Hospital Pharmacy

SN - 0018-5787

IS - 11

ER -