Home » Articles posted by Ciara Roche
Author Archives: Ciara Roche
Advantages and Disadvantages of Research Metrics used to Evaluate a Researcher’s Impact or Influence
Background:
Each year, in Duke University’s Division of Physical Therapy, I teach a class on research methodology. One of the topics we discuss in class involves ways to measure research impact among physical therapists’ (and other professions’) researchers. The discussion is complimentary to those that occur during the Appointment, Promotion and Tenure (AP&T) committee in the Department of Orthopaedics, of which I am a committee member. By definition, research impact metrics are quantitative tools used to assess the influence and productivity of researchers, to give some understanding who are leaders in their fields. Without fail, in the class (each year), there is some debate on the best methods. This blog will discuss four of the most common methods and will evaluate their advantages and disadvantages. The order presented does not imply superiority and these methods are not transferable with evaluating the impact of a single journal publication.
A researcher’s h-index can be found on platforms like Google Scholar, Web of Science, and Scopus, but the values often differ, since each platform uses different evaluation methods for determining a “citation”. In most cases, Google Scholar includes a wider range of publications, such as preprints and less traditional sources such as textbooks (which is why researchers often have much higher h-indexes on Google Scholar), whereas Web of Science and Scopus focus on more curated, peer-reviewed journals (no textbooks), creating discrepancies in the calculated h-index. Whereas an h-index may vary markedly across professionals, Hirsch suggested that an h-index of 3 to 5 can be set as standard for assistant professor, 8 to 12 for associate professor and h-index of 15 to 20 is a good standard for appointment to full professor [1].
Advantages:
- Balances productivity and impact.
- Simple and widely recognized.
- Useful for comparing researchers in the same field.
Disadvantages:
- Does not adjust for highly cited papers, beyond the core publications.
- Biased toward senior researchers with long careers.
- Ignores the contribution of co-authors or positioning of authorship.
- Field-dependent (e.g., higher citation rates in some fields).
M-Index (m-Quotient): Hirsch recognized the limitations of the h-index and its bias toward senior researchers who have had multiple years to acquire citations. He subsequently made an adjustment by taking the h-index and dividing it by the time (in years) since researcher’s initial publication. Thus, if a researcher has an H-index of 20, and their first publication occurred 25 years ago, their M-Index is 0.8 (20/25). Hirsch suggested that an M-index of 1.0 is Very good, 2.0 is Outstanding and 3.0 is Exceptional [1].
Advantages:
- Normalizes the h-index for career length [2].
- Useful for comparing early-career researchers.
Disadvantages:
- Still inherits many of the limitations of the h-index (e.g., less meaningful for researchers with short careers.
- It’s more difficult to understand than an h-index.
Field-Weighted Citation Impact (FWCI): Field-Weighted Citation Impact (FWCI) is a metric used to measure the citation impact of a researcher’s work compared to the expected citation rate in their specific field [3]. It is calculated by taking the total number of citations received by a researcher’s publications, and dividing the average number of citations that similar publications in the same field, publication type, and year that they are expected to receive. The FWCI is a ratio calculation, and includes a ratio of the actual citation count to the expected citation rate. For example, if a researcher’s publications have received 50 citations, but the expected citation rate for similar publications is 25, the FWCI would be 2.0; this means the researcher has been cited twice as much as expected. A FWCI of 1 indicates that the researcher has been cited exactly as expected, whereas a FWCI greater than 1 indicates higher-than-expected citation impact, and a FWCI less than 1 indicates lower-than-expected citation impact.
Advantages:
- Accounts for field-specific citation practices.
- Normalizes impact across disciplines.
Disadvantages:
- Requires access to field-specific data.
- Calculators are often for articles, not researchers.
- Less intuitive for non-specialists.
NIH Relative Citation Ratio (RCR): The National Institutes of Health (NIH) Relative Citation Rate (RCR) is a metric developed by the NIH Office of Portfolio Analysis to measure the scientific influence of a research paper [4]. RCR is first calculated by normalizing the citation rate of all papers to its field and publication years. The process involves estimating the citation rate of the researchers’ field using its co-citation network (also known as Field Weighted Citation Impact, see above). Secondly, the expected citation rate is calculated, by evaluating the rate for NIH-funded papers in the same field and publication years. The RCR compares the researcher’s papers’ citation rates to the expected citation rates. A researcher with an RCR of 1.0 has received citations at the same rate as the median NIH-funded researcher in its field. Values above 1.0 indicates that the researcher is cited as a rate above the median NIH-funded researchers. An RCR of 1.5, would mean the researcher is cited 1.5 times more frequently than the median NIH funded researcher. An RCR of 2.3 means they are cited 2.3 times more frequently, etc. Values below 1.0 suggest they are cited less frequently than the median researchers.
Advantages:
- Normalizes for field and time.
- Useful for comparing researchers within disciplines.
- Website is easy to navigate and is free https://icite.od.nih.gov/analysis
Disadvantages:
- The website is complicated and takes a little time to learn how to navigate.
- It is less known than other methods such as the h-index.
Summary: Each metric has its strengths and weaknesses, and no single metric can fully capture the impact of every single researcher. All methods push the importance of citations, although two (FWCI and RCR) compare these to others in similar fields. In our AP&T meetings, we consider a combination of metrics as the best approach, tailored to the specific context (e.g., field, career stage, or type of impact) and we also look at number of first author or senior author papers; there are researcher impact metrics that do this as well but they are less often used.
Disclaimer: Both Deepseek and Microsoft copilot were used to assist in this blog.
References
- Shah FA, Jawaid SA. The h-index: An Indicator of Research and Publication Output. Pak J Med Sci. 2023 Mar-Apr;39(2):315-316.
- Kurian C, Kurian E, Orhurhu V, Korn E, Salisu-Orhurhu M, Mueller A, Houle T, Shen S. Evaluating factors impacting National Institutes of Health funding in pain medicine. Reg Anesth Pain Med. 2025 Jan 7:rapm-2024-106132.
- Aggarwal M, Hutchison B, Katz A, Wong ST, Marshall EG, Slade S. Assessing the impact of Canadian primary care research and researchers: Citation analysis. Can Fam Physician. 2024 May;70(5):329-341.
- Vought V, Vought R, Herzog A, Mothy D, Shukla J, Crane AB, Khouri AS. Evaluating Research Activity and NIH-Funding Among Academic Ophthalmologists Using Relative Citation Ratio. Semin Ophthalmol. 2025 Jan;40(1):39-43.
“It’s Not You, It’s Us…”: Heterogeneity of Treatment Effects as a Challenge to Effectiveness Trials.
Background:
Comparative effectiveness studies are the cornerstone of medicine and health sciences research. They have a goal of finding ‘the best’ treatment for each associated condition. In comparative effectiveness studies, statistical models are able to provide ‘average’ treatment effects, which are often used to establish standardized mean difference between the interventions; however, it is clear across interventional studies that the ‘average’ effect is not to be consistently expected. Whereas interventional design focuses on central tendency (mean or median of the population), one may more importantly consider the dispersion of data around that point.
Heterogeneity of treatment effects (HTE) refers to the variation in how different individuals respond to the same treatment or intervention. HTE are often represented by standard deviations which are impacted by outliers, demonstrating individuals who respond ‘differently’ to the intervention than the ‘average’ results. There are a number of ways in which HTE can be analyzed or managed in secondary analyses including subgroup analysis of covariates and specific statistical methods to identify heterogeneity [1]. HTE is critically important in interpretation of results in interventional studies; however, they are often poorly reported [1]. Two factors must be considered when understanding HTE in interventional effectiveness trials: 1) what factors contribute to HTE, 2) the limitations and challenges in attempting to control HTE.
Factors contributing to heterogeneity of treatment effects:
A significant proportion of treatment effect may be attributed to non-specific effects- meaning those effects considered outside of the specific treatment mechanisms of the intervention itself [2]. Furthermore, individuals presented with the same stimuli display different responses, which has been proposed to be related to their personal physiological adaptability to pain [3-4]. This has been projected to be one of the reasons orthopaedic manual therapy elicits variable treatment responses (often displayed as ‘responders’ and ‘non-responders’ to treatments) although often reported as efficacious to the ‘average’ individual [5]. Treatment effect should therefore be viewed as the sum of a multitude of contributors that interact with one another and are known to influence outcomes. (Fig1)
Fig1: Factors contributing to treatment effectiveness in interventional trials.
Limitations and challenges in attempting to control heterogeneity of treatment effects:
At first glance, the easy answer is to control HTE through attempting standardization of: 1) patients by narrowing inclusion criteria; 2) environment by using a lab-based design with high levels of control; and 3) intervention through a prescriptive design. This concept is challenged by current limited understanding of the multitude of factors (patient, provider, environmental, systemic), which influence the outcome and therefore must be considered [6]. Increased standardization also results in a reduction in external validity to the clinical setting as patients, environment, and interventions are highly dynamic and pragmatic. While highly controlled designs may be theorized to reduce HTE, it would be impractical to think they would markedly narrow to a homogenous treatment effect. Even so, if the study design is controlled to capture more of the ‘average’ responders and omit the outliers, does it improve meaningfulness of the results any more than what is reported with HTE? In essence, omitting the outliers from research does not reduce the chance that those patients similar to the outliers will walk into the clinic. (Fig 2).
Fig 2: The clinical setting is unable to narrow inclusion criteria as is possible in clinical trial design therefore limiting the external validity of highly controlled clinical trials.
Summary:
An expectation of HTE should be the norm within interventional studies. Comparative effectiveness studies should look to identify which intervention works better for the ‘average’ individual, but should use secondary analysis measures to establish if certain subgroups of individuals (clinical phenotypes) respond differently to the intervention based on a cluster of patient, environmental, or interventional factors [7]. Effect modeling procedures have been proposed to appreciate interaction between intervention and baseline covariates with the goal of developing absolute treatment effects (those anticipated within a certain population/subgroup) with the goal of improving translational clinical value [8]. During study design, investigators should appreciate the benefits and risks of increasing/decreasing control as well as different prospective versus post hoc analysis techniques and should develop studies appropriately based on their purpose.
Patient management, including the use of manual therapies [9], has shifted towards person-centered models as research has established the individuality of experiences in healthcare, therefore research should look to work within these constructs as well appreciating that the days of searching for the ‘holy grail’ of interventions is behind us, but rather a focus should be on identifying which intervention is best suited for which patient, at what time.
References:
- Gabler NB, Duan N, Liao D, Elmore JG, Ganiats TG, Kravitz RL. Dealing with heterogeneity of treatment effects: is the literature up to the challenge? Trials. 2009;10(1):43. doi:10.1186/1745-6215-10-43
- Ezzatvar Y, Dueñas L, Balasch-Bernat M, Lluch-Girbés E, Rossettini G. Which Portion of Physiotherapy Treatments’ Effect Is Not Attributable to the Specific Effects in People with Musculoskeletal Pain? A Meta-Analysis of Randomized Placebo-Controlled Trials. Journal of Orthopaedic & Sports Physical Therapy. 2024;54(6):391-399. doi:10.2519/jospt.2024.12126
- Wan DWL, Arendt-Nielsen L, Wang K, Xue CC, Wang Y, Zheng Z. Pain Adaptability in Individuals With Chronic Musculoskeletal Pain Is Not Associated With Conditioned Pain Modulation. The Journal of Pain. 2018;19(8):897-909. doi:10.1016/j.jpain.2018.03.002
- Zheng Z, Wang K, Yao D, Xue CCL, Arendt-Nielsen L. Adaptability to pain is associated with potency of local pain inhibition, but not conditioned pain modulation: A healthy human study. Pain. 2014;155(5):968-976. doi:10.1016/j.pain.2014.01.024
- Keter D, Cook C, Learman K, Griswold D. Time to evolve: the applicability of pain phenotyping in manual therapy. J Man Manip Ther. 2022;30(2):61-67. doi:10.1080/10669817.2022.2052560”
- Keter D, Loghmani MT, Rossettini G, Esteves JE, Cook CE. Context is Complex: Challenges and opportunities dealing with contextual factors in manual therapy mechanisms research. International Journal of Osteopathic Medicine. Published online January 2, 2025. doi:10.1016/j.ijosm.2025.100750
- Edwards RR, Dworkin RH, Turk DC, et al. Patient phenotyping in clinical trials of chronic pain treatments: IMMPACT recommendations. Pain. 2016;157(9):1851-1871. doi:10.1097/j.pain.0000000000000602
- Kent DM, Paulus JK, Van Klaveren D, et al. The Predictive Approaches to Treatment effect Heterogeneity (PATH) Statement. Ann Intern Med. 2020;172(1):35. doi:10.7326/M18-3667
- Keter D, Hutting N, Vogsland R, Cook CE. Integrating Person-Centered Concepts and Modern Manual Therapy. JOSPT Open. 2023;2(1):60-70. doi:10.2519/josptopen.2023.0812
