Indicators Used in Scientific Research to Evaluate the Quality of a Psychological Counseling System

Last updated: 2026-01-25

What “quality” means in counseling research

In scientific research, the “quality” of a psychological counseling system is rarely treated as a single score. Instead, studies typically evaluate quality across multiple dimensions—most commonly:

• Outcomes (effectiveness): do clients improve (and do gains persist)?
• Process quality: was counseling delivered well (e.g., therapeutic relationship, fidelity, engagement)?
• Safety: does the system avoid harm, detect deterioration, and handle risk (e.g., suicidality) appropriately?
• Privacy/confidentiality & security: does the system protect sensitive client data and communications?
• Implementation/service quality: can it be delivered at scale (reach, adoption, feasibility), with equitable access?
• For digital systems: usability, technical performance, and transparent reporting of intervention features/usage.

A common way to organize indicators is the structure–process–outcome framework (often called the Donabedian model). It distinguishes:

• Structure: the setting and system capabilities (staffing, governance, technology, policies)
• Process: what is done in care (assessment, counseling interactions, follow-up, safety monitoring)
• Outcomes: what changes for clients (symptoms, functioning, well-being, harms)

This model is widely used in quality-of-care research and transfers cleanly to psychological counseling systems (including telehealth and digital delivery).¹

---

A practical taxonomy of quality indicators (with examples)

The table below lists indicator categories commonly used in research, along with widely used measures and operationalizations.

Quality domain	What researchers measure (“indicators”)	Common instruments / operationalizations (examples)
Effectiveness (clinical outcomes)	Symptom severity change, response/remission, reliable/clinically significant change, maintenance at follow-up	Depression: PHQ-92 • Anxiety: GAD-73 • Global distress/psychological functioning: CORE-OM4 • Analysis often includes clinical significance / reliable change concepts5
Client functioning & well-being	Functioning, quality of life, well-being, goal attainment	Examples commonly used across mental health research: disability/function scales, QoL scales, well-being indices, goal attainment (often individualized)
Process quality (relationship & session quality)	Therapeutic alliance; session-level quality; engagement	Working Alliance Inventory (WAI) (and short forms) for alliance6 • Attendance/retention; number of sessions completed; drop-out rates
Client experience	Satisfaction with services; perceived helpfulness	Client Satisfaction Questionnaire (CSQ-8) (and related versions)7
Safety (harms & deterioration)	Adverse events (AEs/SAEs), symptom worsening/deterioration rates, negative/unwanted effects, crisis events	Reviews of psychotherapy trials evaluate AE reporting practices and harms monitoring8 • Deterioration rate meta-analyses operationalize symptom worsening (often via reliable change)9 • Negative Effects Questionnaire (NEQ) assesses unwanted effects of psychological treatment10
Risk management	Detection/triage for suicidality/self-harm; emergency response procedures; escalation pathways	Risk items may be included in outcome measures (e.g., CORE-OM includes “risk” content)4 and in telepsychology practice guidance that addresses emergencies and safety planning11
Privacy/confidentiality	Informed consent for data use; confidentiality practices; data minimization; access controls	Telehealth privacy guidance emphasizes assessing risks for remote communications and protecting PHI (e.g., authentication, secure transmission, storage/recording practices)12
Security	Technical security controls (encryption, authentication, audit logs), threat/risk assessment, incident response	Digital health evaluation frameworks include data protection & security as explicit evidence/assurance requirements (e.g., NICE evidence standards and DTAC requirements)13
Usability (digital counseling systems)	Ease of use, learnability, user satisfaction with interface	System Usability Scale (SUS) is a standard, widely used usability questionnaire14
App/intervention quality (digital mental health)	Engagement, functionality, aesthetics, information quality, subjective quality	Mobile App Rating Scale (MARS) is widely used to rate health app quality in research15
Digital intervention quality incl. privacy	Quality of a digital mental health intervention across multiple domains, including privacy/security considerations	ENLIGHT includes a multidimensional evaluation approach for eHealth/digital interventions; it explicitly includes privacy/security considerations in its assessment approach16
Implementation & service delivery quality	Acceptability, feasibility, appropriateness, fidelity, penetration, sustainability (implementation outcomes)	Implementation outcomes taxonomy commonly used in implementation science17
Population impact / scalability	Reach, adoption, implementation quality, maintenance	RE-AIM framework is often used to report impact beyond efficacy (reach/adoption/implementation/maintenance)18
Research reporting quality (digital/AI)	Transparent reporting of intervention features, delivery mode, usage, and (when AI is used) AI-specific trial/reporting elements	CONSORT-EHEALTH provides reporting guidance for web-based/digital interventions (incl. usage metrics, intervention description)19 • CONSORT-AI and SPIRIT-AI extend trial reporting guidance for AI interventions2021

---

Key indicator domains in more detail

1) Effectiveness (clinical outcomes)

In counseling and psychotherapy research, “effectiveness” is commonly operationalized as change in validated clinical outcomes (pre–post and follow-up), often reported as:

• Mean symptom change on a validated scale (e.g., PHQ-9, GAD-7)²³
• Response / remission rates (definitions vary by condition and measure)
• Reliable change / clinically significant change to distinguish meaningful improvement from measurement noise⁵
• Global outcome measures intended to capture broader distress/well-being (e.g., CORE-OM)⁴

Why “reliable change” shows up so often: counseling outcomes can vary widely by person, and researchers want methods that identify clinically meaningful individual change rather than only group averages.⁵

2) Process quality (how counseling is delivered)

Effectiveness outcomes don’t fully describe quality; research often measures “how” care is delivered, including:

• Therapeutic alliance (relationship quality between client and counselor), commonly measured with the Working Alliance Inventory (WAI) or its short forms.⁶
• Engagement/adherence: number of sessions attended, completion rates, time-in-treatment, dropout (important for both in-person and digital systems).
• Client experience/satisfaction, often measured by the CSQ-8.⁷

For digital counseling systems, process indicators often expand to include usage patterns, because outcomes depend heavily on whether people actually use the tool as intended. Reporting guidelines like CONSORT-EHEALTH highlight the need to describe the intervention clearly and report usage/engagement data.¹⁹

3) Safety (harm prevention, monitoring, and response)

Research uses multiple approaches to safety:

• Adverse events (AEs/SAEs): systematic reviews note that psychotherapy trials vary widely in whether and how they track and report adverse events, which has led to calls for more consistent harms monitoring.⁸
• Deterioration/worsening: deterioration rates are often estimated using symptom measures (sometimes applying reliable change concepts), and meta-analyses show that deterioration is a real phenomenon that should be monitored.⁹
• Negative/unwanted effects: the Negative Effects Questionnaire (NEQ) is a commonly cited tool for measuring unwanted effects of psychological treatment.¹⁰

For counseling systems (especially digital), safety indicators also include risk management capability: detecting crisis signals, escalation pathways, and emergency procedures. Telepsychology practice guidance explicitly includes emergencies/safety planning as part of responsible practice.¹¹

4) Privacy, confidentiality, and security

Because counseling involves highly sensitive information, research and evaluation frameworks treat privacy and security as core quality dimensions—especially for telehealth and digital counseling.

Commonly used privacy/security indicators include:

• Informed consent processes covering remote/digital delivery and data practices (what is collected, who can access it, retention, etc.)
• Secure communications: encryption in transit; authentication; controls to reduce interception risks
• Secure storage and access controls: role-based access, audit logs, least-privilege access
• Data governance: data minimization, retention schedules, deletion/disposal processes, breach response

Even when studies don’t run full security audits, practical privacy guidance for telehealth highlights key technical and procedural safeguards (e.g., evaluating risks of the telehealth setup, securing PHI, handling recordings and access).¹²

On the policy/evidence side, digital health standards frameworks (e.g., NICE evidence standards and DTAC requirements) explicitly require evidence or assurance around data protection, clinical safety, and technical security.¹³

5) Implementation quality (can it work in the real world?)

High-quality systems are not only effective in controlled trials; they also need to be deliverable, acceptable, and sustainable.

Two frameworks widely used in research:

• Implementation outcomes taxonomy (acceptability, feasibility, appropriateness, fidelity, penetration, sustainability, etc.)—often used when evaluating new service models and digital interventions.¹⁷
• RE-AIM (reach, effectiveness, adoption, implementation, maintenance)—often used to evaluate public health impact and scalability beyond efficacy.¹⁸

---

Suggested “minimum indicator set” (commonly seen in studies)

If you want a practical checklist of indicators that map to common research practice, many studies end up including something like:

1. Primary clinical outcome (e.g., PHQ-9 for depression or GAD-7 for anxiety)²³
2. Broad/global outcome (e.g., CORE-OM)⁴
3. Process indicator: therapeutic alliance (WAI short form) and/or engagement (retention/completion)⁶
4. Client experience: CSQ-8 satisfaction⁷
5. Safety: NEQ (negative effects) + deterioration monitoring using symptom measures/reliable change concepts¹⁰⁵
6. For digital systems: SUS (usability) + MARS/ENLIGHT (systematic quality rating)¹⁴¹⁵¹⁶
7. Implementation: selected implementation outcomes (acceptability/feasibility) and/or RE-AIM reach/adoption/maintenance¹⁷¹⁸
8. Privacy/security assurance: documented safeguards and/or assessment against a recognized framework (e.g., DTAC-aligned evidence expectations)¹³

The exact mix should be tailored to the setting (university counseling center vs. clinic vs. digital app), population (youth vs. adults), and risk profile.

---

References

Footnotes

1. Donabedian A. “The quality of care. How can it be assessed?” JAMA (1988). PubMed: https://pubmed.ncbi.nlm.nih.gov/3045356/ ↩

2. Kroenke K, Spitzer RL, Williams JBW. “The PHQ-9: validity of a brief depression severity measure.” Journal of General Internal Medicine (2001). PubMed: https://pubmed.ncbi.nlm.nih.gov/11556941/ ↩ ↩² ↩³

3. Spitzer RL, Kroenke K, Williams JBW, Löwe B. “A brief measure for assessing generalized anxiety disorder: the GAD-7.” Archives of Internal Medicine (2006). PubMed: https://pubmed.ncbi.nlm.nih.gov/16717171/ ↩ ↩² ↩³

4. Evans C, Connell J, Barkham M, et al. “Towards a standardised brief outcome measure: psychometric properties and utility of the CORE-OM.” British Journal of Psychiatry (2002). PubMed: https://pubmed.ncbi.nlm.nih.gov/11925273/ ↩ ↩² ↩³ ↩⁴

5. Jacobson NS, Truax P. “Clinical significance: a statistical approach to defining meaningful change in psychotherapy research.” Journal of Consulting and Clinical Psychology (1991). PubMed: https://pubmed.ncbi.nlm.nih.gov/2030197/ ↩ ↩² ↩³ ↩⁴

6. Horvath AO, Greenberg LS. “Development and validation of the Working Alliance Inventory.” Journal of Counseling Psychology (1989). PubMed: https://pubmed.ncbi.nlm.nih.gov/24885797/ ↩ ↩² ↩³

7. Attkisson CC, Zwick R. “The Client Satisfaction Questionnaire. Psychometric properties and correlations with service utilization and psychotherapy outcome.” Evaluation and Program Planning (1982). PubMed: https://pubmed.ncbi.nlm.nih.gov/10259963/ ↩ ↩² ↩³

8. Duggan C, Parry G, McMurran M, Davidson K, Dennis J. “The recording of adverse events from psychological treatments in clinical trials: evidence from a review of NIHR-funded trials.” Trials (2014). https://trialsjournal.biomedcentral.com/articles/10.1186/1745-6215-15-335 ↩ ↩²

9. Fernández-Álvarez J, Rozental A, Carlbring P. “Deterioration rates in psychotherapy: A systematic review and meta-analysis.” Clinical Psychology Review (2024). PubMed: https://pubmed.ncbi.nlm.nih.gov/38560742/ ↩ ↩²

10. Rozental A, Kottorp A, Boettcher J, Andersson G, Carlbring P. “Negative Effects Questionnaire (NEQ): psychometric properties of an instrument for assessing negative effects in psychological treatments.” Behavioural and Cognitive Psychotherapy (2019). PubMed: https://pubmed.ncbi.nlm.nih.gov/30398856/ ↩ ↩² ↩³

11. Hollis RH, Nowakowski S, Lattie EG, et al. “Guidelines for the practice of telepsychology: a systematic review and compendium of 2024 professional guidelines.” Journal of Telemedicine and Telecare (2025). PubMed: https://pubmed.ncbi.nlm.nih.gov/39639864/ ↩ ↩²

12. U.S. HHS. “HIPAA and Audio-only Telehealth.” (updated guidance). https://www.hhs.gov/hipaa/for-professionals/privacy/guidance/audio-only-telehealth/index.html ↩ ↩²

13. NICE. “Evidence standards framework for digital health technologies.” (and DTAC-related requirements). https://www.nice.org.uk/corporate/ecd7 ↩ ↩² ↩³

14. Brooke J. “SUS: A ‘Quick and Dirty’ Usability Scale.” (1986/1996). Bibliographic record: https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html ↩ ↩²

15. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. “Mobile App Rating Scale: a new tool for assessing the quality of health mobile apps.” JMIR mHealth and uHealth (2015). https://mhealth.jmir.org/2015/1/e27/ ↩ ↩²

16. Baumel A, Faber K, Mathur N, Kane JM, Muench F. “Enlight: A comprehensive quality and therapeutic potential evaluation tool for mobile and web-based eHealth interventions.” Journal of Medical Internet Research (2017). https://www.jmir.org/2017/3/e82/ ↩ ↩²

17. Proctor E, Silmere H, Raghavan R, et al. “Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda.” Administration and Policy in Mental Health and Mental Health Services Research (2011). PubMed: https://pubmed.ncbi.nlm.nih.gov/20957426/ ↩ ↩² ↩³

18. Glasgow RE, Vogt TM, Boles SM. “Evaluating the public health impact of health promotion interventions: the RE-AIM framework.” American Journal of Public Health (1999). PubMed: https://pubmed.ncbi.nlm.nih.gov/10191883/ ↩ ↩² ↩³

19. Eysenbach G; CONSORT-EHEALTH Group. “CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions.” Journal of Medical Internet Research (2011). https://www.jmir.org/2011/4/e126/ ↩ ↩²

20. Liu X, Rivera SC, Moher D, et al. “Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension.” Nature Medicine (2020). https://pubmed.ncbi.nlm.nih.gov/32246076/ ↩

21. Rivera SC, Liu X, Chan A-W, et al. “SPIRIT-AI and CONSORT-AI: new guidelines for clinical trials of artificial intelligence interventions.” BMJ (2020). https://www.bmj.com/content/370/bmj.m3210 ↩