Mental-health technologies should be evaluated not only by what they promise, but by how they operate and what their evidence establishes. MHTS addresses structural safeguards and public accountability, complementing but not replacing clinical evaluation.
Psychological Safety
Technology can influence a person's emotional state through its content, interface, interactions and recommendations. Poorly designed experiences may introduce risks, particularly when used during distress. Potential concerns include:
- Inappropriate intervention sequences
- Messages that intensify fear or shame
- Manipulative engagement mechanisms
- Unnecessary cognitive demands
- Misleading claims about support capabilities
- Inadequate access to human assistance
- Failure to communicate important limitations
For AI-based systems, risks may also arise from generated responses that are inaccurate, inappropriate or inconsistent with the user's circumstances.
Safety therefore requires attention to both the technology's intended purpose and its actual behaviour. MHTS establishes safeguards against several identifiable design risks, including behavioural surveillance, access barriers, dependency engineering and the absence of human escalation.
Conformance with these structural requirements does not establish that every interaction or intervention is clinically safe. Developers remain responsible for evaluating foreseeable risks associated with their particular technology.
Effectiveness vs Conformance
Two questions should be considered separately. They concern different forms of evidence.
-
Conformance
Does the technology satisfy the MHTS requirements?
MHTS conformance concerns the published structural and ethical requirements.
-
Effectiveness
Does the technology achieve its intended mental-health outcomes?
Effectiveness concerns whether the technology produces the outcomes it claims.
A technology may meet the published privacy, accessibility and design requirements without having sufficient evidence to support a claim of clinical effectiveness.
Conversely, evidence that an intervention can produce a beneficial outcome does not establish that the technology's data practices or engagement mechanisms meet MHTS requirements.
For example, research supporting a breathing technique does not establish that an application delivering that technique is anonymous, free of behavioural surveillance or accessible during acute distress. Likewise, an anonymous application may still deliver an intervention with inadequate supporting evidence.
Both questions matter.
Measuring Effectiveness & Impact
Claims about mental-health outcomes should be linked to clearly defined measurements. A developer should explain:
- What outcome is being measured.
- How the measurement is obtained.
- When the measurement occurs.
- Which population the available information represents.
- What limitations affect interpretation.
Examples of potentially relevant non-identifiable indicators include:
- Use
-
The number of times a support function is accessed or completed.
This indicates activity, not necessarily effectiveness.
- Self-Reported Change
-
Optional, non-identifiable reports of perceived emotional change before and after an intervention.
These may indicate perceived short-term benefit but do not automatically establish clinical effectiveness.
- Access to Further Assistance
- Aggregate information indicating whether users selected a human-support pathway, where such information can be measured without identifying or tracking individuals.
- Availability
- Technical information indicating whether relevant support functions are accessible and operational.
- Longer-Term Outcomes
-
Where appropriate and independently supported, research may investigate whether an intervention produces sustained benefits.
Such findings require suitable study designs and should not be inferred from ordinary usage statistics.
A responsible evaluation distinguishes direct observations from interpretations. For example, an increase in sessions does not prove improved mental health. Similarly, a reduction in self-reported distress during an exercise does not, by itself, establish long-term clinical benefit.
Measurement should be proportionate to the technology's intended purpose and should not compromise the privacy requirements established by MHTS.
Publicly Accessible Data
MHTS Standard 7 requires public access to collected non-identifiable data and transparency concerning the measurement of effectiveness and impact. This requirement must operate alongside Standards 1 and 2.
Public accountability must not become a reason to capture identity or introduce behavioural surveillance.
One approach is to collect only the minimum non-identifying information necessary to measure a defined outcome. For example, a system might record an anonymous count of completed exercises or aggregate responses to an optional emotional-state question. The information may then be made publicly available in a form that does not identify individuals.
However, removing names does not automatically make a dataset anonymous.1 Detailed timestamps, rare characteristics, small population groups and combinations of seemingly harmless attributes can create re-identification risks. Developers should therefore consider:
- Whether the information is genuinely non-identifiable.
- Whether unnecessary detail can be excluded.
- Whether aggregation reduces disclosure risks.
- Whether small groups require additional protection.
- Whether published information could be combined with external sources to identify someone.
- Whether the measurement method itself introduces prohibited tracking.
Where a particular dataset cannot be made safely non-identifiable, the collection or reporting approach should be reconsidered.
A suitable public report might contain
- Aggregate outcome counts.
- An explanation of the collection method.
- Relevant limitations.
- The period covered.
It should not contain
- Identifiable user records.
- Information that can reasonably be linked back to individuals.
The objective is to make the technology's claimed impact examinable without compromising the people it is intended to protect.
Transparency of Technology Practices
Public accountability requires more than publishing a favourable statistic. A meaningful explanation should help readers understand what the technology does and how its claims are supported. Relevant information may include:
- Intended Purpose
- What the technology is designed to do, and the circumstances in which it is intended to be used.
- Data Practices
- What information is collected, how it is processed and whether any information is transferred externally.
- Measurement Methods
- How effectiveness and impact indicators are defined, collected and calculated.
- Limitations
- What the available information cannot establish.
- External Processing
- Disclosure of relevant processors and the explicit confirmation required by MHTS concerning the absence of third-party behavioural data processors.
- Changes Over Time
- Information about material changes that affect the interpretation of published results.
Transparency should make claims easier to evaluate. It should not rely on technical complexity or excessive documentation to obscure relevant practices.
Independent Scrutiny & Ongoing Evaluation
Publicly accessible information allows external parties to examine a technology's reported activities and outcomes.
- Researchers
- May examine the methodology used to measure effects.
- Clinicians
- May consider whether the claims are appropriate to the stated evidence.
- Developers
- May compare approaches to privacy-preserving measurement.
- Users and organisations
- May evaluate whether the information supports the claims made by a provider.
Public availability does not guarantee that a particular dataset is complete, accurate or scientifically sufficient. Those qualities still require evaluation. Likewise, a public report should not be mistaken for independent clinical validation unless such validation has actually occurred.
Continuing evaluation is important because technologies change. An update may alter an intervention, introduce a new data processor or modify the design of an engagement feature. Changes that affect the published requirements should prompt renewed review.
The intended outcome is an environment in which mental-health technologies remain open to examination, correction and improvement.