Publications / CPFE — Mental-Health NLP

Under journal review2026

Cross-Platform Generalisation Failure in Mental Health NLP: A Five-Axis Fairness Audit of Transformer Models on Social Media

Rajveer Singh Pall, Sameer Yadav

Mental-health classifiers that look near-perfect in-platform collapse across platforms — on discrimination, calibration, equity, and even their explanations.

Code ↗

The discovery in one figure

ONE PLATFORM: AUC 0.98. FIVE AXES OF RELIABILITY, ALL SHIFTED AT ONCE.Discrimination−30 to −40% AUCCalibrationECE → 0.54EquityDI < 0.17AttributionJaccard ≈ 0Significancein-platformcross-platform
Cross-platform reliability collapses on all five independent axes at once.

The paper in five minutes

Software that scans social-media posts for signs of depression or anxiety is usually trained and tested on a single platform, where it looks near-perfect. We took such models — AUC above 0.98 — and simply moved them to Reddit and Twitter. Accuracy collapsed by a third. Their confidence scores became badly dishonest. Fairness metrics failed for exactly the classes that matter clinically. And the words the models relied on changed almost completely from platform to platform — the model was never reading what its developers thought it was reading.

The research question

Mental-health NLP models report AUCs near 0.99 on one platform. What survives when the same model meets Reddit or Twitter language?

How it works

CPFE: a five-axis cross-platform audit — discrimination, calibration, statistical significance, prediction equity, attribution stability — across BERT/RoBERTa variants, five seeds, with DeLong CIs, bootstrap ECE, Bonferroni-corrected tests, and gradient-saliency Jaccard analysis.

  1. 01
    Train in-platformtransformer classifiers, 35,556 posts, five seeds
  2. 02
    Audit on five axesdiscrimination · calibration · significance · equity · attribution
  3. 03
    Cross platformssame models, Reddit and Twitter language
  4. 04
    Attempt repairtemperature scaling fixes calibration only; fine-tuning recovers more

Experimental results

Within-platform AUC of 0.983–0.987 degrades 30–40% cross-platform; ECE inflates up to 0.54; equity metrics fail for clinical proxy classes; attribution overlap falls to Jaccard ≈ 0 in 13 of 16 model–class pairs. Temperature scaling repairs calibration (−88% ECE) but not discrimination; target-domain fine-tuning recovers far more.

Calibration error (ECE) as the platform changes
Home platform
0.056–0.060
Reddit
0.196–0.229
Twitter
0.499–0.542

Bars drawn at the lower bound of each reported range — the conservative reading.

  • 0.983 → ~0.65AUC under platform shift (−30–40%)
  • ≈ 0attribution Jaccard in 13/16 pairs
  • −88%ECE after temperature scaling

Stated honestlyLabels are clinical proxies from emotion mappings, not diagnoses.

Figures from the paper

Forest plot of cross-platform AUC with DeLong confidence intervals across models and platforms.
Fig. 1Forest plot of cross-platform AUC with DeLong confidence intervals across models and platforms.
Degradation by platform: what a near-perfect in-platform model loses on Reddit and Twitter.
Fig. 2Degradation by platform: what a near-perfect in-platform model loses on Reddit and Twitter.
Calibration curves in-domain versus cross-platform — confidence becomes dishonest under shift.
Fig. 3Calibration curves in-domain versus cross-platform — confidence becomes dishonest under shift.

Figures generated by the paper's own pipeline — reproducible from the repository.

What this changes

Fairness itself is shown to be a deployment-shift casualty: every trustworthiness axis can fail while the benchmark number stays excellent.

Resources

Code ↗
  • NLP · LLMs
  • Fairness
  • Healthcare AI
  • Deployment Shift

Citation

Manuscript under review — citation will be posted on acceptance. Reach me at rajveerpall04@gmail.com.

Where this leads next