Pharmacovigilance ADE Classifier

An ablation study testing whether biomedical-domain pretraining actually helps on adverse drug event (ADE) sentence screening. It does, but the gains are modest, and the statistical protocol used to measure them matters as much as which model wins.

At a Glance

Domain: Biomedical NLP / pharmacovigilance
Role: Solo, independent project (not coursework), 2026
Stack: Python, PyTorch, HuggingFace Transformers, scikit-learn, pandas, paired bootstrap (custom)
Headline metric: PubMedBERT macro-F1 0.938 (accuracy 0.959) vs. BERT baseline 0.920; gap −0.012 (95% CI [−0.024, −0.003], excludes zero)
Status: Complete; analysis reproducible from committed artifacts, no GPU required

Problem / Context

Pharmacovigilance teams read large volumes of case reports and clinical documents to find sentences that describe adverse drug events (ADEs), a binary screening step that runs before any downstream extraction or signal aggregation. The ADE Corpus v2, a benchmark of 23,516 sentences drawn from medical case reports, is widely used for this task. Three transformer encoders (a general-domain BERT baseline, BioBERT with continued pretraining on PubMed + PMC, and PubMedBERT, pretrained from scratch on PubMed with a domain-specific vocabulary) have all been applied to this corpus, but published comparisons rarely separate training-run noise from test-set sampling noise, and rarely report honest confidence intervals at the effect sizes involved (roughly one percentage point macro-F1). This project was designed to answer two specific questions with a replication protocol that keeps both noise sources explicit: does biomedical pretraining help, and does from-scratch pretraining (PubMedBERT) beat continued pretraining (BioBERT)?

Approach

I ran a structured ablation on ADE Corpus v2. Before splitting, I applied a deterministic deduplication pass that collapsed same-label duplicates and would have dropped conflicting-label groups (none existed in this release, but the policy is enforced). This removed 2,621 redundant rows and left 20,895 instances for a stratified 80/10/10 split committed as integer indices, so every model sees identical data.

Each of the three encoders was fine-tuned with a shared two-class classification head (linear layer over the pooled [CLS] representation, via HuggingFace AutoModelForSequenceClassification). Rather than using a single shared learning rate, which would silently penalize backbones with different sensitivities, I ran a 3×3 validation sweep per model and selected the per-model winner: 1e-4 for BERT, 5e-5 for both biomedical models. The ablation itself ran each model across ten random seeds at its sweep-winner rate, writing per-run metrics and raw prediction arrays to disk so all downstream analysis is reproducible without retraining.

The key statistical decision was separating seed variance from test-set sampling variance. Cross-seed standard deviations (0.003–0.007 macro-F1) answer "how stable is training?" The paired bootstrap (1,000 iterations on pooled seed-averaged predictions, 95% percentile CI) answers "how much of the between-model gap is real versus test-set luck?" Keeping these apart prevents noise from one source inflating or masking the other. I also computed PR-AUC alongside macro-F1 because a model can improve probability ranking (PR-AUC) without improving the 0.5 decision boundary (macro-F1), and conflating them obscures what the biomedical pretraining is actually doing.

Training ran on Colab with GPU; all bootstrap and figure work runs locally in seconds from the committed results/ artifacts.

The whole ablation runs from one committed YAML config with unit tests, so the 30-run sweep (3 models × 10 seeds) and every downstream figure can be regenerated without guessing at settings that lived only in a notebook.

Results

Cross-seed mean test metrics at argmax threshold 0.5, ten seeds per model:

Model	Accuracy	macro-F1	F1 (positive class)	PR-AUC
bert-base-uncased	0.9478	0.9200	0.8729	0.9432
BioBERT	0.9559	0.9327	0.8932	0.9549
PubMedBERT	0.9590	0.9377	0.9013	0.9608

Pairwise bootstrap confidence intervals (paired, 1,000 iterations, 95% CI on pooled test predictions):

Metric	Pair (a vs. b)	Gap	95% CI	Excludes zero?
macro-F1	bert-base vs. BioBERT	−0.0036	[−0.0141, +0.0070]	no
macro-F1	bert-base vs. PubMedBERT	−0.0121	[−0.0236, −0.0025]	yes
macro-F1	BioBERT vs. PubMedBERT	−0.0085	[−0.0176, −0.0002]	yes
PR-AUC	bert-base vs. BioBERT	−0.0117	[−0.0202, −0.0052]	yes
PR-AUC	bert-base vs. PubMedBERT	−0.0177	[−0.0311, −0.0067]	yes
PR-AUC	BioBERT vs. PubMedBERT	−0.0060	[−0.0165, +0.0026]	no

The pattern: both biomedical variants outrank BERT on PR-AUC with intervals that exclude zero. On macro-F1, only PubMedBERT shows a clearly distinguishable gap over BERT; BioBERT's macro-F1 improvement is not separable from noise at the 0.5 threshold, even though its probability ordering is better.

PR curves for all three models on the held-out test set — PR curves for all three models on the 2,090-sentence test set. The AUC differences are small but consistent with the ranking in the bootstrap table above.

Cross-seed macro-F1 distribution per model across ten training seeds — Macro-F1 across ten seeds per model. Within-model spread (0.003–0.007 std) is small relative to between-model gaps, but the BioBERT–PubMedBERT gap sits near that noise floor.

Forest plot of pairwise macro-F1 and PR-AUC gaps with 95% bootstrap CIs — Forest plot of pairwise gaps with 95% paired-bootstrap CIs. Intervals that exclude zero are distinguishable at this sample size; four of six pairs meet that bar.

Key Decisions / What I Learned

I deduplicated before splitting, not after. The raw corpus ships with 2,621 exact-text duplicates, and removing them per split would let the same sentence land in both train and test. Doing it once, globally, before any split exists, closes that leak cleanly.

Per-model learning rate selection turned out to matter: BERT's sweep winner was 1e-4, both biomedical models preferred 5e-5. A single shared rate tuned on one backbone would have silently penalized the others, a confound that's invisible in the final numbers but real in the experimental design.

Separating seed variance from test-set sampling variance was the most useful decision in the study. It's what let me say that BioBERT's probability ranking improves over BERT (a distinguishable PR-AUC gap) while its 0.5-threshold decision quality does not (a macro-F1 gap indistinguishable from noise). A noisier evaluation protocol would have hidden that distinction entirely.

The honest read is that the effect sizes here are small, on the order of one percentage point macro-F1, on one narrow binary task on one corpus. I'd rather say that plainly than lead with a headline that oversells what the ablation shows.

Limitations

Single test split. All bootstrap CIs quantify variance within one 2,090-row partition. A cross-validated or multi-split replication would give a more complete picture of test-sampling noise.
Single-seed hyperparameter sweep. Per-model learning rate was selected from one validation run (seed 13). A more thorough protocol would sweep across seeds; this one picked a plausible winner at low cost.
Narrow task. Binary sentence classification is the simplest ADE task on this corpus. Results should not be extrapolated to span extraction, relation linking, clinical-document screening, or non-English text without separate validation.
No external validation. All evaluation is inside ADE Corpus v2. Generalization to FDA adverse-event narratives, EHR notes, or social-media pharmacovigilance feeds is untested.
No error analysis. Only aggregate metrics are reported; no inspection of which sentence types each model misclassifies was performed.

Tech Stack

Modeling: Python, PyTorch, HuggingFace Transformers (AutoModelForSequenceClassification), BERT / BioBERT / PubMedBERT checkpoints
Data: ADE Corpus v2 via HuggingFace Datasets
Evaluation: scikit-learn (metrics, stratified splits), custom paired bootstrap (NumPy)
Analysis / figures: pandas, Matplotlib, Jupyter (analysis notebook runs locally, no GPU)
Training: Google Colab (GPU); all downstream analysis from committed artifacts