Pharmacovigilance ADE Classifier

A rigorous ablation study showing that biomedical-domain pretraining gives measurable but modest gains on adverse-drug-event sentence screening — and that the statistical protocol matters as much as the model choice.

At a Glance

Domain: Biomedical NLP / pharmacovigilance
Role: Solo project, Spring 2026
Stack: Python, PyTorch, HuggingFace Transformers, scikit-learn, pandas, paired bootstrap (custom)
Headline metric: PubMedBERT macro-F1 0.938 vs. BERT baseline 0.920; gap −0.012 (95% CI [−0.024, −0.003], excludes zero)
Status: Complete — analysis reproducible from committed artifacts, no GPU required

Problem / Context

Pharmacovigilance teams read large volumes of case reports and clinical documents to find sentences that describe adverse drug events (ADEs) — a binary screening step that runs before any downstream extraction or signal aggregation. The ADE Corpus v2, a benchmark of 23,516 sentences drawn from medical case reports, is widely used for this task. Three transformer encoders — a general-domain BERT baseline, BioBERT (continued pretraining on PubMed + PMC), and PubMedBERT (pretrained from scratch on PubMed with a domain-specific vocabulary) — have all been applied to this corpus, but published comparisons rarely separate training-run noise from test-set sampling noise, and rarely report honest confidence intervals at the effect sizes involved (roughly one percentage point macro-F1). This project was designed to answer two specific questions with a replication protocol that keeps both noise sources explicit: does biomedical pretraining help, and does from-scratch pretraining (PubMedBERT) beat continued pretraining (BioBERT)?

Approach

I ran a structured ablation on ADE Corpus v2. Before splitting, I applied a deterministic deduplication pass that collapsed same-label duplicates and would have dropped conflicting-label groups (none existed in this release, but the policy is enforced). This removed 2,621 redundant rows and left 20,895 instances for a stratified 80/10/10 split committed as integer indices — so every model sees identical data.

Each of the three encoders was fine-tuned with a shared two-class classification head (linear layer over the pooled [CLS] representation, via HuggingFace AutoModelForSequenceClassification). Rather than using a single shared learning rate — which would silently penalize backbones with different sensitivities — I ran a 3×3 validation sweep per model and selected the per-model winner: 1e-4 for BERT, 5e-5 for both biomedical models. The ablation itself ran each model across ten random seeds at its sweep-winner rate, writing per-run metrics and raw prediction arrays to disk so all downstream analysis is reproducible without retraining.

The key statistical decision was separating seed variance from test-set sampling variance. Cross-seed standard deviations (0.003–0.007 macro-F1) answer "how stable is training?" The paired bootstrap — 1,000 iterations on pooled seed-averaged predictions, 95% percentile CI — answers "how much of the between-model gap is real versus test-set luck?" Keeping these apart prevents noise from one source inflating or masking the other. I also computed PR-AUC alongside macro-F1 because a model can improve probability ranking (PR-AUC) without improving the 0.5 decision boundary (macro-F1), and conflating them obscures what the biomedical pretraining is actually doing.

Training ran on Colab with GPU; all bootstrap and figure work runs locally in seconds from the committed results/ artifacts.

Results

Cross-seed mean test metrics at argmax threshold 0.5, ten seeds per model:

Model	macro-F1	F1 (positive class)	PR-AUC
bert-base-uncased	0.9200	0.8729	0.9432
BioBERT	0.9327	0.8932	0.9549
PubMedBERT	0.9377	0.9013	0.9608

Pairwise bootstrap confidence intervals (paired, 1,000 iterations, 95% CI on pooled test predictions):

Metric	Pair (a vs. b)	Gap	95% CI	Excludes zero?
macro-F1	bert-base vs. BioBERT	−0.0036	[−0.0141, +0.0070]	no
macro-F1	bert-base vs. PubMedBERT	−0.0121	[−0.0236, −0.0025]	yes
macro-F1	BioBERT vs. PubMedBERT	−0.0085	[−0.0176, −0.0002]	yes
PR-AUC	bert-base vs. BioBERT	−0.0117	[−0.0202, −0.0052]	yes
PR-AUC	bert-base vs. PubMedBERT	−0.0177	[−0.0311, −0.0067]	yes
PR-AUC	BioBERT vs. PubMedBERT	−0.0060	[−0.0165, +0.0026]	no

The pattern: both biomedical variants outrank BERT on PR-AUC with intervals that exclude zero. On macro-F1, only PubMedBERT shows a clearly distinguishable gap over BERT; BioBERT's macro-F1 improvement is not separable from noise at the 0.5 threshold, even though its probability ordering is better.

PR curves for all three models on the held-out test set — PR curves for all three models on the 2,090-sentence test set. The AUC differences are small but consistent with the ranking in the bootstrap table above.

Cross-seed macro-F1 distribution per model across ten training seeds — Macro-F1 across ten seeds per model. Within-model spread (0.003–0.007 std) is small relative to between-model gaps, but the BioBERT–PubMedBERT gap sits near that noise floor.

Forest plot of pairwise macro-F1 and PR-AUC gaps with 95% bootstrap CIs — Forest plot of pairwise gaps with 95% paired-bootstrap CIs. Intervals that exclude zero are distinguishable at this sample size; four of six pairs meet that bar.

Key Decisions / What I Learned

Deduplicate before splitting, not after. The raw ADE Corpus v2 ships with 2,621 exact-text duplicates. Deduplicating per split would allow the same sentence to land in both train and test, leaking trivially. Doing it globally first, before any split is generated, closes that gap cleanly.

Per-model learning rate selection is not optional. BERT's sweep winner was 1e-4; both biomedical models preferred 5e-5. Using a single shared rate tuned on one backbone would have silently penalized the others — a confound invisible in the final numbers but real in the experimental design.

Separate the two sources of noise. Seed variance and test-set sampling variance answer different questions. Collapsing them into one standard error mixes "did training converge well?" with "is this test set representative?" Keeping them apart with paired bootstrap made the BioBERT result clearer: its ranking improves (distinguishable PR-AUC gap) but its 0.5-threshold decision quality does not (indistinguishable macro-F1 gap). That distinction would have been invisible with a noisier evaluation protocol.

Effect size matters. The statistically distinguishable gains are on the order of one percentage point macro-F1. That is real, but it is not large — and on a narrow binary screening task on one corpus, it says nothing about NER, relation extraction, or clinical documents. The write-up flags this explicitly rather than leading with "biomedical pretraining is better."

Limitations

Single test split. All bootstrap CIs quantify variance within one 2,090-row partition. A cross-validated or multi-split replication would give a more complete picture of test-sampling noise.
Single-seed hyperparameter sweep. Per-model learning rate was selected from one validation run (seed 13). A more thorough protocol would sweep across seeds; this one picked a plausible winner at low cost.
Narrow task. Binary sentence classification is the simplest ADE task on this corpus. Results should not be extrapolated to span extraction, relation linking, clinical-document screening, or non-English text without separate validation.
No external validation. All evaluation is inside ADE Corpus v2. Generalization to FDA adverse-event narratives, EHR notes, or social-media pharmacovigilance feeds is untested.
No error analysis. Only aggregate metrics are reported; no inspection of which sentence types each model misclassifies was performed.

Tech Stack

Modeling: Python, PyTorch, HuggingFace Transformers (AutoModelForSequenceClassification), BERT / BioBERT / PubMedBERT checkpoints
Data: ADE Corpus v2 via HuggingFace Datasets
Evaluation: scikit-learn (metrics, stratified splits), custom paired bootstrap (NumPy)
Analysis / figures: pandas, Matplotlib, Jupyter (analysis notebook runs locally, no GPU)
Training: Google Colab (GPU); all downstream analysis from committed artifacts