Deployed ScamRadar+ Pipeline
PERFORMANCE
Logistic Regression on 500,000 word + character TF-IDF features + 25 engineered numerical features + modular Rule Engine · trained on 195,776 deduplicated message clusters · decision threshold 0.59
About these numbers
Headline metrics below are for the deployed ScamRadar+ pipeline (classifier + Rule Engine) on a locked one-shot benchmark of 25,306 messages held out from all model selection, hyperparameter search, calibration, and threshold tuning. The pre-augmentation baseline of the same pipeline (before the modern-scam corpus expansion) scored F1 = 0.941 on this benchmark — that reference number is preserved in the API's /stats endpoint. The current deployed pipeline scores F1 = 0.916 for the classifier alone and F1 = 0.913 for the deployed composite. Every scoring event on the benchmark is recorded in the research repository.
Accuracy
Out of all messages, how many did we classify correctly?
Precision
Of messages we called SCAM, what fraction really were scams?
Recall
Of all real scams in the dataset, what fraction did we catch?
F1 Score
Balanced average of Precision and Recall — the key overall quality score.
96.87
%Accuracy
91.31
%F1 Score
99.07
%ROC-AUC
91.02
%Precision
91.60
%Recall
12.00
Scam Types
Signal Analysis · Live Variance
MODEL SIGNALS
Confidence separation, precision–recall tradeoff, and external benchmark headline
Confidence Separation
Score deviation from decision threshold (t = 0.59)
Precision–Recall Balance
P − R gap across threshold sweep (zero = balanced)
External Benchmark
Locked one-shot set · n = 25,306 · never seen at training
Prediction Quality · Threshold Analysis
DIAGNOSTIC CURVES
How model behaviour shifts across operating points and confidence levels
Precision vs Recall
Threshold sweep on external benchmark (production point at t = 0.59)
Confidence Distribution
Score buckets on external benchmark — scam vs legit share
Recall by Scam Category
Real-world detection rate per scam type (external benchmark)
Dashed line = overall recall (0.916) · Per-category recall on 25,306 messages
Classifier Quality · External Benchmark
ROC & CONFUSION
How well the model separates scam from legitimate messages on the 25,306-message held-out set
ROC Curve
ROC-AUC = 0.991 · Near-perfect separation
Confusion Matrix
Predictions on the 25,306-message external benchmark
External benchmark · 25,306 messages · threshold = 0.59 · accuracy = 96.87%
Real-World Performance
BY SCAM CATEGORY
Recall on scam classes · false-positive rate on legitimate classes · external benchmark
Scam-Class Recall
% of scams in each category correctly flagged
Recruitment scams remain the weakest single class — real recruiter language and scam recruiter language look similar at first message.
Legit-Class False-Positive Rate
% of legitimate messages wrongly flagged (lower is better)
Formal emails have the highest false-positive rate — they use scam-adjacent language (deadlines, verification, dollar amounts).
Training Data
DATASET COMPOSITION
253,264 messages across 12 public corpora — every source has a documented URL and license
Message counts per source
Scam vs legitimate breakdown before deduplication
After SHA-1 exact-dedup + MinHash near-dedup: 195,776 unique message clusters used for training. Cluster-aware train / val / test split prevents near-duplicates from crossing the boundary.
One Pipeline · Iteratively Improved
MODEL DEVELOPMENT
External benchmark PR-AUC and F1 across every iteration of the same core ScamRadar+ pipeline
Iterative Development Progression
Each iteration builds on the previous under strict cluster-grouped evaluation