Turn each alert's history into a profile, then rank
Each alert owns hundreds of transaction rows, but a model needs one row per alert. Most of the work is deciding how to summarize a customer's history so that escalated and dismissed alerts separate.
We explored first, built features for every pattern the exploration suggested, and kept only what improved cross-validated AUC. We built two models on different feature sets; both reached the same conclusion, and the submission averages them.
20,000 alerts and the 10 million transactions behind them
- Train alerts
- Test alerts
- Transactions
- Median per alert
- History window
- 180days
| Column | Meaning | Values |
|---|---|---|
| signal_id | Alert identifier, links alerts to transactions | SG_000001 … |
| signal_sanasi | Alert date | |
| eskalatsiya | Target: 1 escalated, 0 dismissed | train only |
| tranzaksiya_vaqti | Transaction timestamp | to the second |
| kirim_chiqim | Direction | kirim (in) · chiqim (out) |
| tranzaksiya_turi | Transaction type | karta (card) · bank_otkazmasi (bank transfer) · naqd (cash) · xalqaro (international) |
| miqdor_indeksi | Standardized transaction size | −2.9 to 6.7, roughly a scaled log amount |
Quality checks
- No missing values or duplicates; every alert has transactions.
- Train and test alert IDs don't overlap.
- Every alert has 180 days of history up to its alert date and nothing from later days. A few rows (0.06%) are stamped on the alert date itself; they are equally common for escalated and dismissed alerts (17.4% vs 17.1% escalation), so they carry no signal. Model A keeps them and Model B drops them.
- Train and test look alike: a classifier trying to tell them apart scores 0.49 AUC (chance).
- No shortcuts: IDs, dates, timestamps and shared transactions carry no signal (≈ 0.50 AUC).
Transaction mix by type and direction
All train transactions, in millions.
About one alert in six is escalated, steadily over two years
of 14,000 training alerts () were escalated. The classes are imbalanced but not extremely, which is why we measure ranking quality (ROC-AUC) rather than accuracy: always predicting "dismissed" would already be 83% accurate.
Escalation rate by alert quarter
Share of alerts escalated; the dashed line is the overall rate.
What 180 days of activity look like
Every history combines a slowly fading stream of everyday activity with a pile-up of transactions in the last minutes before midnight on the day before the alert.
Transactions per alert, by day before the alert
Average transactions per alert per day, days 180 → 2. The final day is left out because its pile-up dwarfs the scale.
The final hour before the alert
Average transactions per alert in each hour before the alert (log scale).
Hour of day
Share of transactions by clock hour.
Incoming vs outgoing
Average per alert, by outcome.
| Dismissed | Escalated |
|---|
Transaction size by type
Density of miqdor_indeksi.
Every customer has their own scale
Alerts by the average miqdor_indeksi of their own transactions.
Simple behaviours barely differ. Relative amounts do.
We compared escalated and dismissed alerts on every obvious behaviour first. None of them separates the classes well, so we looked inside each customer's own profile.
How escalated alerts differ on simple behaviours
Escalated average relative to dismissed average (percent difference). Hover for values and single-feature AUC.
Behavioral fingerprint: raw vs within-customer amounts
Standardized difference (escalated − dismissed) of each alert's average amount per direction × type. Relative means minus that customer's overall average.
Escalated customers move cash in at high amounts relative to their own bank transfers. Measured within the customer, this contrast is the strongest single predictor in the data.
Escalation rate by feature decile
Alerts in ten equal groups by each feature. Dashed line: overall rate, 17.2%.
From observation to feature
| What we saw | Feature family we built | Effect on CV AUC |
|---|---|---|
| Customers differ in scale by ±1 unit (§4) | Within-customer contrasts: each direction × type average minus the customer's own average or another channel | +0.016 · largest gain |
| Types differ sharply in size and role (§2, §4) | Count, share, mean, max and money for each of the 8 direction × type cells | core baseline |
| Means can hide where amounts cluster | Per-cell quantiles, spread, skew; amount histograms | small, near noise |
| Behavior might change before an alert | Statistics over the last 1h, 6h, 1d, 7d, 30d, 90d vs baseline | ≈ 0 |
| A pile-up of transactions right before the alert (§4) | Final-hour size, duration, gaps, mix, amounts vs history | ≈ 0 |
| Classic money-laundering patterns | Same-day pass-through, in→out chains, gaps, monthly trend | ≈ 0 |
Where the model finds its signal
Share of LightGBM split gain by feature family (Model A).
Two complementary models, one ensemble
Model A (1,084 features)
LightGBM, CatBoost and XGBoost with shallow trees (7 leaves, at least 100 alerts per leaf, strong regularization). 5-fold stratified CV repeated over 5 seeds with early stopping; 25 fold models per algorithm. AUC 0.655.
Model B (201 features)
LightGBM, 7 leaves × 300 trees, on direction × type statistics plus within-customer normalization, averaged over 5 seeds. AUC 0.653. The two agree on about 91% of their ranking, so averaging reduces each one's errors.
Cross-validated AUC by step
Out-of-fold ROC-AUC. Axis starts at 0.60.
Escalation rate by predicted risk
Out-of-fold predictions in ten equal groups. Dashed line: overall rate.
Validation
- Out-of-fold scores only: every prediction comes from a model that never saw that alert.
- Stable: model seeds move the score by ±0.0006, and folds agree within the expected noise.
- No train/test shift (adversarial AUC 0.49), no transactions from after the alert date, and no identifiers used as features. Model A also includes the alert's month and weekday, which carry no signal on their own (AUC 0.51).
- Test predictions average 17.3%, matching the training escalation rate of 17.2%.
- Reproducible: one Jupyter notebook rebuilds the submitted file from the five original competition files (≈ 25 minutes, fixed seeds).
Review the riskiest alerts first
An AUC is abstract. In practice the model ranks the day's alerts so that specialists see the most likely escalations first. Drag the slider to choose how much of the queue to review.
- Escalations caught
- In random order
- Share escalated among reviewed
We searched for a hidden pattern and found none
Synthetic data sometimes hides a sharp planted pattern that pushes AUC far above 0.65. In total we tested more than thirty ideas; these are the most important.
| Hypothesis | Test | Best AUC |
|---|---|---|
| Rare planted transactions | Score each transaction, flag each alert by its most suspicious one | 0.537 |
| Hidden amount/type/time pocket | ~200 cells of amount range × type × time before alert | 0.556 |
| Pass-through / layering | Money in, ~same amount out within 1–72 hours | 0.538 |
| Templates or batches | Nearest-neighbour alerts; neighbours by ID and date | 0.506 |
| Timestamp artefacts | Seconds, minutes, weekday, end-of-day clamp | 0.508 |
| Other model types | Neural network over raw transactions; linear and additive models; likelihood ratios | 0.646 |
Model AUC by length of customer history
Out-of-fold AUC in five equal groups of alerts, by transactions in the 180 days.
What we learned
- The data is clean. Each alert has 180 days of prior history and nothing from later days, and train and test match, so cross-validation is a trustworthy guide.
- Obvious triggers don't separate the classes. Volume, cash and international share, time of day and the final-hour pile-up look the same for escalated and dismissed alerts.
- Relative amounts carry the signal. Cash inflows that are large compared with the same customer's bank transfers are the strongest escalation marker. Removing each customer's scale is what makes it visible.
- Two different models agree. Built on different feature sets, both reached about 0.655 and rely on the same key features. Averaged, they reach , and the riskiest 10% of alerts escalate 3.5× as often as the safest 10%.
- The ceiling is in the data. We found no leak, template or planted pattern, and accuracy doesn't improve with longer histories. Gains came from better features, not bigger models.
In one sentence: escalation is predicted less by how much money moves than by how a customer's cash compares with their own bank transfers.