Synthetic network beacon flows

Reference answers and documented traps

18 traffic families. Regular traffic that is benign (push, keep-alives, time sync, update and device-management check-ins) is regular on purpose, and low-and-slow beacons with jitter are hard on purpose. The reference answers give the label and the family of every row; they are kept outside the data folder so you can predict first and compare after.

The traps

• Leakage. triage_tag is written after an analyst looked at the flow. On the test file, reading "escalated or watchlist" as beaconing gives F1 0.909, higher than every baseline below. A model that uses it has learned the answer from the answer.
• Imbalance. About 11.7 percent of the test rows are beaconing. Answering "normal" every time scores a high accuracy and finds nothing; run it and read it.
• The fixed-interval shortcut. The rule "jitter under 5 percent" is one of the shipped baselines (baseline.py --model rule). Run it and score it to see how many test flows it flags and how many of those are normal. It catches most fixed-sleep beacons and misses most jittered, low-and-slow, burst and fronted ones. The normal flows it flags are mostly push, time-sync and update-check traffic.
• Look-alike rows. Beaconing rows from one phone share a configuration. Random folds on the training file give a forest an average precision of 0.803; folds that keep each phone together give 0.760.
• Blank values. Destinations seen once a day have no interval statistics: 25 train rows have no mean gap and 90 have no jitter figure.
• Overlap. About half of the beaconing destinations copy the size, direction, session and destination profile of a normal app.

Baseline scores on the test file

The fixed-interval rule is not in this table, because question 4 of the challenge asks for its false positives: run it and read it off. Threshold 0.5 on a class-balanced fit. Scores are from the shipped baseline.py and evaluate.py. These numbers are for this synthetic set only. They are a floor for the exercise, not a target, and say nothing about real traffic.

Try it with free tools

SELECT dest_age_bucket,
       COUNT(*) AS flows,
       SUM(CAST(label AS INTEGER)) AS beacons,
       ROUND(100.0 * SUM(CAST(label AS INTEGER))
             / COUNT(*), 1) AS pct_beacon
FROM train
GROUP BY dest_age_bucket
ORDER BY pct_beacon DESC;

dest_age_bucket | flows | beacons | pct_beacon
lt_7d | 75 | 13 | 17.3
unknown | 128 | 21 | 16.4
30_365d | 283 | 41 | 14.5
7_30d | 119 | 16 | 13.4
gt_365d | 1470 | 152 | 10.3

test flows 889, beaconing 104, normal 785
threshold 0.500 (applies to the score column; a label column is used as given)
confusion: TP 79  FP 63  FN 25  TN 722
precision 0.556   recall 0.760   F1 0.642   false-alarm rate 0.080
accuracy 0.901 (always answering normal gives 0.883)   balanced accuracy 0.840
ROC AUC 0.945   average precision 0.767   precision@104 0.654   recall at 5% false alarms 0.663

beaconing families (share flagged):
  burst_beacon      14 flows  flagged  10  (0.71)
  fixed_beacon      24 flows  flagged  19  (0.79)
  fronted_beacon    14 flows  flagged  13  (0.93)
  jitter_beacon     23 flows  flagged  15  (0.65)
  lowslow_beacon    29 flows  flagged  22  (0.76)
normal families (false-alarm share):
  browsing         186 flows  flagged   0  (0.00)
  ip_direct_app     27 flows  flagged   5  (0.19)
  keepalive         57 flows  flagged  21  (0.37)
  mdm_checkin       16 flows  flagged   6  (0.38)
  new_cdn_app       42 flows  flagged   0  (0.00)
  push              50 flows  flagged   3  (0.06)
  social_refresh    90 flows  flagged   0  (0.00)
  streaming         51 flows  flagged   0  (0.00)
  sync             100 flows  flagged  12  (0.12)
  telemetry         83 flows  flagged  13  (0.16)
  timesync          26 flows  flagged   0  (0.00)
  update_check      39 flows  flagged   0  (0.00)
  vpn_tunnel        18 flows  flagged   3  (0.17)

Hashes

Check every file after download. The set hash is the SHA-256 of manifest.json, which lists the size and hash of every other file.

Set hash (SHA-256, manifest.json)