Challenge DS-0007

Find the beacon in phone traffic

Turn timing and size behaviour into features, score a rare-class detector honestly, and recognise a leakage column and a shortcut rule that flatters a model.

Expert 50 to 70 minutesNetwork (any phone)8 questions, 180 pointsAlways open, self-paced
Start the questionsRelated article

The task

Is this phone's flow pattern to one destination command-and-control style beaconing, or normal app traffic? Each row summarises one destination as seen from one phone over 24 hours. Many normal apps are as regular as a beacon, and some beacons hide in plain sight. A flagged flow is an indicator consistent with beaconing. It does not prove command-and-control.

All challenge data is synthetic or comes from the Lab's published articles. No account is needed, nothing you type here leaves your browser except each answer you check, and the optional form if you send it.

What you need

Read and download

Tools (all free)

  • Python 3 (standard library) to run the baselines
  • A spreadsheet (optional)
  • Wireshark or Zeek are not needed: the rows are already summarised

Any tool that reads text will do. You do not need the Lab's own software or an account.

Questions

Each answer is checked when you press Check answer, and you see the result straight away. A wrong answer costs nothing, and you may try again. A hint lowers the points you can earn for that question: 3, 5 and 8 points for hints 1, 2 and 3, and a correct answer always earns at least 1 point. Checking needs JavaScript; reading does not.

  1. Question 1 of 820 points

    How many rows of data/train.csv are labelled as beaconing (label = 1)?

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 3 points

    Count rows where the last column is 1.

    Hint 2 costs 5 points

    It is about one row in nine.

    Hint 3 costs 8 points

    A little over 240.

  2. Question 2 of 820 points

    Which column is a leakage column, because an analyst wrote it after looking at the flow?

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 3 points

    Read the data dictionary for the word Leakage.

    Hint 2 costs 5 points

    The values are none, reviewed_ok, watchlist and escalated.

    Hint 3 costs 8 points

    Its name contains the word triage.

  3. Question 3 of 820 points

    What accuracy, as a percentage with one decimal, does a detector get on the 889 test rows by answering 'normal' every time? Use the reference answers.

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 3 points

    Count the normal rows (label 0) in answers/reference_answers.csv for the test flows.

    Hint 2 costs 5 points

    Divide by 889.

    Hint 3 costs 8 points

    It is close to nine in ten.

  4. Question 4 of 825 points

    Run code/baseline.py --model rule, then score it with code/evaluate.py. How many normal flows does this rule flag as beaconing (false positives)?

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 3 points

    Quick start is in README.txt. Run baseline.py with --out, then evaluate.py with that file.

    Hint 2 costs 5 points

    The line starting with confusion gives TP, FP, FN and TN.

    Hint 3 costs 8 points

    Read FP.

  5. Question 5 of 825 points

    How many of the beaconing rows in train.csv have a blank interval_jitter_pct?

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 3 points

    A blank means there were too few flows to compute the value.

    Hint 2 costs 5 points

    Filter label = 1 and look for the empty cell.

    Hint 3 costs 8 points

    A very slow beacon can show up as one or two flows in a day.

  6. Question 6 of 825 points

    Select the columns that must not be used as features, according to the data dictionary.

    Select all that apply.

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 3 points

    Two are identifiers and one is a leakage column.

    Hint 2 costs 5 points

    A phone must be kept out of both sides of a split.

    Hint 3 costs 8 points

    The identifiers are numbered ids; the third is the analyst's tag.

  7. Question 7 of 820 points

    In train.csv, which protocol has the highest share of beaconing rows?

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 3 points

    Group rows by protocol and compute the share with label = 1.

    Hint 2 costs 5 points

    Divide the number of label 1 rows by all rows of that protocol.

    Hint 3 costs 8 points

    Compare four numbers.

  8. Question 8 of 825 points

    On test.csv, read triage_tag as 'escalated or watchlist means beaconing'. What recall does that rule reach, using the reference answers? Give three decimals.

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 3 points

    Recall is the share of real beaconing rows that the rule flags.

    Hint 2 costs 5 points

    Count true beaconing rows among the flagged rows, then divide by all true beaconing rows in the test file.

    Hint 3 costs 8 points

    It is above 0.9, which is the problem: the tag would not exist when you need to decide.

You have finished: what you should have learned

  • Accuracy is the wrong score when one class is rare. Use precision, recall and the false-alarm rate, and compare with always answering normal.
  • A column written after the label leaks. A model that uses it scores higher than any honest baseline, and the score does not warn you.
  • Regular timing is not proof. Push channels, time sync and device-management check-ins are as regular as a fixed beacon.
  • Keep a phone out of both sides of a split, and say that a flagged flow is an indicator consistent with beaconing, not proof of command-and-control.
See your progress mapRead the related article again

Completion note

Typed here only. It appears on the printed note and is not saved or sent anywhere.

UpDownto moveEnterto openEscto close