Challenge DS-0006

Separate stalkerware from look-alikes

Read the permission and behaviour profile of Android apps, build and score a baseline, and recognise leakage, lineage clusters, an ordinal column and benign look-alikes.

Practitioner 40 to 60 minutesAndroid8 questions, 135 pointsAlways open, self-paced
Start the questionsRelated article

The task

Does the permission and behaviour profile of an Android app look like stalkerware, or like a benign app? The set holds 3,000 invented app rows with features an examiner can read from an extraction: requested and granted permissions, accessibility and notification access, a hidden icon, install source and App Ops counts. Several traps are built in on purpose.

All challenge data is synthetic or comes from the Lab's published articles. No account is needed, nothing you type here leaves your browser except each answer you check, and the optional form if you send it.

What you need

Read and download

Tools (all free)

  • Python 3 (standard library) to run the baselines
  • A spreadsheet (optional)
  • scikit-learn (optional)

Any tool that reads text will do. You do not need the Lab's own software or an account.

Questions

Each answer is checked when you press Check answer, and you see the result straight away. A wrong answer costs nothing, and you may try again. A hint lowers the points you can earn for that question: 2, 3 and 5 points for hints 1, 2 and 3, and a correct answer always earns at least 1 point. Checking needs JavaScript; reading does not.

  1. Question 1 of 815 points

    How many rows of data/train.csv are labelled is_stalkerware = 1?

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 2 points

    Count rows where the last column is 1.

    Hint 2 costs 3 points

    The README gives the shape of the set. Check it against your own count.

    Hint 3 costs 5 points

    Between 150 and 200.

  2. Question 2 of 815 points

    Which column is marked LEAKAGE in the data dictionary and must not be used as a feature?

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 2 points

    Search the data dictionary for the word LEAKAGE.

    Hint 2 costs 3 points

    It records an earlier reviewer's note.

    Hint 3 costs 5 points

    Its name contains the word triage.

  3. Question 3 of 815 points

    How many distinct lineage_id values are there across train.csv and test.csv together?

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 2 points

    Collect lineage_id from both files into one set.

    Hint 2 costs 3 points

    A lineage is in train or in test, never both, so you may also count the two files separately and add.

    Hint 3 costs 5 points

    More than 130.

  4. Question 4 of 815 points

    How many rows of train.csv come from a bug report (evidence_source = bugreport) and so have blank App Ops counts?

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 2 points

    Filter on evidence_source.

    Hint 2 costs 3 points

    The App Ops history exists in the database form, not in a bug report dump.

    Hint 3 costs 5 points

    Between 200 and 300.

  5. Question 5 of 820 points

    What accuracy, as a percentage with one decimal, does a model get on the test file by always saying 'benign'?

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 2 points

    Use answers/test_answers.csv.

    Hint 2 costs 3 points

    Count the rows with is_stalkerware = 0 and divide by the number of rows.

    Hint 3 costs 5 points

    It is above 90 percent.

  6. Question 6 of 820 points

    Run code/baseline_stdlib.py --model logreg, then score it with code/evaluate.py against answers/test_answers.csv. How many stalkerware apps in the test file does it find (true positives)?

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 2 points

    The usage lines are in the first lines of evaluate.py and in README.txt.

    Hint 2 costs 3 points

    The confusion line gives TP, FP, FN and TN.

    Hint 3 costs 5 points

    Read TP.

  7. Question 7 of 820 points

    Select the app families that answers/test_answers.csv marks as benign look-alikes.

    Select all that apply.

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 2 points

    Look at the column family_kind.

    Hint 2 costs 3 points

    Look-alikes are benign apps that use the same kind of permissions on purpose.

    Hint 3 costs 5 points

    One is for children, one is for remote help, one finds a lost phone.

  8. Question 8 of 815 points

    net_egress_bucket is an ordinal column. If a tool sorts its values as text, which value comes first?

    Need a hint?

    Three hints, in order. Each one lowers the points you can earn for this question.

    Hint 1 costs 2 points

    The correct order is none, low, medium, high, very_high. Text order is alphabetical.

    Hint 2 costs 3 points

    Put the five names in alphabetical order.

    Hint 3 costs 5 points

    It starts with the letter h.

You have finished: what you should have learned

  • Leakage is a column written after the label or by the same judgement. Ask of each feature whether you would have it when you must decide.
  • Rows from one lineage resemble each other. Split and cross-validate by lineage, or the score flatters the model.
  • Benign look-alikes use the same permissions on purpose. A flag is an indicator worth a closer look, not a finding that someone was watched.
  • Check how an ordinal column sorts, and where a blank comes from. An empty cell can mean the evidence source never held the value.
Next challenge by number: Find the beacon in phone trafficSee your progress mapRead the related article again

Completion note

Typed here only. It appears on the printed note and is not saved or sent anywhere.

UpDownto moveEnterto openEscto close