Challenge DS-0005

Train and doubt a reset classifier

Count and score a classifier on a synthetic set, spot the leakage column and the noise columns, and see why accuracy misleads when one class is rare.

Practitioner 40 to 60 minutesAndroid8 questions, 140 pointsAlways open, self-paced

Start the questions Read the article first

The task

Was this Android phone factory reset in the last 30 days? The set gives 1,400 labelled rows to learn from and 600 to test on, with features an examiner reads: recovery history, setup flags, app install times, account time and clock checks. Every row is invented. The set teaches the method; a score here says nothing about real phones.

All challenge data is synthetic or comes from the Lab's published articles. No account is needed, nothing you type here leaves your browser except each answer you check, and the optional form if you send it.

What you need

Read and download

Article LAB-0001: Samsung recovery and reset trail
Dataset DS-0005: Factory-reset classifier set
factory-reset-classifier-1.0.zip Train, test, answers, data dictionary, baseline code and generator, CC BY 4.0
manifest.json File list with sizes and SHA-256 hashes

Tools (all free)

Python 3 (standard library) to run the baseline and the evaluation script
A spreadsheet (optional, to look at the rows)
scikit-learn (optional, the set also ships a version that uses it)

Any tool that reads text will do. You do not need the Lab's own software or an account.

Questions

Each answer is checked when you press Check answer, and you see the result straight away. A wrong answer costs nothing, and you may try again. A hint lowers the points you can earn for that question: 2, 3 and 5 points for hints 1, 2 and 3, and a correct answer always earns at least 1 point. Checking needs JavaScript; reading does not.

Checking needs JavaScriptYou can read the questions and hints without it, but answers are checked by a script on this page.

Question 1 of 815 points

How many rows of train.csv are positive (reset_30d = 1)?

Answer to question 1

Need a hint?

Three hints, in order. Each one lowers the points you can earn for this question.

Hint 1 costs 2 points
Count rows where the last column is 1.

Hint 2 costs 3 points
In Python, sum(row['reset_30d'] == '1' for row in rows).

Hint 3 costs 5 points
Between 400 and 450.
Question 2 of 815 points

How many rows of train.csv have no recovery history file (recovery_file_present = 0)?

Answer to question 2

Need a hint?

Three hints, in order. Each one lowers the points you can earn for this question.

Hint 1 costs 2 points
Filter on recovery_file_present.

Hint 2 costs 3 points
In the data dictionary, find why about one phone in four has none.

Hint 3 costs 5 points
Between 300 and 400.
Question 3 of 815 points

Which column should be dropped before training, because it is written after the question was answered?

recovery_wipe_entries reviewer_reset_note setup_flag_changes_90d storage_used_pct

Need a hint?

Three hints, in order. Each one lowers the points you can earn for this question.

Hint 1 costs 2 points
Ask of every column: would I have this value at the moment I need the prediction?

Hint 2 costs 3 points
The data dictionary marks one column as a trap in capital letters.

Hint 3 costs 5 points
It is a note typed by a human reviewer.
Question 4 of 820 points

Run code/baseline_stdlib.py with its default model and score it with code/evaluate.py. What accuracy does it reach on the 600 test rows? Give three decimals.

Answer to question 4

Need a hint?

Three hints, in order. Each one lowers the points you can earn for this question.

Hint 1 costs 2 points
Quick start in README.txt: baseline_stdlib.py, then evaluate.py predictions.csv test_answers.csv.

Hint 2 costs 3 points
Run both commands from inside the folder you unzipped.

Hint 3 costs 5 points
The accuracy is a little above 0.9.
Question 5 of 820 points

In the same run, how many phones that were reset in the last 30 days does the model miss (actual 1, predicted 0)?

Answer to question 5

Need a hint?

Three hints, in order. Each one lowers the points you can earn for this question.

Hint 1 costs 2 points
Read the confusion matrix, not only the accuracy.

Hint 2 costs 3 points
Rows of the matrix are the actual class, columns the predicted class.

Hint 3 costs 5 points
The cell is in the bottom left.
Question 6 of 820 points

In the imbalanced variant, what accuracy, as a percentage with one decimal, does a model get by predicting 'not reset' for every test row?

Answer to question 6

Need a hint?

Three hints, in order. Each one lowers the points you can earn for this question.

Hint 1 costs 2 points
Use variants/imbalanced/test_answers.csv.

Hint 2 costs 3 points
Count the rows with reset_30d = 0 and divide by the number of rows.

Hint 3 costs 5 points
It is higher than the accuracy of the real baseline on the main set.
Question 7 of 820 points

Select the columns that carry no signal about the label by design.

Select all that apply.

boot_reason_history_rows epoch_zero_file_time future_file_time_48h apps_installed_30d_fraction setup_flag_changes_90d

Need a hint?

Three hints, in order. Each one lowers the points you can earn for this question.

Hint 1 costs 2 points
Compare each column's average for reset_30d = 1 and reset_30d = 0.

Hint 2 costs 3 points
Three of the five columns give nearly the same average for both classes.

Hint 3 costs 5 points
Two of them are clock checks, and one is a count of boot-reason rows.
Question 8 of 815 points

How many of the 600 test rows have clock_tampered = 1 in test_answers.csv?

Answer to question 8

Need a hint?

Three hints, in order. Each one lowers the points you can earn for this question.

Hint 1 costs 2 points
The file test_answers.csv holds extra columns for scoring by slice.

Hint 2 costs 3 points
Count the rows where clock_tampered is 1.

Hint 3 costs 5 points
Fewer than 100, more than 50.

You have finished: what you should have learned

Read the confusion matrix before the accuracy. Two models with a similar accuracy can miss very different numbers of resets.
A column written after the label, or by the same judgement, leaks. A model that uses it scores higher and nothing in the test score warns you.
When one class is rare, always predicting the common class scores high. Use precision and recall, and compare with that reference point.
Synthetic data teaches the method. It does not measure how a model would do on real phones, and a model trained here must never be used on a case.

Next challenge by number: Separate stalkerware from look-alikes See your progress map Read the article again

Completion note

Your name for the note Optional

Typed here only. It appears on the printed note and is not saved or sent anywhere.

The task

What you need

Read and download

Tools (all free)

Questions

How many rows of train.csv are positive (reset_30d = 1)?

How many rows of train.csv have no recovery history file (recovery_file_present = 0)?

Which column should be dropped before training, because it is written after the question was answered?

Run code/baseline_stdlib.py with its default model and score it with code/evaluate.py. What accuracy does it reach on the 600 test rows? Give three decimals.

In the same run, how many phones that were reset in the last 30 days does the model miss (actual 1, predicted 0)?

In the imbalanced variant, what accuracy, as a percentage with one decimal, does a model get by predicting 'not reset' for every test row?

Select the columns that carry no signal about the label by design.

How many of the 600 test rows have clock_tampered = 1 in test_answers.csv?

You have finished: what you should have learned

Completion note

Thank you