How we work
Methodology
What counts as evidence here, how a result is made reproducible, and the words we use for what we know and what we do not.
Evidence standards
Every claim in an article points to evidence that a reader can look at. A claim is either observed on a named device and build, or it is marked as inferred. We do not carry a result from one device or version to another without testing it, and we say when a result comes from a single device.
- Record the setup. Device class, operating system build, acquisition method, tool versions and the date of the work.
- Keep ground truth for reference images. When a released reference image is made by performing an action on a test device, we record what was done and when, before we look at the evidence.
- Hash the inputs. Every image and file an article relies on has a SHA-256 hash, shown in the dataset record.
- Separate observation from reading. What a file contains is one statement. What it means is another, and it carries its own confidence.
Reproducibility
Each article has a "how to reproduce" entry in its evidence box. It lists the actions, the order, the files to open and the expected result, so another examiner can repeat the test on a comparable device. Where we release a reference image, the steps point at that image. Where we cannot release evidence, we say why and describe the test in enough detail to be repeated on a device of your own.
Claim language
We use a small set of phrases on purpose.
| Phrase | What it means here |
|---|---|
| Indicator | A record that points to an event. It is a reason to look closer, not a conclusion. |
| Consistent with | The evidence fits the explanation. Other explanations may also fit. |
| Observed on | Seen directly on the named device and build. Not carried to others. |
| Does not prove | What the evidence cannot establish. Every article lists these. |
| Not tested | We did not try it. We make no claim either way. |
Confidence
Each article states a confidence level for its main findings.
- Low
- Seen once, or on one device, or the explanation depends on an assumption we could not test.
- Moderate
- Repeated on more than one run or device, checked against a known reference, and no conflicting result found.
- High
- Repeated across builds and devices, matched to a known reference, and explained by the platform's own documentation or source.
What we test on
We work on research devices (test devices) provided to the Lab by their owners and held in its research collection, on extractions taken from them and used read-only for research with their owners' permission, and on synthetic data we create. We do not publish images from cases, real identifiers, real contact data or real messages. Public reference images are produced from test devices in a known state, or built from synthetic content, and are checked for personal data before release.
Case data stays out
Data from cases is never published here. Research devices are provided to the Lab by partners and by their owners for research use only.
Review
An article goes public only after a technical review of the claims, a rights check of every image and quotation, and a check that the language matches the evidence. Each article lists its review date and the date it is due to be reviewed again, because platforms change.
After publication, articles are versioned. Changes appear in the version history and, when they alter a claim, in the corrections log.
Limits
This is a small lab working on a small number of devices. Our results describe what we saw. They are not a survey of every model, and they are not a substitute for testing on the device in front of you. If you see something different, we want to hear about it through Ask the Lab.