Documentation
Methodology
How the datasets are collected, standardised, checked and documented. Each dataset page adds the details that are specific to it.
Sources
Direct Dataset compiles its datasets from publicly available information: public records, published catalogues, advertised listings and published prices and reviews. Each dataset page names its source and states that Direct Dataset is independent of it.
No dataset contains information taken from behind a login, a paywall or a private account.
Updates and currency
There are two kinds of dataset in the catalogue:
- Continuous records, such as transactions, contracts and listing history, are updated daily or monthly. A delivery is current to the day your file is produced, and monthly, quarterly or daily updates can follow.
- Editions, such as a full product catalogue, are complete captures. A fresh edition is produced on request, and recurring captures build a history.
Record counts on this site are counts from the data, never estimates. Continuous datasets grow, so the exact count for your scope is stated in your quotation.
Change history
For sources that change every day, such as property listings, the source is recorded daily and each change is stored as a dated event: first seen, changed, delisted, relisted. The previous and new values are kept, so the state of the market on any past date can be rebuilt.
An item is marked as withdrawn only after it has been absent from complete daily captures. A gap in collection is therefore not recorded as a withdrawal.
Standardisation
- One set of category values across the full period of a dataset, so early and recent records compare directly.
- Stated units on every measure. Where a source uses its own units, they are kept and documented.
- Stable identifiers for records and for locations, so files join to each other and to later updates.
- Derived fields are labelled as derived, with the formula given in the data dictionary.
- Flat, related tables with typed columns. No embedded markup or nested structures in delivered files.
Quality checks
- Record counts by period are compared across captures to detect gaps.
- Duplicate records are removed on the stable identifier.
- Counts published on this site are measured from the data.
- A record count by period can be supplied with a quotation, to reconcile coverage against figures you already hold.
Scope and definitions
Every dataset page has a section called Scope and definitions. It states what each headline measure means, how derived fields are calculated and what the dataset covers. To test the data against your own use, ask for a sample cut to the areas, categories or dates you need.
Personal information
Datasets describe properties, products, prices and businesses. Where a source publishes business contact details or public display names, the dataset page says so, and a version without those fields can be delivered.
Corrections
If you find an error in a dataset or on this site, write to hello@directdataset.com. Confirmed errors are corrected.