# Methodology

> How Direct Dataset collects, standardises, checks and documents its datasets: sources, updates, change history, standardisation and quality checks.
>
> Web page: https://directdataset.com/methodology

How the datasets are collected, standardised, checked and documented. Each dataset page adds the details that are specific to it.

## Sources

Direct Dataset compiles its datasets from publicly available information: public records, published catalogues, advertised listings and published prices and reviews. Each dataset page names its source and states that Direct Dataset is independent of it.

No dataset contains information taken from behind a login, a paywall or a private account.

## Updates and currency

There are two kinds of dataset in the catalogue:

- **Continuous records**, such as transactions, contracts and listing history, are updated daily or monthly. A delivery is current to the day your file is produced, and monthly, quarterly or daily updates can follow.
- **Editions**, such as a full product catalogue, are complete captures. A fresh edition is produced on request, and recurring captures build a history.

Record counts on this site are counts from the data, never estimates. Continuous datasets grow, so the exact count for your scope is stated in your quotation.

## Change history

For sources that change every day, such as property listings, the source is recorded daily and each change is stored as a dated event: first seen, changed, delisted, relisted. The previous and new values are kept, so the state of the market on any past date can be rebuilt.

An item is marked as withdrawn only after it has been absent from complete daily captures. A gap in collection is therefore not recorded as a withdrawal.

## Standardisation

- One set of category values across the full period of a dataset, so early and recent records compare directly.
- Stated units on every measure. Where a source uses its own units, they are kept and documented.
- Stable identifiers for records and for locations, so files join to each other and to later updates.
- Derived fields are labelled as derived, with the formula given in the data dictionary.
- Flat, related tables with typed columns. No embedded markup or nested structures in delivered files.

## Quality checks

- Record counts by period are compared across captures to detect gaps.
- Duplicate records are removed on the stable identifier.
- Counts published on this site are measured from the data.
- A record count by period can be supplied with a quotation, to reconcile coverage against figures you already hold.

## Scope and definitions

Every dataset page has a section called Scope and definitions. It states what each headline measure means, how derived fields are calculated and what the dataset covers. To test the data against your own use, ask for a sample cut to the areas, categories or dates you need.

## Personal information

Datasets describe properties, products, prices and businesses. Where a source publishes business contact details or public display names, the dataset page says so, and a version without those fields can be delivered.

## Corrections

If you find an error in a dataset or on this site, write to [hello@directdataset.com](mailto:hello@directdataset.com?subject=Correction). Confirmed errors are corrected.

---

## All pages on this site, as Markdown

- [Home](https://directdataset.com/index.md): Direct Dataset is an independent provider of public data: documented, record-level datasets on property markets, retail prices and businesses.
- [Datasets](https://directdataset.com/datasets.md): The Direct Dataset catalogue: record-level datasets on property markets, retail prices and consumer products, each with coverage, data dictionary and method.
- [What teams do with the data](https://directdataset.com/use-cases.md): Worked examples of what research, analytics and pricing teams do with Direct Dataset data, each with real figures, the fields used and a sample query.
- [Findings from the data](https://directdataset.com/market-pulse.md): Market Pulse: short, dated findings from Direct Dataset data on property markets and retail prices, with figures and method. Free to quote with attribution.
- [Market statistics, free to quote](https://directdataset.com/statistics.md): Free market statistics from Direct Dataset: reference tables of prices and rents by area, category and brand, worked out from the full datasets. Free to quote.
- [Methodology](https://directdataset.com/methodology.md): How Direct Dataset collects, standardises, checks and documents its datasets: sources, updates, change history, standardisation and quality checks.
- [Licensing and delivery](https://directdataset.com/licensing.md): Licence terms for Direct Dataset data: internal use, research, publication and redistribution, with delivery formats, updates and how quotations work.
- [Data sourcing and compliance](https://directdataset.com/compliance.md): How Direct Dataset sources public data and handles personal information, with answers to the questions a procurement or legal review asks.
- [Cite the data](https://directdataset.com/cite.md): Citation formats and attribution wording for research papers, reports and news stories that use Direct Dataset data, with a citation for every dataset.
- [Custom data](https://directdataset.com/custom-data.md): Datasets collected to order by Direct Dataset: public information from a source you name, as a one-time file or a recurring delivery, with documentation.
- [About Direct Dataset](https://directdataset.com/about.md): Direct Dataset is an independent provider of public data. It gathers publicly available information and supplies it as documented, record-level datasets.
- [Contact](https://directdataset.com/contact.md): Contact Direct Dataset for a quotation, a sample file, a custom data brief or a press data request. All enquiries are handled by email.
- [Terms of use](https://directdataset.com/terms.md): Terms for using the Direct Dataset website. Dataset purchases are governed by the quotation and licence issued with each order.
- [Privacy](https://directdataset.com/privacy.md): How the Direct Dataset website handles personal information: no cookies, no tracking scripts and no forms. Enquiries are handled by email.
- [Dubai Property Sales Transactions](https://directdataset.com/datasets/dubai-property-sales-transactions.md): Record-level dataset of 1.36 million registered Dubai property sales from 1997 to today: price, size, property type, off-plan status and location.
- [Dubai Rental Contracts](https://directdataset.com/datasets/dubai-rental-contracts.md): Contract-level dataset of 8.8 million registered Dubai rental contracts from 2010 to today: rent, contract length, new lease or renewal, unit and location.
- [Retail Price History, US and Europe](https://directdataset.com/datasets/retail-price-history-us-europe.md): Daily lowest advertised prices for 7,063 consumer products in the US, UK, Germany, France, Spain and Italy: 4.1 million price points with the shop on each.
- [Sephora US Customer Reviews](https://directdataset.com/datasets/sephora-us-customer-reviews.md): 4.6 million Sephora US customer reviews since 2008 across 11,280 beauty products: rating, text, reviewer attributes, votes and brand replies.
- [Sephora US Product Catalogue and Price History](https://directdataset.com/datasets/sephora-us-product-catalogue.md): Sephora US product data: 11,682 products from 355 brands with prices, ratings, ingredient lists and categories, and a daily price history from February 2026.
- [Tesco UK Product Catalogue](https://directdataset.com/datasets/tesco-uk-product-catalogue.md): The full Tesco UK product catalogue: 864,749 products with prices, Clubcard offers, nutrition, ingredients, allergens and GTIN barcodes.
- [UAE Property Listings and Asking-Price History](https://directdataset.com/datasets/uae-property-listings-price-history.md): Daily record of UAE property listings on Bayut and Property Finder since June 2026: asking prices, dated price changes, delistings and time on market.
- [Dubai new-lease rents fell 8.6% in a year while renewal rents kept rising](https://directdataset.com/market-pulse/dubai-rental-market-october-2026.md): Dubai rents, October 2026: the median new apartment lease was AED 65,000, down 8.6% in a year like for like, while the median renewal rose to AED 60,000.
- [Dubai property prices and rents by area](https://directdataset.com/statistics/dubai-property-prices-and-rents-by-area.md): Dubai property statistics by area: typical sale prices, price per sq ft and yearly apartment rents by size, for the 40 busiest areas. Free to quote.
- [Skincare ingredient data from the Sephora US range](https://directdataset.com/use-cases/skincare-ingredient-data.md): Use Sephora US product data to analyse skincare ingredients: full ingredient lists for 2,328 products, with the share containing niacinamide and other actives.
- [Catalogue as JSON](https://directdataset.com/datasets.json)
- [Every page in one file](https://directdataset.com/llms-full.txt)
