Real-Time Rental Data Scraping and Analysis for Policy and Research: The Neighborhood Rental Database (NeRD)

• In Event: Innovative Rental Market and Neighborhood Data

Abstract

This paper introduces the Neighborhood Rental Database (NeRD), a novel data infrastructure designed to provide high-frequency, spatially granular insights into rental housing markets in the United States. Traditional data sources used to study rental markets—such as the American Community Survey or proprietary datasets—are often limited by substantial time lags, coarse geographic aggregation, or restricted access. These limitations constrain researchers’ and policymakers’ ability to monitor rapidly evolving market conditions, particularly in periods of economic volatility or policy change. NeRD addresses these gaps by systematically collecting and processing real-time rental listing data from online platforms to generate neighborhood-level measures of rents, availability, unit characteristics, and market dynamics. The primary goal of this project is to improve the measurement of rental market conditions and enable more responsive, evidence-based housing policy. By leveraging web scraping techniques and automated data pipelines, NeRD captures millions of rental listings over time, which are then cleaned, standardized, and harmonized into a longitudinal dataset.A key contribution of the project lies in the development of methods to address common challenges in listing data, including duplicate postings, inconsistent formatting across platforms, missing or noisy location information, and variation in how unit characteristics are reported. Listings are geocoded and linked to consistent geographic boundaries, enabling the construction of indicators at fine spatial scales such as census tracts or neighborhoods, while maintaining comparability over time. In addition to data cleaning and integration, the project develops protocols for tracking listings longitudinally to approximate unit turnover and duration on the market, as well as methods for constructing consistent measures of asking rents and unit composition across heterogeneous data sources. The resulting dataset is designed to be scalable across markets and flexible enough to incorporate new data sources as platforms evolve. Attention is also given to documenting coverage patterns and potential sources of bias in online listing data, including differential representation of units across price points, building types, and neighborhoods. In contrast to traditional sources, NeRD can capture short-term fluctuations and emerging trends that are otherwise obscured in annual or multi-year estimates. Beyond descriptive insights, NeRD provides a foundation for future causal research and policy evaluation. The timeliness and granularity of the data make it particularly well suited for studying the impacts of zoning reforms, tenant protections, and housing subsidy programs. Additionally, by improving visibility into rental markets in lower-income and underrepresented neighborhoods—areas often poorly captured in traditional datasets—NeRD contributes to addressing “data inequity” and supports more equitable policy design. This project demonstrates the potential of new data infrastructure to transform housing research and inform more timely and targeted policy responses.

Author

• Christina Plerhoples Stacy, Urban Institute
  Presenting Author