BazEkon - The Main Library of the Cracow University of Economics

BazEkon home page

Main menu

Author
Marconi Gabriele (Sogeti Luxembourg)
Title
Content Removal Bias in Web Scraped Data: a Solution Applied to Real Estate Ads
Source
Open Economics, 2022, vol. 5, iss. 1, s. 30-42, wykr., bibliogr. 21 poz.
Keyword
Statystyka, Funkcje, Mieszkania
Statistics, Functions, Dwellings
Note
summ.
Abstract
I propose a solution to content removal bias in statistics from web scraped data. Content removal bias occurs when data is removed from the web before a scraper is able to collect it. The solution I propose is based on inverse probability weights, derived from the parameters of a survival function with complex forms of data censoring. I apply this solution to the calculation of the proportion of newly built dwellings with web scraped data on Luxembourg, and I run a counterfactual experiment and a Montecarlo simulation to confirm the findings. The results show that the extent of content removal bias is relatively small if the scraping occurs frequently compared with the online permanence of the data; and that it grows larger with less frequent scraping.(original abstract)
Full text
Show
Bibliography
Show
  1. Ascheri A., Kiss Nagy A., Marconi G., Meszaros M., Paulino R., Reis F. (2021). Competition in urban hiring markets: evidence from online job advertisements. Eurostat Statistical Working Papers, http://dx.doi.org/10.2785/667004
  2. Beblavý, M., Fabo, B. and Lenaerts, K. (2016). Demand for Digital Skills in the US Labour Market: The IT Skills Pyramid. CEPS Special Report No. 154, Available at SSRN: https://ssrn.com/abstract=3047102
  3. Brüning, N. and Mangeol, P. (2020). What skills do employers seek in graduates? Using online job posting data to support policy and practice in higher education. OECD Education Working Papers, No. 231, https://doi.org/10.1787/bf533d35-en10.1787/bf533d35-en
  4. Bricongne, J., Meunier, B. and Sylvain, P. (2021). Web Scraping Housing Prices in Real-time: the Covid-19 Crisis in the UK. Banque de France Working Paper No. 827, http://dx.doi.org/10.2139/ssrn.391619610.2139/ssrn.3916196
  5. Cole S., Hernán M. (2008). Constructing inverse probability weights for marginal structural models. American Journal of Epidemiology, 168, 656-64 doi: 10.1093/aje/kwn164.10.1093/aje/kwn164
  6. Cole, S. and Hernán, M. (2004). Adjusted survival curves with inverse probability weights. Computer Methods and Programs in Biomedicine, 75, 45-49, https://doi.org/10.1016/j.cmpb.2003.10.00410.1016/j.cmpb.2003.10.004
  7. Cox, D., Kartsonaki, C., & Keoghc, R. (2008). Big data: Some statistical issues. Statistics & Probability Letters, 136, 111-115, https://doi.org/10.1016/j.spl.2018.02.01510.1016/j.spl.2018.02.015
  8. Emmert-Streib, F., and Dehmer, M. (2019). Introduction to Survival Analysis in Practice. Machine Learning and Knowledge Extraction, 1, 1013-1038, https://doi.org/10.3390/make103005810.3390/make1030058
  9. Fabo, B. and Kurekova-Mytna, L. (2022). Forthcoming.
  10. Eiglsperger, M. (2019). New features in the Harmonised Index of Consumer Prices: analytical groups, scanner data and web-scraping. Economic Bulletin Boxes, 2, https://www.ecb.europa.eu/pub/economic-bulletin/html/eb201902~a070c3a338.en.html
  11. Han, S. and Anderson, C. (2021). Web scraping for hospitality research: Overview, opportunities, and implications. Cornell Hospitality Quarterly, 62, 89-104, https://doi.org/10.1177/193896552097358710.1177/1938965520973587
  12. Head, A., Lloyd-Ellis, H. and Sun, H. (2014). Search, Liquidity, and the Dynamics of House Prices and Construction. American Economic Review, 104, 1172-1210, https://doi.org/10.1257/aer.104.4.117210.1257/aer.104.4.1172
  13. Lee, E. and Wang, J. (2003). Statistical methods for survival data analysis. John Wiley & Sons.10.1002/0471458546
  14. Observatoire de l'habitat du ministère du Logement (2021). Simulateur des prix. Accessed on 05 February 2022 at https://logement.public.lu/fr/observatoire-habitat/prix-de-vente/simulateur.html
  15. OECD (2021), OECD Skills Outlook 2021: Learning for Life. OECD Publishing, https://doi.org/10.1787/0ae365b4-en10.1787/0ae365b4-en
  16. R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, https://www.R-project.org/
  17. Seaman S. and White I. (2013). Review of inverse probability weighting for dealing with missing data. Statistical Methods in Medical Research, 22, 278-295, https://doi.org/10.1177/096228021039574010.1177/0962280210395740
  18. Statec - Institut National de la Statistique et des Etudes Economiques du Grand-Duché du Luxembourg (2013). Note méthodologique sur la statistique des prix de vente des appartements. Observatoire de l'habitat du ministère du Logement, https://logement.public.lu/dam-assets/documents/observatoire-habitat/Methodologie-PrixVente-Appartements-2013-12.pdf
  19. Whitaker, S. (2018). Big Data versus a survey. The Quarterly Review of Economics and Finance, 67, 285-296, https://doi:10.1016/j.qref.2017.07.01110.1016/j.qref.2017.07.011
  20. Zhang, Z., and Sun, J. (2010). Interval censoring. Statistical methods in medical research, 19, 53-70. https://doi.org/10.1177/096228020910502310.1177/0962280209105023
  21. Zhelyazkova, N., and Ritschard, G. (2018). Parental leave take-up of fathers in Luxembourg. Population Research and Policy Review, 37, 769-793, https://doi.org/10.1007/s11113-018-9470-810.1007/s11113-018-9470-8
Cited by
Show
ISSN
2451-3458
Language
eng
URI / DOI
http://dx.doi.org/10.1515/openec-2022-0119
Share on Facebook Share on Twitter Share on Google+ Share on Pinterest Share on LinkedIn Wyślij znajomemu