Description

Entity resolution is a crucial step for data quality and data integration. Learning-based approaches show high effectiveness at the expense of poor efficiency. To reduce the typically high execution times, we investigate how learning-based entity resolution can be realized in a cloud infrastructure using MapReduce. We propose and evaluate two efficient MapReduce-based strategies for pair-wise similarity computation and classifier application on the Cartesian product of two input sources. Our evaluation is based on real-world datasets and shows the high efficiency and effectiveness of the proposed approaches.

Keywords

• MapReduce, Hadoop
• Entity Resolution, Object matching, Similarity Join, Pair-wise comparison
• Cartesian product
• Machine Learning, Classification

BibTex

@inproceedings{Kolb:2011:LER:2064085.2064087,
  author = {Kolb, Lars and K\"{o}pcke, Hanna and Thor, Andreas and Rahm, Erhard},
  title = {{Learning-based Entity Resolution with MapReduce}},
  booktitle = {Proceedings of the third international workshop on Cloud data management},
  series = {CloudDB '11},
  year = {2011},
  isbn = {978-1-4503-0956-1},
  location = {Glasgow, Scotland, UK},
  pages = {1--6},
  numpages = {6},
  url = {http://doi.acm.org/10.1145/2064085.2064087},
  doi = {http://doi.acm.org/10.1145/2064085.2064087},
  acmid = {2064087},
  publisher = {ACM},
  address = {New York, NY, USA},
  keywords = {Cartesian product, Entity Resolution, Machine Learning, MapReduce},
}