German English

Dedoop: Efficient Deduplication with Hadoop

PDF

Google Scholar

Kolb, L.; Thor, A.; Rahm, E.
Dedoop: Efficient Deduplication with Hadoop
Proc. 38th Intl. Conference on Very Large Databases (VLDB) / Proc. of the VLDB Endowment 5(12), 2012
2012-08

Description

We demonstrate a powerful and easy-to-use tool called Dedoop (Deduplication with Hadoop) for MapReduce-based entity resolution (ER) of large datasets. Dedoop supports a browser-based specification of complex ER workflows including blocking and matching steps as well as the optional use of machine learning for the automatic generation of match classifiers. Specified workflows are automatically translated into MapReduce jobs for parallel execution on different Hadoop clusters. To achieve high performance Dedoop supports several advanced load balancing strategies.

Please visit our project website for further informations about Dedoop.


Poster



Keywords

  • MapReduce, Hadoop
  • Entity Resolution, Object matching, Similarity Join, Pair-wise comparison
  • Clustering, Blocking
  • Overlapping Clusters, Redundant-free comparisons
  • Data Skew, Load Balancing

BibTex

@article{DBLP:journals/pvldb/KolbTR12,
  author    = {Lars Kolb and Andreas Thor and Erhard Rahm},
  title     = {{Dedoop: Efficient Deduplication with Hadoop}},
  journal   = {PVLDB},
  volume    = {5},
  number    = {12},
  year      = {2012},
  pages     = {1878-1881},
  ee        = {http://vldb.org/pvldb/vol5/p1878_larskolb_vldb2012.pdf},
  bibsource = {DBLP, http://dblp.uni-trier.de}
}