German English

Block-based Load Balancing for Entity Resolution with MapReduce


Google Scholar

Kolb, L; Thor, A.; Rahm, E.
Block-based Load Balancing for Entity Resolution with MapReduce
Proc. 20th Intl. Conference on Information and Knowledge Management (CIKM), 2011


The effctiveness and scalability of MapReduce-based implementations of complex data-intensive tasks depend on an even redistribution of data between map and reduce tasks. In the presence of skewed data, sophisticated redistribution approaches thus become necessary to achieve load balancing among all reduce tasks to be executed in parallel. For the complex problem of entity resolution with blocking, we propose BlockSplit, a load balancing approach that supports blocking techniques to reduce the search space of entity resolution. The evaluation on a real cloud infrastructure shows the value and effctiveness of the proposed approach.



  • MapReduce, Hadoop
  • Entity Resolution, Object matching, Similarity Join, Pair-wise comparison
  • Clustering, Blocking
  • Data Skew, Load Balancing


  author    = {Lars Kolb and
               Andreas Thor and
               Erhard Rahm},
  title     = {{Block-based Load Balancing for Entity Resolution with MapReduce}},
  booktitle = {CIKM},
  year      = {2011},
  pages     = {2397-2400},
  ee        = {},
  crossref  = {DBLP:conf/cikm/2011},
  bibsource = {DBLP,}