I have spotted the biggest bottleneck in "bdiff.c". Actually it was pretty easy to find after I recompiled the python interpreter and mercurial for profiling. In "bdiff.c" function "equatelines" allocates the minimum hash table size, which can lead to tons of collisions. I introduced an "overcommit" factor of 16, this is, I allocate 16 times more memory than the minimum value. Overcommiting 128 times does not improve the performance over the 16-times case.

adding dir/bar
adding foo
dir/bar
foo
adding dir/bar_2
adding foo_2
dir/bar_2
foo_2
adding a
adding c
adding b
adding d
removing a
removing c
recording removal of a as rename to b (100% similar)