There are multiple tools which claim option to decompress bzip2 in parallel:
- pbzip2
- lbzip2
Let's compare pbzip2 performance with reference singe thread bzip2:
$ time bzip2 -d /tmp/rib.bz2 --stdout > /dev/nullreal 0m52.188suser 0m52.019ssys 0m0.160s$ time pbzip2 -d /tmp/rib.bz2 --stdout > /dev/nullreal 0m49.380suser 0m49.473ssys 0m0.241s
You may notice that we have no speed improvement at all which means that pbzip2 cannot do decompression in parallel for standard bz2 compressed files.
But lbzip2 actually can do it and it offers great performance improvement:
$ time bzip2 -d /tmp/rib.bz2 --stdout > /dev/nullreal 0m52.790suser 0m52.549ssys 0m0.224s$ time lbzip2 -d /tmp/rib.bz2 --stdout > /dev/nullreal 0m8.604suser 1m8.099ssys 0m0.420s
It's 9 seconds vs 53 seconds. It's 6 times improvement on 8 CPU server.
Conclusions: use lbzip2 for parallel decompression.
No comments :
Post a Comment
Note: only a member of this blog may post a comment.