[Pkg-haskell-maintainers] The core of the big data solutions -- Map

[75293192]

Title:       The core of the big data solutions -- Map
Author:      pengwenwei
Email:       pww71 at foxmail.com 
Language:    c++
Platform:    Windows, linux
Technology:  Perfect hash algorithm
Level:       Advanced
Description: A high performance map algorithm
Section      MFC c++ map stl
SubSection   c++ algorithm
License:     (GPLv3)

Map is widely used in c++ programs. And the bottleneck of program performance is often the performance of map. especially in big data field and the scenarios which can't realize data distribution and parallel processing. Therefore, the performance of map becomes the most critical technology.

My many years of work experience in telecommunition and information security industry Is all about big data analysis. especially in the information security industry, the data analysis is the most complicated. They can't work without map. For example, ip table, mac table, telephone numbers table, dns table etc.


Currently, the STL map and Google's hash map are the most popular maps. But they have some disadvantages. The STL map utilizes binary chop, which causes a bad performance. Google Hash map has the best performance at present, but it has probability of repeat collision. For big data analysis, the probability of repeat collision is unacceptable.

Now I would like to publish my algorithms. It includes three different maps for different scenarios:
1. Memory Map(memMap): It resides in memory and has a good access speed. But its size is limited by memory size.
2. Harddisk Map(diskMap): It utilizes hard disk to store data. So it could accept much more data than memory map.
3. Hashmap(hashMap): It has the best performance and a great lookup speed, but it doesn't have 'insert' and 'delete' functionality.

memMap and diskMap could be converted to hashMap by function memMap2HashMap and diskMap2HashMap. According to the test result, my algorithms' collision probability is zero. About performance, memMap has a comparable performance with google, and hashMap's performance is 100 times better than Google's hashmap.

In summary, my algorithms are perfect zero collision probability hash algorithms.You can refer to following artical to find the key index and compress algorithm theory:
http://blog.csdn.net/chixinmuzi/article/details/1727195

Source code and documents:
https://sourceforge.net/projects/pwwhashmap/files/?source=navbar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-haskell-maintainers/attachments/20150508/afa5c7d6/attachment.html>