Umbrella-ds.com decision system architecture

Umbrella-ds.com Decision system architecture

Umbrella-ds.com is a decision system project. The project is at the beta step of 0.9 release. The graph above shows the main architecture entities of the software. As you can see this is a distributed architecture. One of the key factor of the architecture is that every modules could be parallelized. The use of multithreads and parallel architecture is a necessary step toward computation costs reduction. Instead of having one big computation power, the system required several reduced computation power, several average size servers. Today the system archive is around 4Millions press articles and the databases are around a cumulated 200gb size. The system run on a 4 processors architecture with 128gb ram. To obtain the maximum availability of data we should say that the system should be deployed on several tenth of servers.

Concerning the architecture, we can notice that the Decision system monitor is the main component. It is itself separate into import, export, results servers. It is a component designed by Umbrella-ds.com. The others main components are the crawler monitor which manage the archives process of urls and the crawlers servers. They are also Umbrella-ds.com software engineering work. The Web interface of data mining of the press archive is based on CMS inclusion library. The database servers and the files server are on the shelf open source components.

The next steps of Umbrella-ds.com design should be made with the deployment of the system on several other area of Internet, including United States press and Asia english press. It could also be deployed on middle east english press despite this geographical area is not Cooperation-iws.com main interest.

For more information you can contact me at olivier.avenel@free.fr or at http://www.cooperation-iws.com social networks.

 

Leave a Reply

Your email address will not be published. Required fields are marked *