QualiMaster - A configurable real-time date processing infrastructure mastering autonomous quality adaptation

The recent global financial crisis has shown, that even today, where large parts of the private and working life are supported and facilitated by information technology (IT), an early detection of cross-market risk situations is difficult. Especially in areas such as the financial sector, where in the stock market enormous amounts of data are processed immediately every day, IT faces there limits when it comes to the analyzes of big data. For example, up to 250 gigabytes of data (approximately 54 DVDs) are generated by current trading data and daily currency exchange rates in Europe and America. Country and cross-market risk analyzes, which are necessary for early intervention by central banks, require real-time analysis of this data and the ability to focus on individual phenomena, i.e. to process more data in a more detailed way in risk situations. This also includes data from social networks as, for example, the demise of Lehman-Brothers began with a rumor that the bank could not obtain their daily capital.

Today’s associated IT systems are designed for the maximum case, in which both the maximum amount of data flow and at the same time the maximum processing capacity is required. However, this is neither effective nor cost-effective since the capacity remains unused at times of lower data streams, instead of performing additional detailed analysis. The vision is the automatic and dynamic adaptation of such systems to the individual situation so that existing capacities are optimally utilized.

The group "Software Systems Engineering" by Prof. Dr. Klaus Schmid is working on methods and techniques for efficient software adaptation and how this adaptation can be carried out by software autonomously. With this and other work, the group has made a nationwide name and is therefore (in addition to the L3S Research Center Hanover, the Telecommunication Systems Institute of the Technical University of Crete and the companies Maxeler Technologies Ltd. in London and Spring Techno in Bremen) a partner in the research project "QualiMaster", which is funded by the EU with about 2.9 million Euros over three years. In this EU-project, the researchers of the working group focus in particular on the automatic configuration and adaptation of mechanisms for processing big data with respect to different quality characteristics.

With nearly 100 million messages per second at the stock market (Europe and America) the time factor is an important quality characteristic. Thus, the processing and analysis of these messages must not delay the receipt of results under agreed quality criteria for each recipient. The challenge of considering the relevant quality characteristics and performing an automatic adaptation of processes based on the weighting of these characteristics and the current amount of data will be part of the work of the SSE group in the project.

The researchers support the development of a configurable real-time data processing system for autonomous quality adjustment. In the long term, predictions about the development in the financial market (the analysis and prediction of systemic risk in the project) will be done using such a system. Further, this system can be used in other areas with large amounts of data, like macro-economic analyzes, weather analyzes, the analyzes of social networks, or large scientific experiments, for effectively and efficiently prediction mechanisms.

Further information:


Duration: 3 years

Contact: Dr. Holger Eichelberger, eichelberger(at)sse.uni-hildesheim.de

The QualiMaster-project is funded by the European Commission Grant 619525 , Area Scalable Data Analytics in the 7th Framework Programme. The EU is funding the project with approximately 2.9 million Euros.

Public Deliverables

D1.1Initial Use Cases and Requirements
D1.2Full Use Cases and Requirements
D2.1Approach for Scalable, Quality-aware Data Processing
D2.2Scalable, Quality-aware Data Processing Algorithms V1
D2.3Scalable, Quality-aware Data Processing Algorithms V2
D2.4Final report on Scalable, Quality-aware Data Processing Methods
D3.1Translation of Data Processing Algorithms to Hardware
D3.2Hardware-based Data Processing Algorithms V1
D3.3Hardware-based Data Processing Algorithms V2
D3.4Optimized Translation of Data Processing Algorithms to Hardware
D4.1Quality-aware Processing Pipeline Modelling
D4.2Quality-aware Processing Pipeline Adaptation V1
D4.3Quality-aware Processing Pipeline Adaptation V2
D4.4Quality-aware Processing Pipeline Modelling and Adaptation
D5.1QualiMaster Infrastructure Set-up
D5.2Basic QualiMaster Infrastructure
D5.3QualiMaster Infrastructure V1
D5.4QualiMaster Infrastructure V2
D6.1QualiMaster Applications V1 (internal)
D6.2Intermediary Evaluation Report
D6.3QualiMaster Applications V2 (internal)
D6.4Final Evaluation Report
D7.1Initial Project Fact Sheet
D7.2Project Presentation and Project Web Site
D7.3Dissemination Plan



The research leading to these results has received funding from the European Union Seventh Framework Programme [FP7/2007-2013] under grant agreement n° 619525.