Contact:Telephone: +49 5121 - 883 40348
Fax: +49 5121 - 883 40349
email contact form
Room: C 141 Spl - Gebäude C (Samelson-Campus)
Homepage: https://sse.uni-hildesheim.de/mitglieder/cui-qin/ Homepage
Cui Qin, Holger Eichelberger and Klaus Schmid
Enactment of Adaptation in Data Stream Processing with Latency Implications - A Systematic Literature Review
In: Information and Software Technology, 111: 1-21.
Free Download: https://authors.elsevier.com/a/1Yvdh3O8rCSPcx
[Context] Stream processing is a popular paradigm to continuously process huge amounts of data. Runtime adaptation plays a significant role in supporting the optimization of data processing tasks. In recent years runtime adaptation has received significant interest in scientific literature. However, so far no categorization of the enactment approaches for runtime adaptation in stream processing has been established. [Objective] This paper identifies and characterizes different approaches towards the enactment of runtime adaptation in stream processing with a main focus on latency as quality dimension. [Method] We performed a systematic literature review (SLR) targeting five main research questions. An automated search, resulting in 244 papers, was conducted. 75 papers published between 2006 and 2018 were finally included. From the selected papers, we extracted data like processing problems, adaptation goals, enactment approaches of adaptation, enactment techniques, evaluation metrics as well as evaluation parameters used to trigger the enactment of adaptation in their evaluation. [Results] We identified 17 different enactment approaches and categorized them into a taxonomy. For each, we extracted the underlying technique used to implement this enactment approach. Further, we identified 9 categories of processing problems, 6 adaptation goals, 9 evaluation metrics and 12 evaluation parameters according to the extracted data properties. [Conclusion] We observed that the research interest on enactment approaches to the adaptation of stream processing has significantly increased in recent years. The most commonly applied enactment approaches are parameter adaptation to tune parameters or settings of the processing, load balancing used to re-distribute workloads, and processing scaling to dynamically scale up and down the processing. In addition to latency, most adaptations also address resource fluctuation / bottleneck problems. For presenting a dynamic environment to evaluate enactment approaches, researchers often change input rates or processing workloads.
Holger Eichelberger, Cui Qin and Klaus Schmid
Experiences with the Model-based Generation of Big Data Applications
Lecture Notes in Informatics (LNI) - Datenbanksysteme für Business, Technologie und Web (BTW '17) - Workshopband
Developing Big Data applications implies a lot of schematic or complex structural tasks, which can easily lead to implementation errors and incorrect analysis results. In this paper, we present a model-based approach that supports the automatic generation of code to handle these repetitive tasks, enabling data engineers to focus on the functional aspects without being distracted by technical issues. In order to identify a solution, we analyzed different Big Data stream-processing frameworks, extracted a common graph-based model for Big Data streaming applications and developed a tool to graphically design and generate such applications in a model-based fashion (in this work for Apache Storm). Here, we discuss the concepts of the approach, the tooling and, in particular, experiences with the approach based on feedback of our partners.
Holger Eichelberger, Cui Qin and Klaus Schmid
From Resource Monitoring to Requirements-based Adaptation: An Integrated Approach
Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering Companion (ICPE '17)
In large and complex systems there is a need to monitor resources as it is critical for system operation to ensure sufficient availability of resources and to adapt the system as needed. While there are various (resource)-monitoring solutions, these typically do not include an analysis part that takes care of analyzing violations and responding to them. In this paper we report on experiences, challenges and lessons learned in creating a solution for performing requirements-monitoring for resource constraints and using this as a basis for adaptation to optimize the resource behavior. Our approach rests on reusing two previous solutions (one for resource monitoring and one for requirements-based adaptation) that were built in our group.
Holger Eichelberger, Claudia Niederée, Apostolos Dollas, Ekaterini Ioannou, Cui Qin, Grigorios Chrysos, Christoph Hube, Tuan Tran, Apostolos Nydriotis, Pavlos Malakonakis, Stefan Burkhard, Tobias Becker and Minos Garofalakis
Configure, Generate, Run - Model-based Development for Big Data Processing
European Project Space on Intelligent Technologies, Software engineering, Computer Vision, Graphics, Optics and Photonics - Volume 1: EPS Rome 2016
The development of efficient and robust algorithms for Big Data processing is a demanding task, which has to cope with the characteristics of this type of data (3Vs). Putting such algorithms as processing elements into larger pipelines adds an extra level of complexity, which can be alleviated by relying on a model-based approach including code generation. This allows data analysts to compose such pipelines on a higher level of abstraction, reducing the development effort as well as the risk of errors. In this chapter, we outline a model-based and adaptive approach to the development of data processing pipelines in heterogeneous processing contexts. It relies on a flexible, tool-supported approach to configuration, which embraces three levels: (a) a heterogeneous processing infrastructure - including reconfigurable hardware, (b) the pipelines as well as (c) the stakeholder applications built upon the pipelines. Furthermore, selected aspects of implementing the approach, which is validated in the context of the financial domain, are presented.
Cui Qin and Holger Eichelberger
Impact-minimizing Runtime Switching of Distributed Stream Processing Algorithms
Proceedings of the Workshops of the EDBT/ICDT 2016 Joint Conference
Stream processing is a popular paradigm to process huge amounts of data. During processing, the actual characteristics of the analyzed data streams may vary, e.g., in terms of volume or velocity. To provide a steady quality of the analysis results, runtime adaptation of the data processing is desirable. While several techniques for changing data stream processing at runtime do exist, one specific challenge is to minimize the impact of runtime adaptation on the data processing, in particular for real-time data analytics. In this paper, we focus on the runtime switching among alternative distributed algorithms as a means for adapting complex data stream processing tasks. We present an approach, which combines stream re-routing with buffering and stream synchronization to reduce the impact on the data streams. Finally, we analyze and discuss our approach in terms of a quantitative evaluation.
Impact-minimizing Runtime Adaptation in Cloud-based Data Stream Processing
Proceedings of the PhD Symposium at the 5th European Conference on Service-Oriented and Cloud Computing (ESOCC '16)
Recently, cloud-based data stream processing has been rapidly emerged. In particular, it has become competitive in high scalability for processing huge amounts of data. During such processing, the actual characteristics of data streams may vary, e.g., in terms of volume or velocity. To provide a steady quality of the analysis results, runtime adaptation of the data processing is desirable. While several techniques for changing data stream processing at runtime do exist, one specific challenge is to minimize the impact of runtime adaptation on the data processing, in particular for real-time data analytics. In this research work, we aim at performing a runtime adaptation in cloud-based data stream processing, namely, dynamically switching alternative distributed algorithms which has similar functionality but operate at different characteristics. The goal of this work is to provide a generic approach which can automatically determine the algorithm switch with quality guarantees on the data processing. To minimize the impact on the data streams as well as optimize resource usage during adaptation, we combine stream re-routing with buffering and stream synchronization along with a support of dynamic deployment of alternative stream processing algorithms into the cloud.
Holger Eichelberger, Cui Qin, Roman Sizonenko and Klaus Schmid
Using IVML to Model the Topology of Big Data Processing Pipelines
Proceedings of the 20th International Systems and Software Product Line Conference
pp. 204 - 208.
Creating product lines of Big Data stream processing applications introduces a number of novel challenges to variability modeling. In this paper, we discuss these challenges and demonstrate how advanced variability modeling capabilities can be used to directly model the topology of processing pipelines as well as their variability. We also show how such processing pipelines can be modeled, configured and validated using the Integrated Variability Modeling Language (IVML).
Holger Eichelberger, Cui Qin, Klaus Schmid and Claudia Niederée
Adaptive Application Performance Management for Big Data Stream Processing
In: Softwaretechnik-Trends, 35 (3): 35-37.
Big data applications with their high-volume and dynamically changing data streams impose new challenges to application performance management. Efficient and effective solutions must balance performance versus result precision and cope with dramatic changes in real-time load and needs without over-provisioning resources. Moreover, a developer should not be burdened too much with addressing performance management issues, so he can focus on the functional perspective of the system For addressing these challenges, we present a novel comprehensive approach, which combines software configuration, model-based development, application performance management and runtime adaptation.