Architecture and Knowledge Support for Big Data Systems

Centre for Research on Engineering Software Technologies

Project Context: Big Data Systems (BDS) (i.e., data-intensive applications) have become one of the key priority areas for all sorts of organizations (i.e., private or public)[1],[2]. Nowadays public or private organizations are expected to leverage proprietary and open source data for different purposes such as business strategies, social networking, securing citizens and societies, and promoting scientific endeavors. To effectively and efficiently capture, curate, analyze, visualize and leverage such a large amount of data, a significant number of efforts are being invested to invent new and innovative techniques and technologies for supporting several functions of Big Data systems such as data capture and storage, data transmission, data curation, data analysis, and data visualization. One of the key challenges of designing, deploying, and evolving Big Data systems is designing and evaluating appropriate architectures that can support continuous development and deployment of Big-data systems. Hence, there is a vital need of developing and rolling out approaches and technologies for identifying, capturing critical knowledge and expertise, and making it available for transfer and reuse across various Big-Data systems projects.

We plan to build and evaluate a knowledge base to support the systematic design and evaluation of BDS. For this project, this knowledge base means reusable design knowledge and design artefacts and a tooling infrastructure for managing and sharing the knowledge and artefacts. The design knowledge will consist of a set of design principles, meta-models of describing BD systems’ core functional and non-functional properties, design patterns, other reusable design artefacts and intelligent algorithms to explore the available design artefacts.

[1] DRAFT NIST Big Data Interoperability Framework: Volumen 6 Reference Architecture, Draft Version 1, April, 2014.

[2] Data-intensive applications, challenges, techniques and technologies: A Survey on Big Data by C. L. P. Chen and C. Zhang, Information Sciences, 275, pp. 314-347, 2014.