DataStorm project is developed by University of Stavanger (UiS) and Stanford University (SU) and is based on previous BioSTORM project. It aims at developing integrated real-time monitoring framework for cloud-based systems.
BioSTORM (Biological Spatio-Temporal Outbreak Reasoning Module) project created ontology-driven framework for developing JADE agent systems. Project demonstrated that rich modeling environment that allowed for declarative model definition and its validation was a success in disease outbreak detection modeling. From the time BioSTORM was started streams of data to be analyzed have been rapidly increasing in size and amount. JADE-based deployment become a bottleneck of the framework.
Last year, UiS (with CloudCom.org) and SU have started creating new framework called DataStorm to address the problem of the previous version. Two main goals were: using Hadoop as a main deployment platform and restructuring the framework to facilitate generic data analysis. Initial version of the new framework is described in papers An Ontology-driven Framework for Deploying Cloud-based Systems (submitted for publication at The 2010 IEEE 3rd International Conference on Cloud Computing – CLOUD 2010) and An Integrated Ontology-based Framework for Intelligent Real-time Data Monitoring (submitted for publication at The First International Workshop on Cloud Computing Interoperability and Services – InterCloud 2010). It is planned to be released as open-source afterwords.
Test version runs on small 15 nodes Hadoop cluster at UiS with accessing nodes at UiS and SU. However, we have come to recognize that using Amazon Web Services (AWS) could improve efficiency off the framework, make it available to greater amount of researchers that do not want or are not able to run their own clusters and make collaboration between group of researchers easier.
Our current research plan consists of investigating Amazon CloudFront as a way to provide efficient and low latency...