In a dynamic global economy, companies and organizations have started to rely more and more on statistics gathered from their customers insights and behavior, internal processes and business operations with the aim of finding new opportunities for growth. In order to find and determine this information, large complex sets of data should be generated and analyzed by skilled professionals. The compilation of this large collection of data is known as “Big Data”.
The last years companies and organizations are more and more using Big Data to find new methods to improve the decisions they make, to discover new opportunities and improve the overall performance. For example, big data can be harnessed to address the challenges that arise when information that is dispersed across several different systems that are not interconnected by a central system. By aggregating data across systems, big data can help improve decision-making capability. It also can augment data warehouse solutions by serving as a buffer to process new data for inclusion in the data warehouse or to remove infrequently accessed or aged data. [Tech-Faq Website, 2013]
The big challenge of collecting all this data is to find solutions in how to be converted into useable information by identifying patterns and deviations from those patterns and many companies and organizations are working on it. Developers and software providers are rising to this challenge, turning big data management into a booming industry with major players in both private industry and open source communities. [Villanova University official website, 2013]
Security in Big Data
People create 2.5 quintillion bytes of data every day and almost 90% of the data in the world today has been created in the last two years alone. This means that security and privacy issues are getting bigger as well. The velocity, volume, and variety of big data are factors that affect these issues such as large-scale cloud infrastructures, diversity of data sources and formats, streaming nature of data acquisition and high volume inter-cloud migration. The old, traditional security mechanisms, which were used to secure small-scale static data all these year, are inadequate for the Big Data.
Large-scale cloud infrastructures become really popular and together with them a diversity of software platforms, which is spread across many computers and networks, has increased the surface for attacks on those systems.
For example, analytics for anomaly detection would generate too many outliers. Preventing distributed Denial of Service (DoS) attacks will become more difficult. [US-CERT Website].
Similarly, it is not clear how to retrofit provenance in existing cloud infrastructures. Streaming data demands ultra-fast response times from security and privacy solutions. [Integration Developer News official website, 2013]
Data and transaction logs (files that record and store any action by the database) are stored in multi-tiered storage...