Big data analytics = Big security

In the last years big data has become a hot subject in the field of information technology and gaining more attention from our senior management. Companies are working on this “new” concept of dealing with huge amounts of data and extracting the right information. With the importance of protecting all this data, we can’t ignore the attraction it has on security, including some of the security concerns. Big data is hot and together with security management it can become your most valued asset.

What is big data?

The definition of big data is to some extent still vague. Every company has a different limit when they consider their data sets to be “big data”. In general we can state that it’s big data when it isn’t easy anymore to store, process or transfer it. At this stage a company has to define how it will deal with it, including the use of storage arrays, data processing clusters and advanced tooling for querying and analyzing the data. Commonly we also need visualization tools to craft visual presentations of the huge amount of available data records and make them comprehensible to our users.

From a technical point of view we see a shift in the traditional RMDBS to move or mix with the so called “NoSQL” databases. NoSQL has a slightly different mindset and has (almost) no relations between records. It focuses on storing and provides a better performance when dealing with lot of data records. The tooling on top is responsible to query the right data and link it together, giving its special value.

Bigger and cheaper

Bandwidth, storage and computing power has increased exponentially in the last 10 years. At the same time the prices of all these factors greatly dropped. While this is great for companies and provides the ability to scale with the ongoing demand, it also opened new doors. For example we are able now to collect more user and system created data than ever before.

Every department has ideas on what they would like to store. Clear examples include the traffic patterns from the corporate website, accounting data from many systems and the huge amount of data created by users, like documentation. Also company specific research or mined data will be input for our big data set.

The real power of collecting all data becomes clear when we can combine datasets and link them together, into chains of individual events. For example how long it takes for a new customer to buy something online, the rating of this user online after dealing with our customer service department and the chance this same user comes back to us for a new buy.

Big data analytics

It won’t be a big surprise that storing all the data is a first step, but the real power and value is in analyzing it. With analytics we can define the value, by determining what information we would like to extract. Where previously we had only a single database to query, we now to have to start with a hypothesis, define a link between data records and then report on it. After initial testing we then can request the system for a full analysis, querying the whole data set. Although clever people have to assist in requesting the right data, visualization can be a great help in clarifying what is available and how to present it properly.

Using big data for security

With all information packed into one place, many departments are interested in getting their piece of the pie. Your marketing department, Research and Development, but also IT or corporate security might have great interest to solve their problems by querying the big data set. Since many people within different layers of the organization will show interest sooner or later, companies should be careful with providing full access.

The need for access by our employees also implies that we need a policy together with proper access controls. We should determine upfront who can access what, under what conditions and possibly even when they are allowed to do so. For some companies the value stored in the big data pile might be as high as their human assets, therefore safeguarding is essential. While big data is by some people still considered to be hype, the threat is real. Data leakage could be a disaster for the survival of the company.

While the usage and data analytics need protection, we have to think about using it ourselves. The data could include relevant information for security and risk management purposes, similar to a security information and event management (SIEM) solution. Depending on what data is stored, we can link the right events and determine existing and future threats. We can use it as input for our security metrics, creating management reporting or as part of a security dashboard.

Big data: secure by design

Management should consider the mandatory presence of security professionals when introducing big data analytics. The complexity of dealing with huge amounts of data, storing them and processing them is already high. Therefore the risk to oversee major security gaps is huge. Simply putting a security layer on top of an existing system is costly and might provide low protection to threats like data theft or espionage. Better is to design the solution with security in mind and embed the right security controls from the start. Examples include the several A’s: access controls, authentication, authorization, accounting and availability. The protection of the data and the involved security controls should be embedded, preferable within the hardware and software itself. Just protecting the borders (e.g. the transits) is simply not enough, especially when considering most implementations should be flexible, scalable and available to handle the ever growing amounts of data.

Since data is usually stored on big storage arrays, the requirements of big data solutions should be well defined. This includes requirements regarding performance, scalability and the encryption of data. With more companies working 24×7, also availability might be an important aspect. Security professionals should apply business impact analysis, risk and threat determination and support ongoing assessments, including functional and technical audits.


Big data is getting huge and very important for our future business. Sooner or later it will be one of our most valuable company assets. Threats include theft, unavailability and data leakage, therefore getting your security professionals involved at an early stage is key for the success of the project and adoption within the company.