Big Data has been the biggest news in the world of information technology for some time now. The data dump that technology generated and made accessible is a goldmine that no business can ignore. As we move into the coming year, it has firmly taken hold of businesses, part of our overall IT infrastructure is dealing with Big Data. In fact, if you are not already engaged in dealing with Big Data, you soon will be.
But like all emerging technologies, Big Data comes with its own set of challenges and issues. The reluctance to still board the Big Data bus displayed by some businesses in India is typical because of certain misconceptions. It has thrown up some interesting problems of functionality, technology assistance, and security.
The 3 Vs
16 years ago when we had not even coined the term, Gartner analyst Doug Laney came up with the concept of 3Vs in his seminal work 3D data management: Controlling data volume, variety, and velocity. It referred to the three defining properties of data management. As Big Data takes hold of our information systems, these three properties are as true today as they were before. In fact, many analysts have now added a few more Vs!
- Volume: Sure, any data storage will have big volumes of data, but what we are seeing here is an almost incomprehensible amount of data. Just imagine the millions of streams of texts, pictures, emails, messages that are pouring in every single minute and the mind boggles. Some call it a data dump since there is no way of quantifying it. As more and more of use multiple platforms and technologies to carry out every task, thought and action, this is the volume that is reaching colossal proportions. How do we sift through it? How do we segregate it? How do we pick what is functional and what is not?
- Velocity: Even more daunting than the volume of the data is its velocity or the speed at which data is generated. What seems like an incomprehensible amount today will most likely become insignificant in a month from now as more and more of this data flow in. For analysts, this is almost like standing at the end of a firehouse left at full blast. Sieving through this data to pick the relevant sections every day, is a challenge that they are still figuring out.
- Variety:The interesting (or irritating, depending on your outlook) thing about Internet data is the variety it represents. This is not a neat ledger with a few notes attached. Instead, we have a tsunami of tweets, posts, chats, emails, messages, comments, articles, attachments, videos, podcasts, and blogs. Add to that the layers that form on top with sensor data, encrypted posts and we have data that comes in every shape, size, and costume you can think of.
Data Warehousing Technology
As we see above, Big Data is simply too complex and vast to be effectively utilized by the software we are using for organizational purposes. For smaller business, this is usually the first stumbling block. Do we invest in data warehousing and mining technology or go with what we have? For those uncertain, it could be a good idea to start small and find out the benefits.
However, sooner or later they will have to move on to more specialized software such as Hadoop, and NoSQL and Map Reduce. These are open source technologies which can be scaled up. Many organizations opt to go the third-party way, outsourcing their Big Data work to specialists IT consultants.
The flip side of Big Data is the security threat it poses. We have already seen the volume and velocity at which data is pouring in can be difficult to handle for the best of specialists. As IoT picks up steam, this data will increase further. This makes the possibility of a security threat even more critical. There is a very marked possibility of cyber-attack flowing in through data packs. Encryption has only made this worse. Encrypted data is harder for the security team to crack and is ironical, used by cyber attackers as their Trojan horse.
There can be no doubt that Big Data has the immense use for a business. However, as an emerging technology, it also presents us with a number of challenges. How we, as a Big Data Services provider deal with them and move forward will determine the success of our Big Data strategy.
Rajnikant here as an experienced technical writer at Aegis Infoways since more than 5 years. I like to write technical articles especially for CRM, .Net, Hadoop, Java Development and Microsoft Dynamics 365.