Data Warehousing is close to my heart as I\u2019ve spent a significant part of my career building, modelling or architecting large and complex data warehouses and analytics systems. If I am not building one, I am certainly using one. Finding the data sources, cleaning and loading them into structured star schemas and deriving intelligence has been great fun, besides solving some unique problems along the way.<\/p>\n
The data that we handled (a terabyte a month) then in comparison to the data that we are handling now (a terabyte a day) just tells the story in terms of data explosion. 12 TB of tweets and about 500 million call records every day are just a few things that I keep reading from time to time.<\/p>\n
The more I look back, however, the more I feel that the Data warehouse approach was (and still is) too expensive, too complicated and too slow to get the data to where it\u2019s needed. The thought of waiting for the data to get into a structured database to even begin acting on it is like waiting for the vegetables to be carved up intricately before it\u2019s served. A certain amount of latency was understood by the business as necessary in the case of a Data warehouse.<\/p>\n
And the worst part is that some data (e.g. call recordings or unstructured email messages) that don\u2019t lend themselves to traditional relational databases and cannot be dealt with in traditional ways are simply brushed aside or in some cases expensive technologies are used to mine them separately. A certain amount of constraint (e.g. space, structured data etc.) is again understood as necessary to operate a Data warehouse.\u00a0 Some would argue these things can easily be fixed (like throwing more hardware). Well, I agree, if you have deep pockets.<\/p>\n
Data warehousing, and in particular relational databases, for the last two decades has been seen as the only solution for pulling together disparate data sources to make business sense, until of course Google and Facebook came along and disrupted things a bit with Big Data technologies. They have led the way for a new form of storing and searching data that don\u2019t use any databases, runs on cheap commodity hardware and the best part is: it doesn\u2019t cost half as much and is not constrained by size, volume, where the data lives or whether the data is structured or unstructured.<\/p>\n
Well then, is this the end of the road for big Nellie? Well, with some reservation, we think it is. We say this with much excitement, of course, as it will give way to new technologies and will force the traditional database vendors to innovate (to be fair they already are!). We don\u2019t think the Data warehouses or relational databases would be out any time soon and there will always be a place for it standing shoulder to shoulder other Big Data technologies. \u00a0In future, however, data warehousing or relational databases may not be the default choice or the only choice available to data crunchers.<\/p>\n","protected":false},"excerpt":{"rendered":"
Data Warehousing is close to my heart as I\u2019ve spent a significant part of my career building, modelling or architecting large and complex data warehouses and analytics systems. If I am not building one, I am certainly using one. Finding the data sources, cleaning and loading them into structured star schemas and deriving intelligence has […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_s2mail":""},"categories":[1],"tags":[18,5,8,19],"yoast_head":"\n