Recent blog postsBig Data Illustrated
Big Data; A few words on velocity and volume
In my ongoing discourse on "Big Data", I have so far briefly touched upon the underpinnings of what a Big Data solution might look like in a very general fashion. Yet I think it suffices to say that once we have determined what data model for which we are aggregating data, we can make some observations and lay design criteria for the data structure and the memory architecture that will undergird our solution. I'll introduce three analyses modes which will determine the construction of the memory architecture, memory handling processes and computing resources: (1) Forensic, (2) Near real-time and (3) Real-time. These distinctions and their ramifications will become apparent when the solution time scale is the prime driver. Let's review some memory model and architecture basics before we wade into deep water!
Big Data - The data model must represent reality!
Often when "Big Data" is mentioned, there arises an image of a massive amount of data being collected, manipulated, analyzed, with the resultant information categorized and stored. With the resultant information acting as the oracle that always gives the right answers, whatever the questions might be. Yet what we learned in the earliest days of programming still holds true: GIGO. This short note examines the role of choosing an appropriate data model and how that affects how we collect and process the data.