Today data scientists can be deprived of their strengths when moving to larger datasets – datasets in the realm of ‘big data’ – because large scale tools are too inflexible to support the data science style of working.
Michael Stonebreaker, winner of the Turing Award 2014. said: ‘…the change will come when business analysts who work with SQL on large amounts of data give way to data scientists, which will involve more sophisticated analysis, predictive modelling, regressions and Bayesian classification.
That stuff at scale doesn’t work well on anyone’s engine right now. If you want to do complex analytics on big data, you have a big problem right now
