I've been harping on the importance of GPUs since my October, 2012 blog post Supercomputing for $500 and more recently in my reviews here of the SC13 conference. A couple of news stories this month indicate broader recognition of the growing importance of "Big Compute".
More and more frequently we see organizations make the mistake of mixing and confusing team roles on a data science or "big data" project - resulting in over-allocation of responsibilities assigned to data scientists. For example, data scientists are often tasked with the role of data engineer leading to a misallocation of human capital. Here the data scientist wastes precious time and energy finding, organizing, cleaning, sorting and moving data. The solution is adding data engineers, among others, to the data science team.
Interesting article on how the Weather Company built a new modern data anlytical platform for both better predictions and better global service using Riak NoSQL database and AWS. The platform (SUN - Storage Utility Network) captures 2.25 billion weather data points15 times per hour.
"As with any large-scale, algorithmic-type modeling, the more data you have, the better the predictions will be..."
Magazine Luiza, one of the largest retail chains in Brazil, developed an in-house product recommendation system, built on top of a large knowledge Graph. AWS resources like Amazon EC2, Amazon SQS, Amazon ElastiCache and others made it possible for them to scale from a very small dataset to a huge Cassandra cluster. By improving their big data processing algorithms on their in-house solution built on AWS, they improved their conversion rates on revenue by more than 25 percent compared to market solutions they had used in the past.
The Data Science Association now has a LinkedIn group for DSA members only. To join the group, first become a member of the Data Science Association here on this website, and then apply to enter the LinkedIn group over on LinkedIn by supplying your username or e-mail there in your request.
For my final post on the SC'13 conference that ended this past Friday in Denver, there were two intriguing technologies discussed toward the end.
1. Micon Automata
When I linked my blog post from two days ago, Game changer for HPC: IBM and NVidia novel architecture, on Reddit, it was not well-received by some in the HPC community.
The big annoucnement at SC'13, the International Supercomputing conference sponsored by IEEE and ACM that is in its 25th year and that this year is in Denver, came from an IBM VP, speaking at the Nvidia booth. I believe the VP was Dave Turek.
New York University, the University of California, Berkeley and the University of Washington launch a 5-year, $37.8 million cross-institutional effort
Three core goals: