These days the amount data we produce in forms emails, documents, pictures, videos, tweets etc. is growing at an unimaginable pace.  Often the data is stored online in sites like Facebook and we wish to have it available at our fingertips.    Have you wondered about the challenges that companies face with the amount of data that is being generated?  TiE Boston setup an expert panel  to discuss the  entrepreneurial opportunities in Big Data.

 TiE Boston setup an expert panel of academics and entrepreneurs  working in the area of Big Data on February 16^th, 2011 at the Microsoft New England R&D Center in Cambridge, MA.  Amrith Kumar, Founder & CTO, ParElastic Corporation moderated the following  panelists:
Daniel Abadi Ph.D, Assistant Professor of Computer Science, Yale University  
David Aronoff, General Partner, Flybridge Capital Partners  
Justin Sheehy, Chief Technical Officer, The Basho Team  
Daniel Weinreb, co-founder Symbolics, co-founder Object Design, Member Common Angels

When the data grows and becomes so large that a traditional system such as database management systems can no longer handle it then one has to develop customized solutions to work with the data. These data are called the Big Data and poses difficulties in capture, storage, search, analytics and visualization.  Academics and entrepreneurs are developing solutions to  aid companies to make effective use of the data .

Amrit Kumar initiated the session by posing pertinent questions to the panelists.  What is Big Data? What is coming next? Where are we going?

Professor Daniel Abadi of Yale shared his cutting edge research in developing Petascale(10^12) database systems based on Google’s Hadoop.  “The prototype we built approaches parallel databases in performance and efficiency, yet still yields the scalability, fault tolerance, and flexibility of MapReduce-based systems,” said Abadi.

Justin Sheehy said, “Big Data is coming now because we can do something with it.  We can gather and process data to produce useful business applications”. The opportunities lie both in the area of storage and making the data available, and processing the data to extract useful information.

While the stage is set for creative thinkers to do something big with Big Data, they should be aware of the theoretical limitations. Daniel Weinreb brought up the limitations imposed by the CAP Theorem.  According to the CAP Theorem it is impossible for a distributed system to provide the following three at the same time:   C –Consistency Consistency (all nodes see the same data at the same time), A - Availability (node failures do not prevent survivors from continuing to operate), and  P - Partition tolerance (the system continues to operate despite arbitrary message loss)

Finally the panel concluded the early adopters of the Big Data technologies would be the finance and marketing industry.   Thus business opportunities for entrepreneurs lie in this area.

|