Everyone knows that many entrepreneurs went to Stanford, but few know that UC Berkeley alumni are just as entrepreneurial.

For example, many would be surprised to learn that most startups are now locating much closer to Berkeley than they are to Palo Alto. A recent article in the New York Times listed the top 50 startups worldwide likely to become the next “unicorns” (billion dollar evaluation). Impressively, not only are half of them in the Bay Area, a third are in San Francisco by itself. (The rest: New York City 8, China 4, E.U. 3, Boston 2, Chicago 2, India 2, Southern California 2, Arlington 1, and Cape Town 1.)

I thought these potential unicorns might predict where future successful startups will locate, so to see where Silicon Valley is headed, I mapped the 25 Bay Area startups and their geographic center. It looks like the next heart of Silicon Valley is San Francisco. Histories of creating regions like Silicon Valley have often pointed to the critical role of great universities. So what great technology research university is closest to San Francisco? UC Berkeley, which is just a 20-minute BART trip away.

Location of 25 promising startups in the San Francisco Bay Area according to 8/23/15 NY Times

In Spring 2015, we ran an experimental course that explored the intersection of three trends related to the AMPLab mission and its role in educating students at the world’s best public university.

First, as the boundaries of computing grow, the scale and complexity of the problems that we tackle grows. This means that most impactful work is done collaboratively, and as we have seen in the AMPLab, across previously separate disciplines.

Second, as David Patterson pointed out in his NYT OpEd, there is need for long-term and sustained system building to tackle large-scale social problems that goes beyond one night hackathons.

Third, as David Corman from the NSF pointed out in his keynote at PerCom 2015, many students are now entering EECS departments with significant prior programming expertise, and we need to develop instructional techniques that can build on that base while challenging them and deepening their understanding.

In our course, we attempted to address all these issues by scaling up the”undergraduate assistant” concept 10 fold. We ran a small (~11 people) undergraduate research seminar that allowed students with skills across a variety of disciplines to work in small groups on various aspects of our e-mission research project.

In contrast to other undergraduate project classes, the students worked on three aspects of a single large-scale system, and the goal was to build a real, end to end system rather than a prototype.

The Center for Information Technology Research in the Interest of Society (CITRIS) ran a nice story on the class this week.
http://citris-uc.org/one-ph-d-students-mission-to-reduce-our-carbon-footprint/

Did it work? In many ways, yes. The exercise of preparing for the class – planning a syllabus, making the code ready for others to contribute, turning on automated tests – was a forcing function to help clarify the focus of the project. The students liked the class (rating: 6.6/7) and felt that it was useful (rating: 6.2/7). We built a system that works together end to end and generates results.

But it was definitely not perfect. The end to end system is startup/prototype quality. It is good enough to play around with possibilities, but it is not yet good enough to use for a real world evaluation.

The two biggest challenges were a reflection of those faced in any team setting, but magnified by the differences between the academic and industry environments.

First, this was just one class on students’ packed schedules and they had to prioritize it accordingly. But that made coordinating dependencies between components tricky – if students A and B were responsible for modules A and B, and B depended on A, but student A had a midterm one week, she was not going to work on module A until it was done. And by the time she could get to module A, it might be midterm time for student B.

Second, we had a strong Lake Woebegon effect – everybody needed to get an A. In a normal class setting, having some students turn in B or C level work does not affect others. But here, a student who tried to use B or C level work had to spend time cleaning it up before she could use it, which lead to resentment. This is true of all project classes, but was amplified here because all project teams were working on a single system.

Added in were a steep learning curve, different skill levels across the interdisciplinary teams, and a “testing takes too much time” attitude. We had to spend significant time and effort as the “glue” – establishing an overall framework structure, integrating all the disparate pieces at the end and forcing the development of at least minimal test cases.

If we were to do it again, it would be interesting to structure it as a year long course. That would help with the learning curve, and give us another semester to make the system robust and conduct a comprehensive evaluation. We would also plan for building more of the framework ourselves in order to reduce dependencies between students. It is an interesting pedagogical challenge to figure out how to do this while still giving students the experience of building something large and complex.

When might that happen? Not this Fall, for sure. How about next Spring or later? Only time will tell… :)

At the Scala By the Bay Meetup they showed a video featuring Matt Massie, Frank Nothaft, Matei Zaharia, and me talking about helping people in general and those with cancer in particular by developing open-source tools that rely on Spark and Scala for genetic processing, such as SNAP and ADAM . Here is the NY Times oped they refer to in the video.

We are happy to announce Splash: a general framework for parallelizing stochastic learning algorithms on multi-node clusters. If you are not familiar with stochastic learning algorithms, don’t worry! Here is a brief introduction to their central role in machine learning and big data analytics.

What is a Stochastic Learning Algorithm?

Stochastic learning algorithms are a broad family of algorithms that process a large dataset by sequential processing of random subsamples of the dataset. Since their per-iteration computation cost is independent of the overall size of the dataset, stochastic algorithms can be very efficient in the analysis of large-scale data. Examples of stochastic algorithms include:

• Stochastic Gradient Descent (SGD)
• Stochastic Dual Coordinate Ascent (SDCA)
• Markov Chain Monte Carlo (MCMC)
• Gibbs Sampling
• Stochastic Variational Inference
• Expectation Propagation.

Stochastic learning algorithms are generally defined as sequential procedures and as such they can be difficult to parallelize. To develop useful distributed implementations of these algorithms, there are three questions to ask:

• How to speed up a sequential stochastic algorithm via parallel updates?
• How to minimize the communication cost between nodes?
• How to design a user-friendly programming interface for Spark users?

Splash addresses all three of these issues.

What is Splash?

Splash consists of a programming interface and an execution engine. You can develop any stochastic algorithm using the programming interface without considering the underlying distributed computing environment. The only requirement is that the base algorithm must be capable of processing weighted samples. The parallelization is taken care of by the execution engine and is communication efficient. Splash is built on Apache Spark, so that you can employ it to process Resilient Distributed Datasets (RDD).

On large-scale datasets, Splash can be substantially faster than existing data analytics packages built on Apache Spark. For example, to fit a 10-class logistic regression model on the mnist8m dataset, stochastic gradient descent (SGD) implemented with Splash is 25x faster than MLlib’s L-BFGS and 75x faster than MLlib’s mini-batch SGD for achieving the same accuracy. All algorithms run on a 64-core cluster.

To learn more about Splash, visit the Splash website or read our paper. You may also be interested in the Machine Learning Packages implemented on top of Splash. We appreciate any and all feedback from early users!

We’ve written about machine learning pipelines in this space in the past. At the AMPLab Retreat this week, we released (live, on stage!) KeystoneML, a software framework designed to simplify the construction of large scale, end-to-end, machine learning pipelines in Apache Spark. KeystoneML is alpha software, but we’re releasing it now to get feedback from users and to collect more use cases.

Included in the package is a type-safe API for building robust pipelines and example operators used to construct them in the domains of natural language processing, computer vision, and speech. Additionally, we’ve included and linked to several scalable and robust statistical operators and machine learning algorithms which can be reused by many workflows.

Also included in the code are several example pipelines that demonstrate how to use the software to reproduce recent academic results in computer vision, natural language processing, and speech processing. Here’s an example:

Sample SIFT and Fisher Vector based Image Classification pipeline, included in KeystoneML.

This pipeline for image classification reproduces a recent academic result in image classification from Chatfield, et. al. when applied to the “VOC 2007” dataset, and is automatically distributed to run on a cluster.

Users familiar with the new spark.ml package may recognize several similarities in the API concepts and interfaces presented between these two projects. This is no coincidence since we contributed to the design and initial implementation of spark.ml. However, KeystoneML provides both a richer set of built-in operators – for example, image featurizers and large-scale linear solvers – and modifies the spark.ml interface to provide type-safety and ensure further robustness.

To try out KeystoneML, head over the the project home page and check out the quick start and programming guides, and check out the code on GitHub. We’d love to hear feedback from early users — please feel free to join the mailing list.

Our friends at Intel Shanghai are helping to spread the word about Spark!

AmpCamp@China 2015 will be held on May 23 2015 at Intel’s Zizhu Campus in Shanghai.

UC Berkeley AMPLab is not only a world leading research lab in big data and analytics, but have also produced BDAS (Berkeley Data Analytics Stack), an open source, next-generation data analytics stack that has wide adoption in both industry and research. Intel and AMPLab have long and close collaborations, especially on the open source development of many projects in BDAS (such as Spark, Shark, Tachyon, SparkR, GraphX, MLlib, etc.).

This is the first time an AMP Camp will take place abroad! Between 2010 and 2014, Berkeley AMPLab successfully held five Amp Camp events in the States; the last event (AMPCamp 5) had over 200 people on-site and over 1800 views of the live stream from over 40 countries. This year, Intel Shanghai is collaborating with AMPLab to present this hugely successful event to the big data industry, developer community and academia in China.

The collaboration with UC Berkeley AMPLab is a part of Intel’s larger open source efforts in big data and analytics. For years, Intel has been playing a key role in many open source big data projects (e.g., Hadoop, Spark, HBase, Tachyon, etc.), collaborating with the academia to apply advanced big data research to real-world problems, and partnering with the industry and community to build web-scale data analytics systems.

Come and join AmpCamp@China 2015 to hear about the cutting edge components of BDAS, meet the world-class big data engineers (including Apache project committers and PMC members) at Intel’s Zizhu Campus in Shanghai, and get a sneak peek of the latest research progress from AMP Lab researchers.

For more information, please visit the Intel Shanghai AMPCamp page.

The exercises and course work from AMPCamp 5 will be used at this event, and are found here.