I am happy to announce that the next major release of Shark, 0.2, is now available. Shark is a large-scale data warehouse system for Spark designed to be compatible with Apache Hive. It can answer Hive QL queries up to 30 times faster than Hive without modification to the existing data or queries. Shark supports Hive’s query language, metastore, serialization formats, and user-defined functions.

We released Shark 0.2 on Oct 15, 2012. The new version is much more stable and also features significant performance improvements. The full release notes are posted on Shark’s github wiki, but here are some highlights:

• Hive compatibility: Hive version support is bumped up to 0.9, and UDFs/UDAFs are fully supported and can be distributed to all slaves using the ADD FILE command.
• Simpler deployment: Shark 0.2 works with Spark 0.6’s standalone deployment mode, which means you can run Shark in cluster mode without depending on Apache Mesos. Also we worked on simplifying the deployment, and you can now set up a single node Shark instance in 5 mins, and launch a cluster on EC2 in 20 mins.
• Thrift server mode: Ram Sriharsha from Yahoo contributed a patch for the Shark Thrift server, which is compatible with Hive’s Thrift server.
• Performance improvements: We rewrote Shark’s join and group by code and workloads can observe 2X speedup.

In addition to the 0.2 release, we are working on the next major version, 0.3, expected to be released in November. Below is a preview of some of the features:

• Columnar compression: We are adding fast columnar data compression. You can fit more data into your cluster’s memory without sacrificing query execution speed.
• Memory Management Dashboard: We are working on a dashboard that shows key characteristics of the cluster, e.g. what tables are in memory versus on disk.
• Automatic optimizations: Shark will automatically determine the right degree of parallelism and users will not have to worry about setting configuration variables.

How many of our friends and neighbors are aware of California’s Proposition 30? Despite the high stakes and its potential impact on everyone from students and parents to business leaders, many Californians are not yet aware of it. Can social networks rapidly transmit knowledge about timely issues like this?

In 2009, the DOD launched an experiment to see if social networks could be mobilized to address time-critical tasks. Within three days, a team from MIT recruited thousands of people to help, and they successfully located the DOD’s 10 red weather balloons within nine hours. One key to their success was their recursive incentive mechanism.  These are well-known in marketing as pyramid schemes, but they are also useful to extract information from social networks. A Nature article published last week shows that social media, in particular messages from close friends, can have a small but significant influence on voting behavior which can make a difference in tight races. Polls show that voters who are aware of Proposition 30 are split at close to 50/50.

Can an intangible incentive structure be designed to rapidly mobilize citizens to learn about a pressing issue? Working with the AMP Lab and the CITRIS Data and Democracy Initiative, we are studying this question with a project to study how activity spreads across populations.

The website for the Proposition 30 Awareness Project allows visitors to register their awareness, then receive a custom web link to share with their friends and family using email, Facebook or Twitter. Visitors can return at any time to monitor their growing influence graph and track their influence score.   Influence is computed using a variant of the Kleinberg and Raghavan algorithm.

This experiment is the first step toward a general-purpose tool that will allow citizens to initiate their own awareness campaigns on any issue. The first example emphasizes awareness of Proposition 30 and does not advocate a position. It includes links to the California Voters Guide and to campaigns on both sides of the issue. Visitors may also indicate their position for or against the proposition and join an online discussion afterward.

We welcome you to participate in the study by visiting: http://tinyurl.com/prop30p

One of my favorite things about working in the AMPLAb is our focus on extending impact beyond the academic community and into the “real world”. This coming week, we will host the first UC Berkeley AMP Camp, a practitioner’s guide to cutting-edge large scale data management. The AMP Camp will feature hands-on tutorials on big data analysis using the AMPLab software stack, including Spark, Shark, and Mesos. These tools work hand-in-hand with technologies like Hadoop to provide high performance, low latency data analysis. AMP Camp will also include high level overviews of warehouse scale computing, presentations on several big data use-cases, and talks on related projects in the lab (see the full agenda).

AMP Camp will be live streamed directly from the U.C. Berkeley campus on Tuesday August 21 and Wednesday August 22nd. Participants will be able to engage in real time tutorials and interact with instructors directly through Piazza. Attending the live-streamed AMPCamp is completely free and requires only a brief registration. Around 200 attendees will also participate on-site (unfortunately on-site registration has sold out!).

Modern “big data” infrastructure increasingly consists of open-source projects with steep learning curves. Hands-on instruction is a critical prerequisite for understanding how to use these technologies and tie them together to extract value from massive data sets. Through AMP Camp and our broader community development initiatives, the AMPLab hopes to accelerate adoption of our Berkeley Data Analytics Stack (BDAS) and educate the community at large about data management.

We hope you all register for AMPCamp live stream – see you on the 21st!

Carat aims to detect energy bugs—app behavior that is consuming energy unnecessarily—using data collected from a community of mobile devices. As featured on TechCrunch, our mobile app generates personalized recommendations for improving battery life and is now available for iOS and Android devices.

Users install and run the Carat app on their mobile phone, tablet, or music player. The app intermittently takes measurements about the device and how it is being used. These measurements are sent to our servers, which perform a statistical analysis. The analysis, a Spark application running on Amazon Web Services, infers how devices are using energy and under what circumstances. After considering a few days worth of data, the personalized results are sent back to users in the form of actionable advice about how to improve their battery life.

Using data from our beta deployments, we have already identified thousands of energy bugs manifesting in the wild. Carat not only detects such misbehavior, but assists diagnosis by correlating with device features like the operating system version and what resources are available (e.g., WiFi or GPS). Some of these bugs caused afflicted users to see battery life that was shorter by hours, even compared to other users of the same app (but where the bug did not trigger). Users were told what misbehavior Carat found on their devices and how to fix it, such as by restarting the app or upgrading the OS.

We are excited to finally be live on both Apple’s App Store and Google’s Play Store. Please try out the app, leave positive ratings on the respective stores to help us spread the word, and send us any comments or questions. Thank you!

Matlab is a great language for prototyping ideas. It comes with many libraries specially for machine learning and statistics. But  when it comes to processing the natural language Matlab is extremely slow. Because of this, many researchers use other languages to pre-process the text, convert the text to numerical data and then bring the resulting data to Matlab for more analysis.

I used to use Java for this. I would usually tokenize the text with Java, then save the resulting matrices to the disk and read them in Matlab. After a while this procedure became cumbersome. I had to go back and forth between Java and Matlab, the procedure is prone to human errors and the codebase just  looks ugly.

Recently, together with Jason Chen, we have started to put together an NLP toolbox for Matlab. It is still a work in progress and we are still developing the toolbox but you can download the latest version from our github repository [link]. There is also an installation guide that helps you properly install it on your machine. I have built a simple map-reduce tool that allows you to utilize all of cores on the CPU for many functions.

So far the toolbox has modules for text tokenization (Bernoulli, Multinomial, tf-idf, n-gram tools), text preprocessing (stop word removal, text cleaning, stemming) and some learning algorithms (linear regression, decision trees, support vector machines and a Naïve Baye’s classifier). we have also implemented evaluation metrics (precession, recall, F1-score and MSE). The support vector machine tool is a wrapper around the famous LibSVM and we are working on another wrapper for SVM-light. A part-of-speech tagger is coming very soon too.

I have been focusing on getting different parts running efficiently. For example, the tokenizer uses Matlab’s native hashmap data structure (container maps) to efficiently pass over the corpus and tokenize it.

We are also adding examples and demos for this toolbox. The first example is a sentiment analysis tool that uses this library to predict whether a movie review is positive or negative. The code reaches the F1 score of 0.83, meaning that out of 200 movie reviews it made a mistake in classifying only 26 of them.

Please try the toolbox and note that it is still a work in progress, some functions are still slow and we are working to improve them. I would love to hear what you think. If you want us to implement something that might be useful to you just let us know.