Virtual Memory Minimum Too Low

Get started with Hadoop: From evaluation to your first production cluster

Hadoop has come a long way in a relatively short time. Google papers on Google File System (GFS) and MapReduce inspired work on co-locating data storage and computational processing in individual notes spread across a cluster. Then, it was just over five years ago, in early 2006, that Doug Cutting joined Yahoo and set up a 300-node research cluster there, adapting the distributed computing platform that was formerly a part of the Apache Nutch search engine project. What began as a technique to index and catalog web content has extended to a variety of analytic and data science applications, from ecommerce customer segmentation and A/B testing to fraud detection, machine learning and medical research. Now, the largest production clusters are 4,000 nodes with about 15 petabytes of storage in each cluster. For example, at Yahoo they run over 42,000 Hadoop nodes storing 200 petabytes of data.

In just the nine months since I wrote an introduction to this emerging stack , it has become easier to install, configure and write programs to use Hadoop. Not surprisingly with an emerging technology, there is still work to do. As Tom White notes in his Hadoop: The Definitive Guide, Second Edition :

This piece provides tips, cautions and best practices for an organization that would like to evaluate Hadoop and deploy an initial cluster. It focuses on the Hadoop Distributed File System (HDFS) and MapReduce. If you are looking for details on Hive, Pig or related projects and tools, you will be disappointed in this specific article, but I do provide links for where you can find more information. You can also refer to the presentations at the Yahoo Developer Network Hadoop Summit 2011 on June 29, 2011 in Santa Clara, Calif., and Hadoop World 2011 , sponsored by Cloudera, in New York City on November 8-9, 2011.

Start with a free evaluation in stand-alone or pseudo-distributed mode

If you have not done so already, you can begin evaluating Hadoop by downloading and installing one of the free Hadoop distributions. The Apache Hadoop website offers a Single Node Setup guide .

You can start an initial evaluation by running Hadoop in either local stand-alone or pseudo-distributed mode on a single machine. You can pick the flavor of Linux you prefer, or use Solaris. In stand-alone mode, no daemons run; everything runs in a single Java virtual machine (JVM) with storage using your machine's standard file system. In pseudo-distributed mode, each daemon runs its own JVM but they all still run on a single machine, with storage using HDFS by default. For example, I'm running a Hadoop virtual machine in pseudo-distributed mode on my Intel-processor MacBook Pro, using VMWare Fusion, Ubuntu Linux, and Cloudera's Distribution including Apache Hadoop (CDH) version CDH3.

Mastering Windows XP Home Edition

SQL Server 2005 Bible

ITQ Level 2 IT Trouble-shooting for Users using Windows XP

PC Repair and Maintenance, A Practical Guide

The A+ Certification and PC Repair Handbook