Friday, October 23, 2015

How to install JDK in Linux

INTRODUCTION

This post will be the step by step guide to install the JDK in Linux Platform.
  1. DOWNLOAD

Download jdk-7u79-linux-x64.tar.gz file the oracle website.
  1. INITIAL SETUP

Create a new folder 'java' under /usr
  1. UNZIP THE DOWNLOADS

Unzip jdk-7u79-linux-x64.tar.gz to /usr/java
Now the folder structure would be  /usr/java/jdk1.7.0_79
  1. SET JAVA_HOME

    1. Check already set or not => $ echo $JAVA_HOME
    2. If Not, => $ export JAVA_HOME=/usr/java/jdk1.7.0_79
    3. Check JAVA_HOME now => $ echo $JAVA_HOME
/usr/java/jdk1.7.0_79
  1. SET CLASSPATH VARIABLE

  1. Check already set or not => $ echo $CLASSPATH
  2. If Not,  => $ export CLASSPATH=$JAVA_HOME'\lib'
  3. Check CLASSPATH now => $ $ echo $CLASSPATH
/usr/java/jdk1.7.0_79\lib
  1. SET PATH VARIABLE

  1. Check already set or not => $ echo $PATH
  2. If not, => $ export PATH=$JAVA_HOME'\bin'
by default there will be some PATH variable set already, then do as 6.4
  1. Check PATH now => $ $ echo $PATH
/usr/java/jdk1.7.0_79/bin
  1. If already PATH set, then $ export PATH=$PATH:$JAVA_HOME'/bin'
<existing path variables>:/usr/java/jdk1.7.0_79/bin
Now you will be able to run any java program in Linux. Please reply in comment box if you are facing any error during any of the above steps.

Tuesday, October 6, 2015

Hadoop Word Count Example using Maven and Eclipse

INTRODUCTION


Hadoop
  • Hadoop is a Framework written completely in Java.
  • Developed by Doug Cutting from Yahoo.
  • Hadoop 0.1.0 was released in April 2006.
  • It is an Open source project of the Apache Software Foundation.
  • Suitable for running in distributed environment & storage.
  • It will process very large data sets on computer clusters built from commodity hardware.
  • Hadoop framework is useful
    • When you must process lots of unstructured data.
    • When your processing can easily be made parallel.
    • When running batch jobs is acceptable.
    • When you have access to lots of cheap hardware.

Map-Reduce
  • It is the heart of Hadoop.
  • It is a programming model or Algorithm for data processing.
  • Hadoop can run MapReduce programs written in various languages (Java, Ruby, Python etc.,).
  • MapReduce programs are inherently parallel.
  • Works in Master-Slave Model.
  • Mapper Program
    • Performs filtering and sorting.
  • Reducer Program
    • Performs a summary operation.
HDFS (Hadoop Distributed File System)
  • Java-based file system to store a large volume of data.
  • Scalability of up to 200 PB of storage and a single cluster of 4500 servers.
  • Supporting close to a billion files and blocks.
  • Access
    • Java API
    • Python/C for Non-Java Applications
    • Web GUI through HTTP
    • FS Shell -  shell-like commands that directly interact with HDFS
  • Features are
    • HDFS can handle large data sets.
    • Since HDFS deals with large-scale data, it supports a multitude of machines.
    • HDFS provides a write-once-read-many access model.
    • HDFS is built using the Java language making it portable across various platforms.
    • Fault Tolerance and availability are high.

SOFTWARES & TOOLS

  1. Eclipse (Mars2)
  2. Maven
  3. Java 1.6

USECASE

Find the words and calculate word count in the input files of a directory. (Word Count)

IMPLEMENTATION

Step 1

Create a Maven project in Eclipse and update the below dependencies into pom.xml
<dependency>
 <groupId>jdk.tools</groupId>
 <artifactId>jdk.tools</artifactId>
 <scope>system</scope>
 <version>1.6</version>
 <systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
</dependency>

<dependency>
 <groupId>org.apache.hadoop</groupId>
 <artifactId>hadoop-common</artifactId>
 <version>2.7.1</version>
</dependency>

<dependency>
 <groupId>org.apache.hadoop</groupId>
 <artifactId>hadoop-core</artifactId>
 <version>1.2.1</version>
</dependency>

Step 2

Create a Mapper Class which should extend org.apache.hadoop.mapreduce.Mapper
package jbr.hadoopex;

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
 private final static IntWritable one = new IntWritable(1);
 private Text word = new Text();

 public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
   String line = value.toString();
   StringTokenizer tokenizer = new StringTokenizer(line);
   while (tokenizer.hasMoreTokens()) {
     word.set(tokenizer.nextToken());
     context.write(word, one);
   }
 }
}

Step 3

Create Reducer Class which extends org.apache.hadoop.mapreduce.Reducer
package jbr.hadoopex;

import java.io.IOException;
import java.util.Iterator;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
 public void reduce(Text key, Iterator<IntWritable> values, Context context) throws IOException, InterruptedException {
   int sum = 0;
   while (values.hasNext()) {
     sum += values.next().get();
   }
   context.write(key, new IntWritable(sum));
 }
}

Step 4

Create a Client Program which will call the Mapper and Reducer and show the output.
package jbr.hadoopex;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class WordCount {

 public static void main(String[] args) throws Exception {
   Configuration conf = new Configuration();

   Job job = new Job(conf, "wordcount");
   job.setJarByClass(WordCount.class);

   job.setOutputKeyClass(Text.class);
   job.setOutputValueClass(IntWritable.class);

   job.setMapperClass(WordCountMapper.class);
   job.setReducerClass(WordCountReducer.class);

   job.setInputFormatClass(TextInputFormat.class);
   job.setOutputFormatClass(TextOutputFormat.class);

   FileInputFormat.addInputPath(job, new Path(args[0]));
   FileOutputFormat.setOutputPath(job, new Path(args[1]));

   job.waitForCompletion(true);
 }
}

BUILD & DEPLOYMENT

  1. Create the Jar which includes all the above Mapper, Reducer & Client Programs
  2. Copy the Jar into HDFS
  3. Create an input folder into HDFS (/user/ranjith/mapreduce/input/)
  4. Create a file(words.txt) with some words and copy to input folder
  5. Create an output (/user/ranjith/mapreduce/output/) folder which will be used by Map-Reduce program and copy the output files into it.

INPUT

  1. Create two files under input folder
  2. Check the files
$ bin/hadoop dfs -ls /user/ranjith/mapreduce/input/
You will see the below files.
/user/ranjith/mapreduce/input/file01
/user/ranjith/mapreduce/input/file02
  1. Check the content of the file01
$ bin/hadoop dfs -cat /user/ranjith/mapreduce/input/file01
File01 content is:
Hello World Bye World
  1. Check the content of file02
$ bin/hadoop dfs -cat /user/ranjith/mapreduce/input/file02
File02 content is:
Hello Hadoop Goodbye Hadoop

RUN

Enter the below command to view the output.
> hadoop jar WordCount.jar jbr.hadoopex.WordCount /user/ranjith/mapreduce/input/words.txt /user/ranjith/mapreduce/output

  • hadoop - keyword to run the map-reduce program
  • jar - denotes the program archive file
  • jbr.hadoopex.WordCount - Client program
  • /user/ranjith/mapreduce/input/words.txt - Input file in HDFS
  • /user/ranjith/mapreduce/output - output location in HDFS

OUTPUT


Bye 1
Goodbye 1
Hadoop 2
Hello 2
World 2

ERRORS & SOLUTION

Below are the some of the errors which I have encountered during development and deployment of Hadoop Program. I have provided the solution for all those errors. Hope it will be helpful.

ERROR 1

ERROR:
> hadoop WordCount.jar jbr.hadoopex.WordCount /ranjith/learnings/mapreduce/testdata/input/emp.txt /hadoop/user/<USERNAME>/ranjith/learnings/mapreduce/testdata/output
Error: Could not find or load main class WordCount.jar

REASON
Missed to include 'jar' keyword

SOLUTION
Add 'jar' keyword after 'hadoop'
hadoop jar WordCount.jar jbr.hadoopex.WordCount /ranjith/learnings/mapreduce/testdata/input/emp.txt /hadoop/user/<USERNAME>/ranjith/learnings/mapreduce/testdata/output

ERROR 2

ERROR:
> hadoop jar WordCount.jar jbr.hadoopex.WordCount /ranjith/learnings/mapreduce/testdata/input/emp.txt /hadoop/user/<USERNAME>/ranjith/learnings/mapreduce/testdata/output
15/10/06 05:33:24 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/10/06 05:33:24 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
15/10/06 05:33:24 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/<USERNAME>/.staging/job_1443059102069_45474
15/10/06 05:33:24 WARN security.UserGroupInformation: PriviledgedActionException as:<USERNAME> (auth:SIMPLE) cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://ranjith/learnings/mapreduce/testdata/input/emp.txt
Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://ranjith/learnings/mapreduce/testdata/input/emp.txt
       at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:321)
       at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:264)
       at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:385)
       at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:589)
       at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:606)
       at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:490)
       at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
       at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
       at java.security.AccessController.doPrivileged(Native Method)
       at javax.security.auth.Subject.doAs(Subject.java:422)
       at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
       at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
       at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313)
       at jbr.hadoopex.WordCount.main(WordCount.java:68)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:483)
       at org.apache.hadoop.util.RunJar.main(RunJar.java:212)



SOLUTION
Move the input file into hdfs instead running from the local

ERROR 3

ERROR:
> hadoop jar WordCount.jar jbr.hadoopex.WordCount hdfs://ranjith/mapreduce/input/words.txt hdfs://ranjith/mapreduce/output
Exception in thread "main" java.lang.IllegalArgumentException: java.net.UnknownHostException: ranjith
       at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:373)
       at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:258)
       at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:153)
       at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:628)
       at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:574)
       at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:147)
       at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)
       at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
       at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
       at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
       at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
       at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
       at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
       at jbr.hadoopex.WordCount.main(WordCount.java:65)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:483)
       at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.net.UnknownHostException: ranjith
       ... 19 more


SOLUTION
Provide full hdfs path:
hadoop jar WordCount.jar jbr.hadoopex.WordCount /user/<USERNAME>/ranjith/mapreduce/input/words.txt /user/<USERNAME>/ranjith/mapreduce/output

ERROR 4

ERROR:
> hadoop jar WordCount.jar jbr.hadoopex.WordCount /user/<USERNAME>/ranjith/mapreduce/input/words.txt /user/<USERNAME>/ranjith/mapreduce/output
15/10/06 05:49:22 WARN security.UserGroupInformation: PriviledgedActionException as:<USERNAME>(auth:SIMPLE) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://user/<USERNAME>/ranjith/mapreduce/output already exists
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://user/<USERNAME>/ranjith/mapreduce/output already exists
       at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
       at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:554)
       at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:430)
       at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
       at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
       at java.security.AccessController.doPrivileged(Native Method)
       at javax.security.auth.Subject.doAs(Subject.java:422)
       at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
       at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
       at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313)
       at jbr.hadoopex.WordCount.main(WordCount.java:68)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:483)
       at org.apache.hadoop.util.RunJar.main(RunJar.java:212)


SOLUTION
I have created the output directory before running the program. It is not required. So, delete the output directory from HDFS and run again

ERROR 5

ERROR:
15/10/06 05:58:18 INFO mapreduce.Job: Task Id : attempt_1443059102069_45567_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class jbr.hadoopex.WordCountMapper not found
       at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2047)
       at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)
       at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:742)
       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
       at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
       at java.security.AccessController.doPrivileged(Native Method)
       at javax.security.auth.Subject.doAs(Subject.java:422)
       at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
       at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.ClassNotFoundException: Class jbr.hadoopex.WordCountMapper not found
       at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1953)
       at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2045)
       ... 8 more

SOLUTION
Add below Code in Main Program.
job.setJarByClass(WordCount.class);

Hope this example helps to create a Map-Reduce program in Hadoop Framework. Please share your comments below.
Happy Knowledge Sharing!!!