hadoop 설치

System Engineering Lab.

Taehoon Kim

Index

Hadoop Settup

Hadoop Example (wordcount)

Hadoop Programing (HDFS)

2014-09-24 2 Sejong System Engineering Lab

How to install Hadoop?

• 설치 환경 – CentOS 6.4 64bit

– JDK-8

– Hadoop-1.2.1



1. 인증 없이 SSH 원격 접속을 위한 RSA-key 전달

2. JDK 설치

3. HDFS, MapReduce, 환경변수 등 설정



• JDK 설치 및 시스템 환경변수 설정 #wget Oracle-JDK-8

#tar xvfz jdk-8.tar.gz

#ln –s ~/jdk-8 /usr/local/java

#vim /etc/profile (last line add) export JAVA_HOME=/usr/local/java

export PATH=$JAVA_HOME/bin:$PATH

export CLASS_PATH=$JAVA_HOME/lib:$CLASS_PATH

:wq

#source /etc/profile

#java -version


http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html






• Hostname 설정 #hostname

#vim /etc/sysconfig/network HOSTNAME=hadoop1

:wq

#reboot & hostname

• Domain name 설정 #vim /etc/hosts 210.107.232.81<tap>hadoop1

210.107.232.82<tap>hadoop2

:wq



• 인증 없이 SSH 원격 접속을 위한 RSA-key 전달

#ssh-keygen Enter file in which to save the key (/root/.ssh/id_rsa): <enter>

Enter passphrase (empty for no passphrase): <enter>

Enter same passphrase again: <enter>

#ls /home/.ssh id_rsa id_rsa.pub known_hosts

#ssh-copy-id –i ~/.ssh/id_rsa.pub root@[$HOSTNAME] (yes/no)? <yes>

password: <passwd>

#ssh root@hadoop2 암호를 묻지 않고 접속하면 성공



• Hadoop 설치 및 HDFS, MapReduce 환경 설정 #wget hadoop-1.2.1.tar.gz.

#tar xvfz hadoop-1.2.1.tar.gz

#ln –s ~/hadoop-1.2.1/ /usr/local/hadoop

#cd hadoop-1.2.1/conf

#ls

... core-site.xml hdfs-site.xml mapred-site.xml master slave

Next step..


http://apache.tt.co.kr/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz




• Hadoop home 설정 #vi /etc/profile export HADOOP_INSTALL=/usr/local/hadoop

export PATH=$HADOOP_INSTALL/bin:$PATH

:wq

#source /etc/profile

• hadoop-env.sh 설정 #vi hadoop-env.sh export JAVA_HOME=/usr/local/java

:wq



• core-site 설정 #vi core-site.xml <configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://hadoop1:9000</value>

</property>

</configuration>



• hdfs-site 설정 #vi hdfs-site.xml <configuration>

<property>

<name>dfs.name.dir</name>

<value>/usr/local/hadoop/name</value>

</property>

<property>

<name>dfs.data.dir</name>

<value>/usr/local/hadoop/data</value>

</property>

</configuration>



• mapred-site 설정 #vi mapred-site.xml

<configuration>

<property>

<name>mapred.job.tracker</name>

<value>hdfs://hadoop1:9001</value>

</property>

<property>

<name>mapred.system.dir</name>

<value>/usr/local/hadoop/mapred/system</value>

</property>

<property>

<name>mapred.local.dir</name>

<value>/usr/local/hadoop/mapred/local</value>

</property>

</configuration>



• SeondaryNamaNode 설정 #vi masters hadoop1

• datanode, taskTracker 설정 #vi slave hadoop1

hadoop2



• Hadoop 실행 //namenode에서만 실행

#cd bin

#hadoop namenode –format

#start-all.sh

#jps (namenode)

JobTracker

DataNode

NameNode

SecondaryNameNode

Jps

TaskTracker


#jps (datanode)

DataNode TaskTracker Jps

Example WordCount

• Hadoop 실전 WordCount 동작 #hadoop fs –copyFromLocal Harry_Potter* test

#hadoop fs –ls test -rw-r--r-- 2 root supergroup 461676 2014-09-17 15:40 /user/root/test/Harry_Potter_1.txt

-rw-r--r-- 2 root supergroup 504398 2014-09-17 15:40 /user/root/test/Harry_Potter_2.txt

#hadoop jar hadoop-examples-1.2.1.jar wordcount test result 14/09/17 15:47:02 INFO mapred.JobClient: map 0% reduce 0%

14/09/17 15:47:09 INFO mapred.JobClient: map 50% reduce 0%




14/09/17 15:47:20 INFO mapred.JobClient: Job complete: job_201409171536_0001

#hadoop fs –cat result/part-r-00000


Hadoop Programing

1. Install Eclipse & Project setting

2. Ant setting

3. Hadoop Programing


Hadoop Programing

• Install Eclipse www.eclipse.org 에서 Eclipse Standard 4.4 다운로드

압축 해제 후 이클립스 실행

Workspace 설정 후 프로젝트 새


http://www.eclipse.org/

Hadoop Programing

2014-09-24 Sejong System Engineering Lab 20

New -> Java Project Click!

Next Click !

1. Install Eclipse & Project setting

Hadoop Programing


Add External JARs.. Click!

OK Click !

Hadoop에서 제공하는 jar 파일을 모두 추가함 - HADOOP_INSTALL/*.jar - HADOOP_INSTALL/lib/*.jar

**하위 디렉터리에 있는 jar파일도 모두 추가**

Hadoop Programing


2. Ant setting - 해당 프로젝트에서.. 1. build 디렉터리 생성 2. build.xml 파일 생성 3. build.xml 내용 추가 (우측 코드 참조)

4. build.xml 우측 클릭 5. Run As 선택 -> Ant Build 실행

Hadoop Programing


3. Hadoop Programing - 해당 프로젝트에서.. 1. 새 프로젝트 생성 2. jar에 있는 클래스 import 3. configuration 설정 4. 권한 설정 5. Run As 선택 -> Ant Build 실행

Hadoop Programing

• Ant로 컴파일 된 jar 파일을 Hadoop 명령어로 실행 #hadoop jar wikibooks-hadoop-exampless.jar SingleFileWriteRead hello.txt HelloHadoop $$ inputString:HelloHadoop

#hadoop fs –ls test hello.txt -rw-r--r-- 2 root supergroup 461676 2014-09-17 15:40 /user/syseng/hello.txt

#hadoop fs –cat hello.txt HelloHadoop


Thank you!


hadoop 설치

Software