Hadoop 시리즈. Yarn 구성해보기

Hadoop 시리즈. Hadoop3 설치하기 - 기본 설치

위 포스팅에 이어서 Yarn을 설치해보도록 하겠습니다.

 

노드	역할	Yarn
story-hadoop-master01	Active Namenode	Resource Manager
story-hadoop-master02	Standby Namenode	Resource Manager (history, timeline)
story-hadoop-master03	Observer Namenode
story-hadoop-worker01	Datanode	Node Manager
story-hadoop-worker02	Datanode	Node Manager
story-hadoop-worker03	Datanode	Node Manager

 

Step 1. 공통

yarn-site.xml

<?xml version="1.0"?>
<configuration>
	<property>
      <name>yarn.resourcemanager.ha.enabled</name>
      <value>true</value>
    </property>

    <property>
      <name>yarn.resourcemanager.cluster-id</name>
      <value>yarn-cluster</value>
    </property>

    <property>
      <name>yarn.resourcemanager.ha.rm-ids</name>
      <value>rm1,rm2</value>
    </property>

    <property>
      <name>yarn.resourcemanager.hostname.rm1</name>
      <value>story-hadoop-master01</value>
    </property>

    <property>
      <name>yarn.resourcemanager.hostname.rm2</name>
      <value>story-hadoop-master02</value>
    </property>

    <property>
      <name>yarn.resourcemanager.webapp.address.rm1</name>
      <value>story-hadoop-master01:8088</value>
    </property>

    <property>
      <name>yarn.resourcemanager.webapp.address.rm2</name>
      <value>story-hadoop-master02:8088</value>
    </property>

    <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
    </property>

    <property>
      <name>hadoop.zk.address</name>
      <value>story-hadoop-master01:2181,story-hadoop-master02:2181,story-hadoop-master03:2181</value>
    </property>

    <property>
      <name>yarn.timeline-service.entity-group-fs-store.active-dir</name>
      <value>/ats/active/</value>
    </property>

    <property>
      <name>yarn.timeline-service.entity-group-fs-store.done-dir</name>
      <value>/ats/done/</value>
    </property>

    <property>
      <name>yarn.timeline-service.webapp.address</name>
      <value>story-hadoop-master02:8188</value>
    </property>

    <property>
      <name>yarn.timeline-service.webapp.address.rm1</name>
      <value>story-hadoop-master01:8188</value>
    </property>

    <property>
      <name>yarn.timeline-service.webapp.address.rm2</name>
      <value>story-hadoop-master02:8188</value>
    </property>

    <property>
      <name>yarn.timeline-service.webapp.https.address.rm1</name>
      <value>story-hadoop-master01:8190</value>
    </property>

    <property>
      <name>yarn.timeline-service.webapp.https.address.rm2</name>
      <value>story-hadoop-master02:8190</value>
    </property>
</configuration>

 

Step 2. Resource Manager

Step 2-1. 준비

hadoop fs -mkdir /ats
hadoop fs -mkdir /ats/active
hadoop fs -mkdir /ats/done

 

Step 2-2. 실행

# 실행
yarn --daemon (start|stop|status) --config /etc/hadoop/conf resourcemanager

 

Step 2-3. historyserver

mapred --daemon start historyserver

{ 히스토리 실행 서버 }:19888 URI로 접근을 시도하면 됩니다.

 

Step 2-4. timeline server

yarn --daemon start timelineserver

{ 타임라인 실행 서버 }:8188 URI로 접근을 시도하면 됩니다.

 

Step 2-5. 확인

$ jps
690 JournalNode
931 NameNode
4038 JobHistoryServer
4262 Jps
1913 DFSZKFailoverController
3450 ResourceManager
28221 QuorumPeerMain
4175 ApplicationHistoryServer

 

Step 2-6. Port 정보 참조

YARN Service Ports - Hortonworks Data Platform

 

Step 3. Node Manager

Step 3-1. 실행

# 실행
yarn --daemon (start|stop|status) --config /etc/hadoop/conf nodemanager

 

Step 3-2. 확인

$ jps
15030 NodeManager
15128 Jps
14812 DataNode

 

Step 3-3. 테스트

$ yarn jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 50 100
Number of Maps  = 50
Samples per Map = 100
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Wrote input for Map #10
Wrote input for Map #11
Wrote input for Map #12
Wrote input for Map #13
Wrote input for Map #14
Wrote input for Map #15
Wrote input for Map #16
Wrote input for Map #17
Wrote input for Map #18
Wrote input for Map #19
Wrote input for Map #20
Wrote input for Map #21
Wrote input for Map #22
Wrote input for Map #23
Wrote input for Map #24
Wrote input for Map #25
Wrote input for Map #26
Wrote input for Map #27
Wrote input for Map #28
Wrote input for Map #29
Wrote input for Map #30
Wrote input for Map #31
Wrote input for Map #32
Wrote input for Map #33
Wrote input for Map #34
Wrote input for Map #35
Wrote input for Map #36
Wrote input for Map #37
Wrote input for Map #38
Wrote input for Map #39
Wrote input for Map #40
Wrote input for Map #41
Wrote input for Map #42
Wrote input for Map #43
Wrote input for Map #44
Wrote input for Map #45
Wrote input for Map #46
Wrote input for Map #47
Wrote input for Map #48
Wrote input for Map #49
Starting Job
Job Finished in 3.784 seconds
Estimated value of Pi is 3.14160000000000000000