이야기박스

Hadoop 시리즈. Yarn 구성해보기 본문

Computer & Data/Big Data

Hadoop 시리즈. Yarn 구성해보기

박스님 2021. 9. 26. 15:27
반응형

 

Hadoop 시리즈. Hadoop3 설치하기 - 기본 설치

이전 포스팅에서 하둡, 특히 HDFS란 무엇인가에 대해서 가볍게 알아보았다면, 이번에는 직접 설치해여 몸으로 하둡을 겪어보려고 합니다. Hadoop 시리즈. HDFS 맛보기 하둡 분산형 파일 시스템 (HDFS;

box0830.tistory.com

위 포스팅에 이어서 Yarn을 설치해보도록 하겠습니다.

 

노드 역할 Yarn
story-hadoop-master01 Active Namenode Resource Manager
story-hadoop-master02 Standby Namenode Resource Manager (history, timeline)
story-hadoop-master03 Observer Namenode  
story-hadoop-worker01 Datanode Node Manager
story-hadoop-worker02 Datanode Node Manager
story-hadoop-worker03 Datanode Node Manager

 

Step 1. 공통

yarn-site.xml

더보기 
<?xml version="1.0"?>
<configuration>
	<property>
      <name>yarn.resourcemanager.ha.enabled</name>
      <value>true</value>
    </property>

    <property>
      <name>yarn.resourcemanager.cluster-id</name>
      <value>yarn-cluster</value>
    </property>

    <property>
      <name>yarn.resourcemanager.ha.rm-ids</name>
      <value>rm1,rm2</value>
    </property>

    <property>
      <name>yarn.resourcemanager.hostname.rm1</name>
      <value>story-hadoop-master01</value>
    </property>

    <property>
      <name>yarn.resourcemanager.hostname.rm2</name>
      <value>story-hadoop-master02</value>
    </property>

    <property>
      <name>yarn.resourcemanager.webapp.address.rm1</name>
      <value>story-hadoop-master01:8088</value>
    </property>

    <property>
      <name>yarn.resourcemanager.webapp.address.rm2</name>
      <value>story-hadoop-master02:8088</value>
    </property>

    <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
    </property>

    <property>
      <name>hadoop.zk.address</name>
      <value>story-hadoop-master01:2181,story-hadoop-master02:2181,story-hadoop-master03:2181</value>
    </property>

    <property>
      <name>yarn.timeline-service.entity-group-fs-store.active-dir</name>
      <value>/ats/active/</value>
    </property>

    <property>
      <name>yarn.timeline-service.entity-group-fs-store.done-dir</name>
      <value>/ats/done/</value>
    </property>

    <property>
      <name>yarn.timeline-service.webapp.address</name>
      <value>story-hadoop-master02:8188</value>
    </property>

    <property>
      <name>yarn.timeline-service.webapp.address.rm1</name>
      <value>story-hadoop-master01:8188</value>
    </property>

    <property>
      <name>yarn.timeline-service.webapp.address.rm2</name>
      <value>story-hadoop-master02:8188</value>
    </property>

    <property>
      <name>yarn.timeline-service.webapp.https.address.rm1</name>
      <value>story-hadoop-master01:8190</value>
    </property>

    <property>
      <name>yarn.timeline-service.webapp.https.address.rm2</name>
      <value>story-hadoop-master02:8190</value>
    </property>
</configuration>

 

Step 2. Resource Manager

Step 2-1. 준비

hadoop fs -mkdir /ats
hadoop fs -mkdir /ats/active
hadoop fs -mkdir /ats/done

 

Step 2-2. 실행

# 실행
yarn --daemon (start|stop|status) --config /etc/hadoop/conf resourcemanager

 

Step 2-3. historyserver

mapred --daemon start historyserver

{ 히스토리 실행 서버 }:19888 URI로 접근을 시도하면 됩니다.

 

Step 2-4. timeline server

yarn --daemon start timelineserver

{ 타임라인 실행 서버 }:8188 URI로 접근을 시도하면 됩니다.

 

Step 2-5. 확인

$ jps
690 JournalNode
931 NameNode
4038 JobHistoryServer
4262 Jps
1913 DFSZKFailoverController
3450 ResourceManager
28221 QuorumPeerMain
4175 ApplicationHistoryServer

 

Step 2-6. Port 정보 참조

 

YARN Service Ports - Hortonworks Data Platform

The following table lists the default ports used by the various YARN services. Table 1.18. YARN Service Ports Service Servers Default Ports Used Protocol Description Need End User Access? Configuration Parameters Resource Manager WebUI Master Nodes (Reso

docs.cloudera.com

 

Step 3. Node Manager

Step 3-1. 실행

# 실행
yarn --daemon (start|stop|status) --config /etc/hadoop/conf nodemanager

 

Step 3-2. 확인

$ jps
15030 NodeManager
15128 Jps
14812 DataNode

 

Step 3-3. 테스트

$ yarn jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 50 100
Number of Maps  = 50
Samples per Map = 100
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Wrote input for Map #10
Wrote input for Map #11
Wrote input for Map #12
Wrote input for Map #13
Wrote input for Map #14
Wrote input for Map #15
Wrote input for Map #16
Wrote input for Map #17
Wrote input for Map #18
Wrote input for Map #19
Wrote input for Map #20
Wrote input for Map #21
Wrote input for Map #22
Wrote input for Map #23
Wrote input for Map #24
Wrote input for Map #25
Wrote input for Map #26
Wrote input for Map #27
Wrote input for Map #28
Wrote input for Map #29
Wrote input for Map #30
Wrote input for Map #31
Wrote input for Map #32
Wrote input for Map #33
Wrote input for Map #34
Wrote input for Map #35
Wrote input for Map #36
Wrote input for Map #37
Wrote input for Map #38
Wrote input for Map #39
Wrote input for Map #40
Wrote input for Map #41
Wrote input for Map #42
Wrote input for Map #43
Wrote input for Map #44
Wrote input for Map #45
Wrote input for Map #46
Wrote input for Map #47
Wrote input for Map #48
Wrote input for Map #49
Starting Job
Job Finished in 3.784 seconds
Estimated value of Pi is 3.14160000000000000000
반응형

'Computer & Data > Big Data' 카테고리의 다른 글

Hadoop 시리즈. Hive metastore 3버전 설치  (0) 2022.01.10
Hadoop 시리즈. Spark 구성해보기  (0) 2021.09.26
Hadoop. Namenode heap 메모리에 관하여..  (0) 2021.09.26
HDFS 포트 정리  (0) 2021.09.26
Hadoop 시리즈. Hadoop3 설치하기 - Observer node  (0) 2021.09.26
'Computer & Data/Big Data' Related Articles more