Hadoop 시리즈. Spark 구성해보기

Hadoop 시리즈. Yarn 구성해보기

지난번에 설치하였던 Yarn에 이어서 하둡 클러스터에 스파크를 구성하는 포스팅을 다루어보겠습니다.

 

 

Step 1. Spark Install

Downloads | Apache Spark

스파크 바이너리 파일은 위 공식 페이지에서 받을 수 있습니다.

 

Step 1-1. 바이너리 설치

wget https://downloads.apache.org/spark/spark-3.0.2/spark-3.0.2-bin-hadoop3.2.tgz

 

Step 1-2. 압축해제

sudo tar -zxvf spark-3.0.2-bin-hadoop3.2.tgz -C /opt/
sudo ln -s /opt/spark-3.0.2-bin-hadoop3.2/ /opt/spark

 

Step 1-3. 경로 구성

sudo mkdir -p /var/log/spark
sudo chown -R deploy:deploy /var/log/spark

sudo mkdir -p /etc/spark
sudo ln -s /opt/spark/conf /etc/spark/conf

 

Step 1-4. config

cp /etc/spark/conf/spark-env.sh.template /etc/spark/conf/spark-env.sh

==>
export YARN_CONF_DIR=/etc/hadoop/conf

 

Step 1-5. bashrc

echo 'export PATH=\\$PATH:/opt/spark/bin' >> ~/.bashrc
source ~/.bashrc

 

Step 2. 확인

Step 2-1. 설치 확인

$ spark-shell
21/09/26 15:12:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://story-hadoop-master01:4040
Spark context available as 'sc' (master = local[*], app id = local-1632636774538).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.2
      /_/

Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_292)
Type in expressions to have them evaluated.
Type :help for more information.

scala>

 

Step 2-2. Submit test

spark-submit \
--master yarn \
--deploy-mode cluster \
--driver-memory 1g \
--executor-memory 1g \
--num-executors 2 \
--total-executor-cores 1 \
--class com.examples.WordCount \
/opt/spark/examples/jars/spark-examples_2.12-3.0.2.jar