Deploy YARN in Kubernetes

Original link:

In the course of micro service practice, we often ask ourselves: what kind of application should be made micro service? What do you need to be aware of when applying micro services? This article discusses the issues that need to be done to discuss microarchitecture and what kind of application is micro service-friendly for the problems encountered in deploying YARN clusters in Kuberenetes.

Start the YARN / HDFS cluster

In the YARN default configuration, the cluster node starts with ssh:

  • Configure nodes in the cluster to communicate without a password [1]
  • Modify the slaves file, in which the cluster node's host name or IP is added
  • Start the cluster with

In this startup process, ssh plays a remote execution role; but remote execution, start the service is part of the work done by Kuberenets, so the need for the following transformation of the process:

  • Configuration YARN standalone ssh without password visits; due to start- reasons, even if the machine's YARN service, also need to start through ssh
    > Although the configuration of the local ssh no password visits, but still the following tips:

  The authenticity of host ' (' can not be established. 
ECDSA key fingerprint is SHA256: QQYvT8PFYzVL8oD5fIySTEH5LjPCN8BtayCDLsvp42k.
Are you sure you want to continue connecting (yes / no)?

In the micro-service environment does not want to need to manually participate in the environment, because you need to set the unknown host; modify ~/.ssh/config as follows:

  StrictHostKeyChecking no 
UserKnownHostsFile / dev / null

  • The slaves file is not configured; the nodes in the cluster are started by Kuberentes
  • Start each node with

In the above process, because there is no configuration slaves file, will only start the machine's services, such as ResourceManager or NodeManager. Dockerfile this step to make the following Dockerfile :

The master node's Dockerfile is as follows:

  FROM ubuntu: 14.04 

RUN apt-get update && apt-get-y install openjdk-7-jdk openssh-server

COPY hadoop-2.7.2.tar.gz / opt /

RUN cd / opt / && tar zxvf hadoop-2.7.2.tar.gz && rm hadoop-2.7.2.tar.gz

COPY ./ /opt/
COPY core-site.xml.template / root /
COPY yarn-site.xml.template / root /
COPY hdfs-site.xml.template / root /

COPY .ssh /root/.ssh

RUN chmod + x /opt/

Hdfs ports

EXPOSE 31010 50020 50070 50075 50090 8020 9000

Mapred ports

EXPOSE 19888

Yarn ports

EXPOSE 8030 8031 ​​8032 8033 8040 8042 8088

Other ports

EXPOSE 49707 2122


The contents of the file are as follows:

! / Bin / sh

Export JAVA_HOME = / usr

HADOOP_PREFIX = / opt / hadoop-2.7.2
HADOOP_CONF_DIR = / opt / hadoop-2.7.2 / etc / hadoop

Sed "s / __HOSTNAME __ /" $ (hostname) "/ g" /root/core-site.xml.template> $ HADOOP_CONF_DIR / core-site.xml
Sed "s / __HOSTNAME __ /" $ (hostname) "/ g" /root/yarn-site.xml.template> $ HADOOP_CONF_DIR / yarn-site.xml
Sed "s / __HOSTNAME __ /" $ (hostname) "/ g" /root/hdfs-site.xml.template> $ HADOOP_CONF_DIR / hdfs-site.xml

Mkdir / opt / hadoop272 / dfs / name -p
Mkdir / opt / hadoop272 / dfs / data -p

$ HADOOP_PREFIX / bin / hdfs namenode -format yarn_272_hdfs

$ HADOOP_PREFIX / sbin / --config $ HADOOP_CONF_DIR --script hdfs start namenode
$ HADOOP_PREFIX / sbin / --config $ HADOOP_CONF_DIR start resourcemanager

While true; do sleep 1000; done

The above Dockerfile has several places to be improved:

  1. Dockerfile used multiple times in COPY , which increases the number of mirrored layers
  2. Currently used a pre-generated .ssh file that can be real time by
  3. In the file, HDFS is formatted each time, and if the plug-in is stored, it should be reformatted every time
  4. The ResourceManager/NodeManager is currently started in the same container and will be deployed separately in a different container
  5. The current YARN / HDFS can only run in the background, you need to improve the detection part: when there is no java program running, the container exit

DataNode and NodeManager Dockerfile similar, but in addition to the start of the service is not the same, the YARN / HDFS configuration file changes are different, the details will be described in the following pages; specific reference to Dockefile.agent and

Connect the ResourceManager with the NodeManager

In the above pages, how to start YARN/HDFS cluster, but after the start of each node also need to communicate with each other to work. For the need to modify the corresponding configuration file YARN / HDFS: core-site.xml , yarn-site.xml and hdfs-site.xml .

Corresponding to the ResourceManager / NodeManager communication configuration In the yarn-site.xml and core-site.xml , in order for the ResourceManager and the NodeManager to communicate with each other, the following template is created in the mirror:


<Name> fs.defaultFS </ name>
<Value> hdfs: // __HOSTNAME__: 9000 </ value>
</ Property>


<Name> yarn.resourcemanager.address </ name>
<Value> __HOSTNAME __: 8032 </ value>
</ Property>

<Name> yarn.resourcemanager.scheduler.address </ name>
<Value> __HOSTNAME __: 8030 </ value>
</ Property>
<Name> yarn.resourcemanager.resource-tracker.address </ name>
<Value> __HOSTNAME __: 8031 ​​</ value>
</ Property>
<Name> yarn.resourcemanager.admin.address </ name>
<Value> __HOSTNAME __: 8033 </ value>
</ Property>
<Name> yarn.resourcemanager.webapp.address </ name>
<Value> __HOSTNAME __: 8088 </ value>
</ Property>

In different mirrors, __HOSTNAME__ is replaced with a different value: in , it is replaced by the hostname of the container; in , it is replaced by yarn_master . yarn_master is the ResourceManager in the Kubernetes Service, so after configuring the Kube-DNS, each NodeManager node can find and communicate with the ResourceManager node.

After the above configuration, ResourceManager and NodeManager can communicate normally; but NameNode and DataNode still can not communicate. This is mainly because the NameNode will be connected to the DataNode ip / hostname check, because no DataNode configuration corresponding DNS, NameNode will reject the last of the ip / hostname. HDFS provides the following configuration, you can ignore the ip / hostname check:


<Name> dfs.namenode.datanode.registration.ip-hostname-check </ name>
<Value> false </ value>
</ Property>

But the above configuration does not completely solve the problem: Although NameNode no longer ip / hostname detection, but access to DataNode will get the wrong ip: NameNode use the host ip instead of the container ip and DataNode communication.

Known Hadoop Mirror

Different vendors have provided Hadoop mirrors on the Docker Hub, but are mostly single mirrors and require manual configuration to work properly, such as ssh's passwordless access. There is no community version of the Hadoop image, but has created a HADOOP-13397 to track the requirements.

  • Sequenceiq / hadoop-docker , Hortonwork, single mirror, manual configuration
  • Cloudera / quickstart , Cloudra, single mirror, manual configuration
  • Maprtech / mapr-sandbox-base , MapR, single mirror |

Summary of questions

  • Kubernetes / Marathon and other software for the application has provided a powerful deployment, start and other functions, so when the cluster software to Kubernetes / Marathon migration should note the following:
    1. Avoid using the deployment tools of the cluster itself, for example, ssh in YARN
    2. As far as possible to do the information isolation, for example, to avoid ssh no password visits: no password visits, each container need to know all other containers ssh information
    3. For the Docker to provide front-end script to run, to reduce the secondary development, for example, YARN Dockerfile java detection script
  • In the network, the following aspects can effectively reduce the micro-service process encountered problems:
    1. The initial connection as far as possible for the one-way connection, the subsequent connection to create dynamically; to avoid interdependence, for example, HDFS NameNode and DataNode connection, only yarn_master can establish communication
    2. Try to use IP to access, to avoid the hostname dependencies; not recommended for the establishment of hostname DNS: Each new docker has a new hostname, docker start and stop will increase the burden on DNS

Related discussion

  • Http:// … .html
  • Https:// … / yarn
  • Https://
  • Https:// … ull / 8
  • Http: //mail-archives.apache.or …% 253E

Heads up! This alert needs your attention, but it's not super important.