DockOne Share (100): Build a container infrastructure that supports multiple hatching engines

Container industry from the initial monomer era has entered the layout engine era, has now formed Kubernetes, Mesos, SwarmKit three pillars of the potential. Under all cabling engines, infrastructure resources are primarily containerized hosts, container networks, and container storage. By decoupling and splitting these infrastructure services and reassembling them like LEGO toys, you can Build Kubernetes, Mesos, SwarmKit and other choreography engines.

Infrastructure Services Definition

First, we need to define the next definition of the container infrastructure, the basic resources that each container will need to run, and the various underlying services surrounding the container, which can be considered infrastructure services, such as:

  1. Host container, for the docker daemon to provide operating carrier.
  2. Container network, which contains the basic physical network and we often use the SDN container network.
  3. Container storage, providing additional storage resources for stateful and stateless containers.
  4. Container log, a systematic collection of container logs, and collation analysis.
  5. Container monitoring, collection of container performance data, can provide a reference for the autoscale action.
  6. Service discovery, in the micro-service scenarios, can help service between the flexible access to find the address.
  7. Load balancing, because the micro-service are basically multi-instance scenarios, so the use of load balancing will be very large.
  8. Health check, check whether the status (port) of the micro service is available, and associate the business recovery mechanism.
  9. Metadata services, you can view the overall cluster of host information, micro service information, container information, etc., can be in the micro-service linkage between play a very good role.

Of course, these basic services are not fragmented, there will certainly be a call between each other, in addition to consider the deployment of basic services maintenance, the need for a management mechanism, that is, we need to define a simple set of engine (of course it will not like Kubernetes , Mesos so huge load) to carry out maintenance. Research and development of the open source container management platform Rancher, found that its management philosophy is the case, Rancher has been released to the v1.2 version, so we follow the v1.2 version of the analysis of its infrastructure management mechanism.

In the v1.2 version of the overall structure (as shown below), Cattle engine down into the infrastructure engine, which in the v1.1 era has long been reflected. Cattle is more of a management tool for infrastructure that can be used to deploy other services and layout engines, and of course it can still be used, and friends who are accustomed to stack-service can still use it, and the introduction of Rancher scheduler But also greatly enhanced its scheduling capabilities. Rancher can already be deployed on Docker1.12, before the use of accidentally installed wrong Docker version of the friend this time can be assured.
Infrastructure infrastructure is the basic engine is Cattle, while the general standard of basic services, including healthcheck, ipsec, network-services, scheduler, these are from the original agent-instance services separated, these services can be any combination, you can do Some custom. Healthcheck for the health of the container services, ipsec for the construction of the container network (of course, under certain conditions can also be replaced with vxlan), network-services is mainly to provide rancher-dns and rancher-metadata and CNI plug-in management, scheduler from the original Rancher-server is dedicated to container dispatch.

Infrastructure Services Portfolio Framework

With our previously defined infrastructure services, how to manage it is not a small challenge. Rancher proposed the concept of an environment template, we can create their own template to specify the need to combine the infrastructure services. As shown below:
We can according to their own needs, select the IPsec network or VXLAN network, storage, the default integration of NFS, EBS, EFS support, there are related needs can also enable the corresponding module. This template definition can be found at … lates .

Container network plugin management

In the previous version of Rancher, users often criticized Rancher's network only IPsec, no other choice. The development of the container community is very rapid, a variety of container network plugs surging, want to compete in the rivers and lakes. Rancher v1.2 version of the times with the times, the previous network implementation has been modified to support the CNI standard, in addition to IPsec and the realization of a relatively high voice VXLAN network, while increasing the CNI plug-in management mechanism, so that we can Hacking access to other third party CNI plugins. The following comparison chart can be seen, the original container network card ip implementation has been removed, replaced by the CNI standard to achieve:
The new IPSec network is based on CNI Bridge on the realization of the CNI configuration information can be dynamically incoming way, so that users can do some custom network operation, the whole schematic as follows:
But because IPSec has encryption and decryption performance loss, so Rancher also provides VXLAN implementation, VXLAN implementation is also based on CNI bridge, using rancher rancher-cni-bridge, rancher-cni-ipam, network configuration information to metadata injection. The difference is that rancher-net container internal, rancher-net activation is vxlan driver, it will generate a vtep1042 device, and open udp 4789 port, this device based on udp 4789 vxlan overlay built two ends of the communication, for the machine through the container Eth0 take the bridge communication, for other Host containers, it is forwarded through the routing rules to the vtep1042 device, and then through the overlay to the remote host, from the peer host bridge forwarded to the corresponding container. The whole process as shown:
In addition, the current container CNI network program is actually very much. The CNI plug-in a variety of, Calico, Weave and so on, each plug-in deployment is not the same. The flexibility to access these plugins should also be the point at which the container infrastructure services should meet. Rancher in order to simplify the deployment of network plug-in proposed network-manager, , it can be compatible with the mainstream CNI plug-in, it actually defines a deployment framework, so CNI plug-in in the framework Internal deployment. Network-manager is deployed as a container, because each plug-in initialization may need to expose the port or join some NAT rules, so the network-manager can dynamically set the initialization rules of different plug-ins, it is the practice of metadata as host port and host Nat the regular data source, and then get the data generated after the corresponding Iptables rules are added to the Host. For the real CNI plug-in, you need to deploy the corresponding CNI plug-in executor (calico / weave), / etc / cni directory to deploy the CNI plug-in configuration in the / opt / cni directory in the network-manager container. These two directory mappings The Docker volume rancher-cni-driver, that is, Host / var / lib / docker / volumes / rancher-cni-driver directory.

Storage volume plugin management

In the v1.2 version, the storage volume management changes are relatively large, basically abandoned the previous convoy components, using the new Rancher Storage . This consideration is mainly to be compatible with the community, rancher-storage compatible with the Docker native volume plugin and Kubernetes flexvolume specification, has now achieved support for nfs efs ebs.

Containerized host plugin management

Containers need to run the carrier, the carrier we all know is commonly used Linux OS, Linux can run on bare metal, you can also run on the virtual machine. On the host container, in fact, we will first think of the use of docker-machine, docker-machine in the default integration of a lot of drivers, which we can in the local environment or in the cloud environment to quickly create a containerized host. But in actual use, we usually have two pain points:

  1. Machine driver is very much, how to manage it.
  2. Machine driver parameters are usually a lot, and non-specific staff is difficult to identify, it is best to need a UI, you can make more people easy to use.

Rancher based on docker-machine on the packaging, we can manage a variety of drivers can be added or deleted. At the same time, but also for each driver custom UI. If you do not set the custom UI, will be based on the driver's parameters to generate parameters corresponding to the form.
Like the AWS EC2 driver settings more complex, so the UI has done a lot of optimization, allowing users to fill in the key and secret, step by step to set:
Such as the OpenStack driver, the form entry is automatically generated based on the relevant drive parameters:

Multi-arranging engine support

In the v1.2 version, Rancher continued its own product features, is the support of multi-arranging engine. Rancher still supports Kubernetes, Mesos, Swarm three cabling engine, Kubernetes can support the newer v1.4.6 version, in recent days Kubernetes v1.5 also released, Rancher also quickly compatible support, would like to try the new version of k8s Friends can use Rancher for quick deployment and experience. Since all of the deployment process code is open, users can still customize the deployment version. UI can also do some simple deployment details of the changes, such as Kubernetes version, Cloud Provider settings, Etcd data backup mechanism and so on. As shown below:
Also worth mentioning is that Rancher support the new version of Swarm Mode is Swarmkit engine. The currently supported version is v1.12.3, so when deploying, install the corresponding Docker Engine. Similar to Kubernetes' support, this also supports some parameter settings, such as the number of managers in the Swarm cluster:
The realization of the principle, Rancher to achieve a swarmkit-mon, which built a Docker, and mapped the dock on the docker.sock, so you can control the swarmkit-mon container Docker to create SwarmKit cluster. Swarmkit-mon implementation is relatively simple, mainly including two shell script: responsible for the management of SwarmKit cluster and Rancher linkage, Agent node information needs to read through rancher-metadata, set the Host Label is directly call Rancher API; health. Sh is responsible for monitoring the state of the swarmkit node (by reading Swer.LocalNodeState with docker.sock) and collaborating with giddyup to expose the health check port so that Rancher Cattle's healthcheck can be used to ensure high availability of the swarmkit-mon service Host of the swarmkit-mon problems can be automatically rebuilt recovery. Principle as shown:

Actual – Laptop deployment Rancher v1.2

The actual combat selection in the MBP, the use of Mac Docker Engine RancherOS VM to build. Why choose RancherOS? Mainly to consider its compatibility with Rancher is better, so that in the early encounter less point compatibility issues, quickly enter the product characteristics of the experience stage.
First open the Docker, and configure the registry mirror, configure the restart Docker. Mirror the service can go to the application of a common cloud manufacturers, such as I am here is the use of Ali cloud registry mirror, as shown:
Open terminal, install Rancher Server:

  $ Docker run -d --restart = unless-stopped -p 8080: 8080 rancher / server: stable 

To add the Host node, you need to create the Host through the docker-machine. The specifications used here are 2-core 2G (specific performance can be adjusted according to their own MBP). The script (

  #! / Usr / bin / env bash 
ROS_ISO_URL = 'http: //'
Docker-machine create -d virtualbox \
--virtualbox-boot2docker-url $ ROS_ISO_URL \
- virtualbox-cpu-count $ ROS_CPU_COUNT \
- virtualbox-memory $ ROS_MEMORY \
$ 1
Docker-machine ls

Add a node to execute:

  $ ./ ros-1 

After the completion of the installation, you can enter the virtual machine to set up:

  $ Docker-machine ls 
Ros-1 - virtualbox Running tcp: // 2376 v1.12.3

Into the VM

$ Docker-machine ssh ros-1

RancherOS set registry mirror

$ Sudo ros config set rancher.docker.extra_args \
"['--registry-mirror', 'https: //']"
$ Sudo system-docker restart docker

Since we want to use VirtualBox virtual machine to form a small cluster, it is recommended to Rancher's Host Registration URL is set to , as shown below:
Add Rancher agent also should pay attention to, CATTLE_AGENT_IP parameters to be set to virtual machine network segment IP, as shown below:
So you can basically completely unlock the Rancher v1.2 various functions, and complete demonstration of various features.

    Heads up! This alert needs your attention, but it's not super important.