Configuring the SDN

admin

Overview
Available SDN Providers
Configuring the Pod Network with Ansible
* V! S- l8 J8 f% @/ z' ~5 aConfiguring the Pod Network on Masters
Configuring the Pod Network on Nodes
Migrating Between SDN Plug-ins
External Access to the Cluster Network
Using Flannel
Overview
6 z! }: Z  M; F0 _The OpenShift SDN enables communication between pods across the OpenShift Container Platform cluster, establishing a pod network. Two SDN plug-ins are currently available (ovs-subnet and ovs-multitenant), which provide different methods for configuring the pod network. A third (ovs-networkpolicy) is currently in Tech Preview.
5 P: M! Y( f9 X
Available SDN Providers
The upstream Kubernetes project does not come with a default network solution. Instead, Kubernetes has developed a Container Network Interface (CNI) to allow network providers for integration with their own SDN solutions.
7 ?9 ?( ~6 P. M* v. H+ o
There are several OpenShift SDN plugins available out of the box from Red Hat, as well as third-party plug-ins.

  g% D; F1 C2 C- t# X% `4 SRed Hat has worked with a number of SDN providers to certify their SDN network solution on OpenShift Container Platform via the Kubernetes CNI interface, including a support process for their SDN plug-in through their product’s entitlement process. Should you open a support case with OpenShift, Red Hat can facilitate an exchange process so that both companies are involved in meeting your needs.
; {/ y) s. q6 o9 O' ?8 x
The following SDN solutions are validated and supported on OpenShift Container Platform directly by the 3rd party vendor:

! K( v  q/ ^: `1 wCisco Contiv (™)

Juniper Contrail (™)
* `6 @5 S& I! U5 }7 j, G
Nokia Nuage (™)

Tigera Calico (™)
; {& `0 ?- e% c7 F
Configuring the Pod Network with Ansible
For initial advanced installations, the ovs-subnet plug-in is installed and configured by default, though it can be overridden during installation using the os_sdn_network_plugin_name parameter, which is configurable in the Ansible inventory file.

' z& u- C5 J* N7 a- i5 YExample 1. Example SDN Configuration with Ansible
# Configure the multi-tenant SDN plugin (default is 'redhat/openshift-ovs-subnet')
# _$ p2 {$ t4 M4 {# os_sdn_network_plugin_name='redhat/openshift-ovs-multitenant'

# Disable the OpenShift SDN plugin
# openshift_use_openshift_sdn=False
7 m- V/ b" L7 X/ g$ O
# Configure SDN cluster network CIDR block. This network block should
- v! M  Z; a& R( V; L. I$ N# be a private block and should not conflict with existing network
# blocks in your infrastructure that pods may require access to.
# Can not be changed after deployment.
* d3 f- w1 X0 X/ D( t4 l#osm_cluster_network_cidr=10.1.0.0/16

# default subdomain to use for exposed routes
#openshift_master_default_subdomain=apps.test.example.com
/ o, H1 c! a& b' o  {% r  G1 _
7 i5 T4 r4 D. Q0 ~7 l# Configure SDN cluster network and kubernetes service CIDR blocks. These
6 r$ ~0 v( N; `4 ^# network blocks should be private and should not conflict with network blocks
# in your infrastructure that pods may require access to. Can not be changed
# after deployment.
3 n8 [4 }2 o# y" G#osm_cluster_network_cidr=10.1.0.0/16
4 V2 [0 ]2 f, f  F  G' N# u#openshift_portal_net=172.30.0.0/16
3 D7 S8 T( m1 t" ]9 H
# Configure number of bits to allocate to each host’s subnet e.g. 8
# would mean a /24 network on the host.
1 U  N8 I9 s9 N* T+ }#osm_host_subnet_length=8

# This variable specifies the service proxy implementation to use:
# either iptables for the pure-iptables version (the default),
, R! I) L$ Q0 t+ d# or userspace for the userspace proxy.
#openshift_node_proxy_mode=iptables
For initial quick installations, the ovs-subnet plug-in is installed and configured by default as well, and can be reconfigured post-installation using the networkConfig stanza of the master-config.yaml file.
# p- u3 _) q3 z7 c( e, e
Configuring the Pod Network on Masters
Cluster administrators can control pod network settings on masters by modifying parameters in the networkConfig section of the master configuration file (located at /etc/origin/master/master-config.yaml by default):
8 Z) @3 i# k/ [  Q# T
networkConfig:
  clusterNetworkCIDR: 10.128.0.0/14
  hostSubnetLength: 9
  networkPluginName: "redhat/openshift-ovs-subnet"
6 [* B; g, j9 z$ M) b  serviceNetworkCIDR: 172.30.0.0/16
3 O7 e4 m, I0 Q" R: \Cluster network for node IP allocation
% n; p6 a! Z( R( A! C$ sNumber of bits for pod IP allocation within a node
Set to redhat/openshift-ovs-subnet for the ovs-subnet plug-in or redhat/openshift-ovs-multitenant for the ovs-multitenant plug-in
: |! ]8 S: M4 @Service IP allocation for the cluster
The serviceNetworkCIDR and hostSubnetLength values cannot be changed after the cluster is first created, and clusterNetworkCIDR can only be changed to be a larger network that still contains the original network. For example, given the default value of 10.128.0.0/14, you could change clusterNetworkCIDR to 10.128.0.0/9 (i.e., the entire upper half of net 10) but not to 10.64.0.0/16, because that does not overlap the original value.
( p1 F8 z0 }; FConfiguring the Pod Network on Nodes
Cluster administrators can control pod network settings on nodes by modifying parameters in the networkConfig section of the node configuration file (located at /etc/origin/node/node-config.yaml by default):
  A# j+ g$ M6 r, d/ K  F' d
. x4 ?* }, F$ Z. ~, enetworkConfig:
0 z% Y( N. f! N  mtu: 1450
  networkPluginName: "redhat/openshift-ovs-subnet"
Maximum transmission unit (MTU) for the pod overlay network
Set to redhat/openshift-ovs-subnet for the ovs-subnet plug-in or redhat/openshift-ovs-multitenant for the ovs-multitenant plug-in
Migrating Between SDN Plug-ins
If you are already using one SDN plug-in and want to switch to another:
* P) P: m4 x" @
Change the networkPluginName parameter on all masters and nodes in their configuration files.
! n6 t* h  D% G: O2 q6 X4 M
0 n6 O  H7 K9 L3 h. t9 Z- J7 w& MRestart the atomic-openshift-master service on masters and the atomic-openshift-node service on nodes.

If you are switching from an OpenShift SDN plug-in to a third-party plug-in, then clean up OpenShift SDN-specific artifacts:
7 S) [) E* @, p$ j& W
$ oc delete clusternetwork --all
, T! @1 \, L/ u. `$ b$ oc delete hostsubnets --all
$ oc delete netnamespaces --all
When switching from the ovs-subnet to the ovs-multitenant OpenShift SDN plug-in, all the existing projects in the cluster will be fully isolated (assigned unique VNIDs). Cluster administrators can choose to modify the project networks using the administrator CLI.
# K) A7 h7 E; D( J
5 R/ D* ^/ b+ X9 ?5 D6 RCheck VNIDs by running:
0 B. W5 y+ c5 S- _3 z7 _3 I
) n$ |" Q  Y5 v9 C* i3 a$ oc get netnamespace
  T! C5 K  J/ VExternal Access to the Cluster Network
If a host that is external to OpenShift Container Platform requires access to the cluster network, you have two options:

Configure the host as an OpenShift Container Platform node but mark it unschedulable so that the master does not schedule containers on it.

5 w: N9 X6 q8 U. w7 ^/ |Create a tunnel between your host and a host that is on the cluster network.
9 p- a' u! w* W, Q3 }& w- {
7 v  O& F1 O2 w8 JBoth options are presented as part of a practical use-case in the documentation for configuring routing from an edge load-balancer to containers within OpenShift SDN.

1 m5 z1 ~# `, b1 wUsing Flannel
; U8 I4 ^. k! y5 q2 J5 U: x- r0 HAs an alternate to the default SDN, OpenShift Container Platform also provides Ansible playbooks for installing flannel-based networking. This is useful if running OpenShift Container Platform within a cloud provider platform that also relies on SDN, such as Red Hat openstack Platform, and you want to avoid encapsulating packets twice through both platforms.
9 X- @6 C# M4 p% u% H% g
Flannel uses a single IP network space for all of the containers allocating a contiguous subset of the space to each instance. Consequently, nothing prevents a container from attempting to contact any IP address in the same network space. This hinders multi-tenancy because the network cannot be used to isolate containers in one application from another.

Depending on whether you prefer mutli-tenancy isolation or performance, you should determine the appropriate choice when deciding between OpenShift SDN (multi-tenancy) and flannel (performance) for internal networks.

Flannel is only supported for OpenShift Container Platform on Red Hat OpenStack Platform.
The current version of Neutron enforces port security on ports by default. This prevents the port from sending or receiving packets with a MAC address different from that on the port itself. Flannel creates virtual MACs and IP addresses and must send and receive packets on the port, so port security must be disabled on the ports that carry flannel traffic.
  I" K1 b9 M" B! l  \To enable flannel within your OpenShift Container Platform cluster:
0 C8 K! D2 n% Q2 t/ S5 h/ P& `
Neutron port security controls must be configured to be compatible with Flannel. The default configuration of Red Hat OpenStack Platform disables user control of port_security. Configure Neutron to allow users to control the port_security setting on individual ports.
2 x0 `; A6 G+ l, S- k6 C
On the Neutron servers, add the following to the /etc/neutron/plugins/ml2/ml2_conf.ini file:

1 `/ G' v9 q" W" f$ t7 ]* P[ml2]
...
$ I5 O. j! N4 F/ Textension_drivers = port_security
$ U' Y1 |5 g2 i5 d1 SThen, restart the Neutron services:
  F) L) }8 r9 Q/ T! k, K  B; _
service neutron-dhcp-agent restart
/ C, R" U. X# d8 s2 Vservice neutron-ovs-cleanup restart
service neutron-metadata-agentrestart
service neutron-l3-agent restart
5 ^5 Q# z; F: K) M% oservice neutron-plugin-openvswitch-agent restart
+ k, [9 X+ w. M% a, K: g6 Lservice neutron-vpn-agent restart
service neutron-server  restart
When creating the OpenShift Container Platform instances on Red Hat OpenStack Platform, disable both port security and security groups in the ports where the container network flannel interface will be:

( y- \7 u! a4 j7 y0 zneutron port-update $port --no-security-groups --port-security-enabled=False
Flannel gather information from etcd to configure and assign the subnets in the nodes. Therefore, the security group attached to the etcd hosts should allow access from nodes to port 2379/tcp, and nodes security group should allow egress communication to that port on the etcd hosts.
% z" Q& v' v0 T9 Q% E9 |  qSet the following variables in your Ansible inventory file before running the installation:
- V1 g% M4 I. k+ m7 z% X# K! g
( z6 o5 N! d% ~! b( o, ]' Dopenshift_use_openshift_sdn=false
" U2 b9 S- M6 f: R$ p5 H/ ~3 F! b/ lopenshift_use_flannel=true
/ [/ v6 l: W; s! o  ~- Bflannel_interface=eth0
Set openshift_use_openshift_sdn to false to disable the default SDN.
Set openshift_use_flannel to true to enable flannel in place.
7 }+ G- `, E  K3 v) [1 MOptionally, you can specify the interface to use for inter-host communication using the flannel_interface variable. Without this variable, the OpenShift Container Platform installation uses the default interface.

- B3 F# |% B# |: I( XCustom networking CIDR for pods and services using flannel will be supported in a future release. BZ#1473858
6 E$ K3 ]) B# h$ _' BAfter the OpenShift Container Platform installation, add a set of iptables rules on every OpenShift Container Platform node:

iptables -A DOCKER -p all -j ACCEPT
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
; p; l8 d2 E, y9 S, @- VTo persist those changes in the /etc/sysconfig/iptables use the following command on every node:
3 A- C2 d. R: G! [# c
8 G: I% x% O! \. i  }cp /etc/sysconfig/iptables{,.orig}
sh -c "tac /etc/sysconfig/iptables.orig | sed -e '0,/:DOCKER -/ s/:DOCKER -/:DOCKER ACCEPT/' | awk '"\!"p && /POSTROUTING/{print \"-A POSTROUTING -o eth1 -j MASQUERADE\"; p=1} 1' | tac > /etc/sysconfig/iptables"
The iptables-save command saves all the current in memory iptables rules. However, because Docker, Kubernetes and OpenShift Container Platform create a high number of iptables rules (services, etc.) not designed to be persisted, saving these rules can become problematic.
To isolate container traffic from the rest of the OpenShift Container Platform traffic, Red Hat recommends creating an isolated tenant network and attaching all the nodes to it. If you are using a different network interface (eth1), remember to configure the interface to start at boot time through the /etc/sysconfig/network-scripts/ifcfg-eth1 file:

DEVICE=eth1
TYPE=Ethernet
+ F/ z9 j4 u* }% V) r& uBOOTPROTO=dhcp
ONBOOT=yes
1 S& {( k0 G1 P0 dDEFTROUTE=no
. }- s6 O! j( NPEERDNS=no