Install and configure Quantum network + Linuxbridge for RHEL6.3 (Folsom release)

This page describes the instructions for manually install and configure the OpenStack Folsom release with quantum network and Linuxbridge on RHEL 6.3 OS. Specifically, the
instructions describe how to install and configure the quantum network component on distributed environment. 

If you have any questions about the documentation or if you find any bugs in the documentation, please send an email to:

Jian Hua Geng <gengjh@cn.ibm.com>

Reference document link:

http://docs.openstack.org/trunk/openstack-network/admin/content/ch_install.html

https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst

http://wiki.openstack.org/ConfigureOpenvswitch

http://docs.openstack.org/trunk/openstack-compute/install/yum/content/

1. Introduction

This instruction only cover the installation and configuration of Quantum network + Linuxbridge to work with Nova compute, assume you have a workable OpenStack environment base on Folsom release already. You can follow this document to
setup your Cloud environment first. We will deploy Network node from OpenStack Folsom release on a single machine running RHEL 6.3, and deploy the quantum-server on your existing Cloud controller node. Then we will run the quantum-linuxbridge-agent on compute
machine.

 

2. Preparing for installation

 

2.1 Setting up the physical network

On the Network Node, we suggest using 3 NICs: eth0 for public network (i.e. Internet), eth1 for VMs traffic (used by OVS), and eth2 for management network For compute nodes, we suggest using 2 NIC : eth2 for management network
and eth1 for VMs traffic. The NIC with on IP address which is used for VMs traffic (eth1) should be connected to the VLAN trunk port on physical switch sine we will use the "vlan" network type to configure quantum-linuxbridge-agent.

 

2.2 Setting up yum repos on OS

Follow this guide to setup your
yum repos for RHEL.

 

3 Install and configure cloud controller and compute node

Since this instruction only focues on the Quantum installation and configuration, so please follow this guide to
install cloud controller and compute node to make your OpenStack environment ready first.

3.1 On Identity node

1. Create Quantum Service user

keystone user-create --name=quantum --pass=$SERVICE_PASSWORD --tenant-id [service tenant id]

keystone user-add --user_id [quantum user id] --role_id [admin role id] --tenant_id [service tenant id]

 

2. Create endpoint and service for Quantum

keystone service-create --name quantum --type network --description 'OpenStack Networking Service'

keystone endpoint-create --region $REGION --service-id $ID --publicurl 'http://$IP:9696/' --adminurl 'http://$IP:9696/' --internalurl
'http://$IP:9696/'

 

3.2 On Network node

1. Run the services on network node: quantum-linuxbridge-agent, quantum-dhcp-agent and quantum-l3-agent

2. Run the following command to make sure the “bridge” kernel module is loaded:

modprobe bridge

lsmod |grep bridge

bridge 79950 0

stp 2173 1 bridge

llc 5642 2 bridge,stp

3. Run the following command to make sure the “8021q” kernel module is loaded:

modprobe 8021q

lsmod |grep 8021

8021q 25058 0

garp 7344 1 8021q

4. Enable the IP forwarding

3.3 On Controller node

1. Need run the service quantum-server

2. Make sure the “bridge” model is loaded

 

3.4 On Compute node

1. Need run the services: openvswitch, quantum-linuxbridge-agent and openstack-nova-compute

2. Make sure the “bridge” kernel module is loaded

3. Make sure the “8021q” kernel module is loaded

4. Enable the IP forwarding

 

3.5 Create ovs quantum database

Following is using the mysql as an example:

create database quantum_linux_bridge;

create user 'quantum'@'localhost' identified by 'quantum';

create user 'quantum'@'%' identified by 'quantum';

grant all on quantum_linux_bridge.* to 'quantum'@'%';

 

3.6 Configure the quantume-server

1. Edit the api paste configuration file /etc/quantum/api-paste.ini with:

auth_host = $KEYSTONE_SERVICE_HOST

auth_port = $KEYSTONE_AUTH_PORT

auth_protocol = http

admin_tenant_name = service

admin_user = quantum

admin_password = passw0rd

 
2. Edit the quantum configuration file /etc/quantum/quantum.conf with:

core_plugin = quantum.plugins.linuxbridge.lb_quantum_plugin.LinuxBridgePluginV2

api_paste_config = /etc/quantum/api-paste.ini

auth_strategy = keystone

rpc_backend=quantum.openstack.common.rpc.impl_qpid

#Qpid broker hostname

qpid_hostname = $QPID_HOST (make sure this qpid IP is set to your controller node)

qpid_port = 5672

 

Note: the auth in /etc/qpidd.conf should be set to ‘no’ if we do not provide the qpid_username and qpid_password in the quantum.conf

3. Create the additional directories which are required at quantum server running:

mkdir -p /var/run/quantum

chown -R quantum:quantum /var/run/quantum

 
4. See the additional step to generate quantum tables in database if there tables can not be created auto after quantum server service start.

3.7 Configure quantum-linuxbridge-agent

1. Edit the Linuxbridge plugin configuration file /etc/quantum/plugins/linuxbridge/linuxbridge_conf.ini with:

sql_connection = mysql://quantum:quantum@$CONTROLLER_NODE/quantum_linux_bridge?charset=utf8

[VLANS]

tenant_network_type=vlan

network_vlan_ranges = physnet1:1000:1100 (Note: Your networking switches must be configured to enable the specific VLAN
tags you specify in your Compute setup)

[LINUX_BRIDGE]

physical_interface_mappings = physnet1:eth0 (all VM communication between the nodes will be done via eth0)

 

Note: Since the network type is vlan, the eth0 should be connected to the trunk point on the physical switch.

2. Edit the Linuxbridge service file /etc/init.d/quantum-linuxbridge-agent:
daemon --user quantum --pidfile $pidfile "$exec --config-file /etc/quantum/quantum.conf --config-file $config --log-file /var/log/quantum/quantum-linuxbridge.log &>/dev/null & echo \$! > $pidfile"

3.8 Configure quantum-l3-agent

1. Edit l3 agent configuration file /etc/quantum/l3_agent.ini with:

 

interface_driver = quantum.agent.linux.interface.BridgeInterfaceDriver

auth_url = http://$KEYSTONE_SERVICE_HOST:$KEYSTONE_AUTH_PORT/v2.0
auth_region = RegionOne

admin_tenant_name = service

admin_user = quantum

admin_password = passw0rd

use_namespaces = False (RHEL6.3 does not support use name space, so we need disable it)

external_network_bridge =

2. Create a shell script to start l3 agent service:

quantum-l3-service

 

3.9 Configure quantum-dhcp-agent

1. Edit dhcp agent configuration file /etc/quantum/dhcp_agent.conf with:

interface_driver = quantum.agent.linux.interface.BridgeInterfaceDriver

use_namespaces = False

2. Create a shell script to start dhcp agent:

quantum-dhcp-agent

3. Create the additional directories which are required at quantum dhcp agent running:

mkdir -p /opt/stack

chown -R quantum:quantum /opt/stack

 

3.10 Configure nova

1. Clean the compute node environment before re-run the nova related service:

iptables -S -v | sed "s/-c [0-9]* [0-9]* g" | grep "nova" | grep "\-A" | sed "s/-A/-D/g" | awk '{print "sudo iptables",$0}' | bash

iptables -S -v -t nat | sed "s/-c [0-9]* [0-9]* g" | grep "nova" | grep "\-A" | sed "s/-A/-D/g" | awk '{print "sudo iptables -t nat",$0}' | bash

iptables -S -v | sed "s/-c [0-9]* [0-9]* g" | grep "nova" | grep "\-N" | sed "s/-N/-X/g" | awk '{print "sudo iptables",$0}' | bash

iptables -S -v -t nat | sed "s/-c [0-9]* [0-9]* g" | grep "nova" | grep "\-N" | sed "s/-N/-X/g" | awk '{print "sudo iptables -t nat",$0}' | bash

killall -9 dnsmasq

rm -rf /var/lib/nova/networks/*

sysctl -w net.ipv4.ip_forward=1

 

2. nova configuration in nova.conf on compute node:

sql_connection = mysql://nova:nova@$CONTROLLER_NODE/nova

compute_driver = libvirt.LibvirtDriver

network_api_class=nova.network.quantumv2.api.API

quantum_admin_username=quanutm

quantum_admin_password=passw0rd quantum_admin_auth_url=http://$KEYSTONE_SERVICE_HOST:$KEYSTONE_AUTH_PORT/v2.0"

quantum_auth_strategy=keystone

quantum_admin_tenant_name=service

quantum_url=http://$QUANTUM_SERVER_HOST:$QUANTUM_SERVER_PORT

libvirt_vif_type=ethernet

libvirt_vif_driver = nova.virt.libvirt.vif.QuantumLinuxBridgeVIFDriver

linuxnet_interface_driver = nova.network.linux_net.QuantumLinuxBridgeInterfaceDriver

 

Note: You need comment out the ‘network_manager’ in the nova.conf file, otherwise you will encounter the error: ImportError: No module named quantum.manager

 

3.11 Other related configuration on compute node

1. modify the cgroup_device_acl field in /etc/libvirt/qemu.conf to include “/dev/net/tun” as shown below and restart libvirt. Otherwise VMs will fail to boot with the message “‘tap’ could not be initialized” in the nova-compute log.

cgroup_device_acl = [

"/dev/null", "/dev/full", "/dev/zero",

"/dev/random", "/dev/urandom",

"/dev/ptmx", "/dev/kvm", "/dev/kqemu",

"/dev/rtc", "/dev/hpet","/dev/net/tun",

]

2. In /etc/libvirt/qemu.conf add/edit the following lines

clear_emulator_capabilities = 0

user = root

group = root

3. disable SELinux

4. make sure the tunctl rpm is installed.

 

3.12 Create network and run VM with a floating ip

quantum net-create net1 shared true

quantum subnet-create tenant_id <tenant id> ip_version 4 gateway 10.0.0.1 <net1 id> 10.0.0.0/24 //This is for internal network

quantum router-create tenant_id <tenant id> router1

quantum router-interface-add <router1 id> <net1's subnet id>

quantum net-create ext_net router:external=True share true

quantum subnet-create ip_version 4 <ext_net id> 172.15.1.0/24 enable_dhcp=False //This is for external network

quantum router-gateway-set <router id> <ext_net id>

 

Set the 'router_id' in the l3 agent configuration file and restart the l3 agent

quantum floatingip-create <ext_net id>

nova boot image $(glance index | grep RHEL6.3 | awk '{print $1}') flavor 1 nic net-id=<net1 id> mytest-3

quantum floatingip-associate $(quantum floatingip-list |grep <Floating ip you want to assign to the mytest-3> | awk '{print $2}') $(quantum port-list |grep <The internal ip address of mytest-3> | awk '{print $2}')

Update the security group to enable ping and ssh on VMs:

nova secgroup-add-rule default tcp 22 22 0.0.0.0/0

nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0

 

3.12 Additional steps when run quantum services:

1. Add following in the /etc/sudoers to ensure quantum user run command without a tty:

quantum ALL=(ALL) ALL

quantum ALL=(ALL) NOPASSWD: ALL

 

2. Make the directory /opt/stack/data writable by the quantum user

3. Fix the no module name "l3_nat_agent" defect https://bugs.launchpad.net/quantum/+bug/1059890

in the file /usr/lib/python2.6/site-packages/quantum-2012.2.1-py2.6.egg-info/entry_points.txt, change the "l3_nat_agent" to "l3_agent"

4. If the tables can not created automatically, you need manually update the params when start quantum-server

such as: python /usr/bin/quantum-server config-file /etc/quantum/plugins/linuxbridge/linuxbridge_conf.ini config-file /etc/quantum/quantum.conf log-file /var/log/quantum/server.log

5. You may get the following error when start l3 agent service, since the ip netns does not supported by RHEL6.3.

Stdout: '/usr/bin/quantum-rootwrap: Incorrect configuration file: ip\n'

Stderr: ''

Traceback (most recent call last):

File "/usr/bin/quantum-l3-agent", line 9, in <module>

load_entry_point('quantum==2012.2.1', 'console_scripts', 'quantum-l3-agent')()

File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 530, in main

mgr = L3NATAgent(conf)

File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 129, in init

self._destroy_all_router_namespaces()

File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 136, in _destroy_all_router_namespaces

for ns in root_ip.get_namespaces(self.conf.root_helper):

File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 124, in get_namespaces

output = cls._execute('', 'netns', ('list',), root_helper=root_helper)

File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 56, in _execute

root_helper=root_helper)

File "/usr/lib/python2.6/site-packages/quantum/agent/linux/utils.py", line 55, in execute

raise RuntimeError(m)

RuntimeError:

Command: ['sudo', 'quantum-rootwrap', 'ip', 'netns', 'list']

Exit code: 97

Stdout: '/usr/bin/quantum-rootwrap: Incorrect configuration file: ip\n'

Stderr: ''

output = cls._execute('', 'netns', ('list',), root_helper=root_helper)

File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 56, in _execute

root_helper=root_helper)

File "/usr/lib/python2.6/site-packages/quantum/agent/linux/utils.py", line 55, in execute

raise RuntimeError(m)

RuntimeError:

Command: ['sudo', 'quantum-rootwrap', 'ip', 'netns', 'list']

Exit code: 97

Stdout: '/usr/bin/quantum-rootwrap: Incorrect configuration file: ip\n'

Stderr: ''

This problem was fixed in https://review.openstack.org/#/c/14079/

6. Update the iptables rule on Quantum network node to make sure it will not block dnsmasq DHCP request

7. Upgrade dnsmasq to the latest version if the vm can not get the IP address, the related question in openstack community: https://answers.launchpad.net/quantum/+question/211021