OpenvSwitch架构

Openvswitch的架构

数据库结构和OVS-VSCTL

# ps aux | grep openvswitch
root 1117 0.0 0.0 21200 1580
? S< Jun09 0:35 ovsdb-server /etc/openvswitch/conf.db
-vconsole:emer -vsyslog:err -vfile:info
--remote=punix:/var/run/openvswitch/db.sock
--private-key=db:Open_vSwitch,SSL,private_key
--certificate=db:Open_vSwitch,SSL,certificate
--bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir
--log-file=/var/log/openvswitch/ovsdb-server.log
--pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
root
1153 0.6 1.1 169508 24016 ? S<Ll Jun09 16:24 ovs-vswitchd
unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err
-vfile:info --mlockall --no-chdir
--log-file=/var/log/openvswitch/ovs-vswitchd.log
--pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor

我们发现有两个进程：

ovsdb-server 维护数据库/etc/openvswitch/conf.db

ovs-vswitchd 核心daemon

两者通过unix domain socket /var/run/openvswitch/db.sock 互相通信

ovs-vsctl就是通过和ovsdb-server通信，来修改数据库。

ovs-vswitchd会和ovsdb-server进行通信，来对虚拟设备做相应的修改。

所以ovs-vsctl里面的命令大多数都是对数据库的操作，因而我们必须好好的了解数据库。

如果我们cat /etc/openvswitch/conf.db，我们会发现它是json格式的。





数据库可以通过ovsdb-client dump将数据库内容打印出来

# ovsdb-client dump
Bridge table
_uuid controller datapath_id
datapath_type external_ids fail_mode flood_vlans flow_tables ipfix
mirrors name netflow other_config
ports
protocols sflow status stp_enable
------------------------------------ ---------- ------------------
------------- ------------ --------- ----------- ----------- -----
------- ------ ------- ------------
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------- ----- ------ ----------
929ab1c2-1146-411d-8557-af7498a26444 [] "0000080027dfbff7"
"" {} [] [] {} []
[] br-ex [] {}
[52c02f9d-db2f-4c50-84ce-7a377530ad3b,
79f74c54-056d-4c60-82d7-46412ceee17e]
[] [] {} false
12ebfe38-6dab-402a-8fb5-aa814a5a3f52 [] "00003afeeb122a40"
"" {} [] [] {} []
[] br-int [] {}
[02049620-d3b4-4ecf-9d9e-f8b40a039f64,
035eed84-ce54-44c4-97d5-92f9cc9d662e,
18eacfea-0982-4c89-b2aa-3e2c7dc2b6d3,
d43963b0-be0b-4265-b0bb-4ebc928d3ee0,
d564225b-60e4-4786-a5fc-6b1c8febd0fb] [] [] {} false
ccfc8a92-2d29-4a55-ae4b-21f59eeeaed7 [] "0000928afccc554a"
"" {} [] [] {} []
[] br-tun [] {}
[6d14a97d-28e8-42f8-b5d6-ec655ccc7b91,
8b0946ae-3177-4d5d-bc8e-6d04536701ad]
[] [] {} false
Controller table
_uuid connection_mode controller_burst_limit controller_rate_limit
enable_async_messages external_ids inactivity_probe is_connected
local_gateway local_ip local_netmask max_backoff other_config role
status target
----- --------------- ---------------------- ---------------------
--------------------- ------------ ---------------- ------------
------------- -------- ------------- ----------- ------------ ----
------ ------
Flow_Sample_Collector_Set table
_uuid bridge external_ids id ipfix
----- ------ ------------ -- -----
Flow_Table table
_uuid flow_limit groups name overflow_policy
----- ---------- ------ ---- ---------------
IPFIX table
_uuid cache_active_timeout cache_max_flows external_ids obs_domain_id obs_point_id sampling targets
----- -------------------- --------------- ------------ ------------- ------------ -------- -------
Interface table
_uuid admin_state bfd bfd_status
cfm_fault cfm_fault_status cfm_health cfm_mpid cfm_remote_mpids
cfm_remote_opstate duplex
external_ids
ifindex ingress_policing_burst ingress_policing_rate lacp_current
link_resets link_speed link_state mac mac_in_use mtu
name ofport ofport_request options other_config
statistics

status
type
------------------------------------ ----------- --- ----------
--------- ---------------- ---------- -------- ----------------
------------------ ------
--------------------------------------------------------------------------------------------------------------------------------------------------------
------- ---------------------- --------------------- ------------
----------- ----------- ---------- --- ------------------- ----
---------------- ------ -------------- ---------------- ------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------
--------
54ab6189-2611-40cc-884f-5ce20913bc32 down {} {}
[] [] [] [] []
[] full
{}
3 0 0 []
0 1000000000 down [] "08:00:27:df:bf:f7" 1500
"eth1" 3 [] {} {}
{collisions=0, rx_bytes=0, rx_crc_err=0, rx_dropped=0, rx_errors=0,
rx_frame_err=0, rx_over_err=0, rx_packets=0, tx_bytes=558, tx_dropped=0,
tx_errors=0, tx_packets=7} {driver_name="e1000",
driver_version="7.3.21-k8-NAPI", firmware_version=""} ""
6da8e9cd-98df-451e-b7d9-fcddefe0325e up {} {}
[] [] [] [] []
[] []
{}
0 0 0 []
0 [] up [] "c2:9e:64:de:bd:db" []
patch-int 1 [] {peer=patch-tun} {}
{collisions=0, rx_bytes=1340, rx_crc_err=0, rx_dropped=0, rx_errors=0,
rx_frame_err=0, rx_over_err=0, rx_packets=18, tx_bytes=90, tx_dropped=0,
tx_errors=0, tx_packets=1}
{}
patch
eda8218d-1d5e-4c52-ad89-24f8d0445ef9 up {} {}
[] [] [] [] []
[] []
{}
0 0 0 []
0 [] up [] "fa:91:44:32:65:f9" []
patch-tun 1 [] {peer=patch-int} {}
{collisions=0, rx_bytes=90, rx_crc_err=0, rx_dropped=0, rx_errors=0,
rx_frame_err=0, rx_over_err=0, rx_packets=1, tx_bytes=1340,
tx_dropped=0, tx_errors=0, tx_packets=18}
{}
patch
ee54427c-133c-4b2a-a641-c2624732eb66 up {} {}
[] [] [] [] []
[] []
{}
5 0 0 []
0 [] up [] "08:00:27:df:bf:f7" 1500
br-ex 65534 [] {} {}
{collisions=0, rx_bytes=648, rx_crc_err=0, rx_dropped=0, rx_errors=0,
rx_frame_err=0, rx_over_err=0, rx_packets=8, tx_bytes=0, tx_dropped=0,
tx_errors=0, tx_packets=0}
{driver_name=openvswitch}
internal
a45575fd-5179-459e-8951-3553c51f4aaa up {} {}
[] [] [] [] []
[] []
{}
6 0 0 []
2 [] up [] "3a:fe:eb:12:2a:40" 1500
br-int 65534 [] {} {}
{collisions=0, rx_bytes=648, rx_crc_err=0, rx_dropped=0, rx_errors=0,
rx_frame_err=0, rx_over_err=0, rx_packets=8, tx_bytes=2078,
tx_dropped=0, tx_errors=0, tx_packets=27}
{driver_name=openvswitch}
internal
801c0dec-44db-4161-b791-08828d542ecf up {} {}
[] [] [] [] []
[] []
{}
8 0 0 []
2 [] up [] "92:8a:fc:cc:55:4a" 1500
br-tun 65534 [] {} {}
{collisions=0, rx_bytes=648, rx_crc_err=0, rx_dropped=0, rx_errors=0,
rx_frame_err=0, rx_over_err=0, rx_packets=8, tx_bytes=0, tx_dropped=0,
tx_errors=0, tx_packets=0}
{driver_name=openvswitch}
internal
e8193562-c3bb-4b48-a9bf-00bba5f2a213 up {} {}
[] [] [] [] []
[] full {attached-mac="fa:16:3e:37:28:e6",
iface-id="7d18228b-476f-4b50-804a-18e60e6b0e6f", iface-status=active,
vm-uuid="413d9fb4-34e5-4032-95ff-dea80f1f4adc"} 17
0 0 [] 0
10000000000 up [] "66:b8:10:c8:dc:df" 1500 "qvo7d18228b-47"
4 [] {} {} {collisions=0,
rx_bytes=530, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0,
rx_over_err=0, rx_packets=7, tx_bytes=0, tx_dropped=0, tx_errors=0,
tx_packets=0} {driver_name=veth, driver_version="1.0",
firmware_version=""} ""
b33cdfe2-dab1-449e-b2be-ce85e2d0ac79 up {} {}
[] [] [] [] []
[] full {attached-mac="fa:16:3e:58:be:c6",
iface-id="eea9263a-5e4b-4e5b-9923-3d59ca752082", iface-status=active,
vm-uuid="98b592b1-7778-46f1-93df-3c5079650b71"} 14
0 0 [] 0
10000000000 up [] "7a:d4:a4:8d:60:b7" 1500 "qvoeea9263a-5e"
3 [] {} {} {collisions=0,
rx_bytes=530, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0,
rx_over_err=0, rx_packets=7, tx_bytes=390, tx_dropped=0, tx_errors=0,
tx_packets=5} {driver_name=veth, driver_version="1.0",
firmware_version=""} ""
c0a5afe4-b7a8-4230-b3bb-af11a33de14e up {} {}
[] [] [] [] []
[] full {attached-mac="fa:16:3e:6f:5e:bf",
iface-id="c64bd111-eff3-4c65-b096-53c3b3188a43", iface-status=active,
vm-uuid="e2009bf8-de2b-4bff-a96a-627a61caf9e7"} 11
0 0 [] 0
10000000000 up [] "e2:d8:61:c4:1d:01" 1500 "qvoc64bd111-ef"
2 [] {} {} {collisions=0,
rx_bytes=280, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0,
rx_over_err=0, rx_packets=4, tx_bytes=250, tx_dropped=0, tx_errors=0,
tx_packets=3} {driver_name=veth, driver_version="1.0",
firmware_version=""} ""
Manager table
_uuid connection_mode external_ids inactivity_probe is_connected max_backoff other_config status target
----- --------------- ------------ ---------------- ------------ ----------- ------------ ------ ------
Mirror table
_uuid external_ids name output_port output_vlan select_all select_dst_port select_src_port select_vlan statistics
----- ------------ ---- ----------- ----------- ---------- --------------- --------------- ----------- ----------
NetFlow table
_uuid active_timeout add_id_to_interface engine_id engine_type external_ids targets
----- -------------- ------------------- --------- ----------- ------------ -------
Open_vSwitch table
_uuid
bridges
cur_cfg db_version external_ids
manager_options next_cfg other_config ovs_version ssl statistics
system_type system_version
------------------------------------
------------------------------------------------------------------------------------------------------------------
------- ---------- --------------------------------------------------
--------------- -------- ------------ ----------- --- ----------
----------- --------------
ab19a0f9-1dc8-44d0-ac00-a9687bb43fdd
[12ebfe38-6dab-402a-8fb5-aa814a5a3f52,
929ab1c2-1146-411d-8557-af7498a26444,
ccfc8a92-2d29-4a55-ae4b-21f59eeeaed7] 273 "7.3.0"
{system-id="6b963f5d-45e7-409f-b5ee-8e30006dcd73"} []
273 {} "2.0.1" [] {} Ubuntu
"14.04-trusty"
Port table
_uuid bond_downdelay bond_fake_iface
bond_mode bond_updelay external_ids fake_bridge
interfaces lacp mac name
other_config qos statistics status tag trunks vlan_mode
------------------------------------ -------------- ---------------
--------- ------------ ------------ -----------
-------------------------------------- ---- --- ----------------
------------ --- ---------- ------ --- ------ ---------
52c02f9d-db2f-4c50-84ce-7a377530ad3b 0 false
[] 0 {} false
[ee54427c-133c-4b2a-a641-c2624732eb66] [] [] br-ex
{} [] {} {} [] [] []
18eacfea-0982-4c89-b2aa-3e2c7dc2b6d3 0 false
[] 0 {} false
[a45575fd-5179-459e-8951-3553c51f4aaa] [] [] br-int
{} [] {} {} [] [] []
8b0946ae-3177-4d5d-bc8e-6d04536701ad 0 false
[] 0 {} false
[801c0dec-44db-4161-b791-08828d542ecf] [] [] br-tun
{} [] {} {} [] [] []
79f74c54-056d-4c60-82d7-46412ceee17e 0 false
[] 0 {} false
[54ab6189-2611-40cc-884f-5ce20913bc32] [] [] "eth1"
{} [] {} {} [] [] []
6d14a97d-28e8-42f8-b5d6-ec655ccc7b91 0 false
[] 0 {} false
[6da8e9cd-98df-451e-b7d9-fcddefe0325e] [] [] patch-int
{} [] {} {} [] [] []
035eed84-ce54-44c4-97d5-92f9cc9d662e 0 false
[] 0 {} false
[eda8218d-1d5e-4c52-ad89-24f8d0445ef9] [] [] patch-tun
{} [] {} {} [] [] []
d564225b-60e4-4786-a5fc-6b1c8febd0fb 0 false
[] 0 {} false
[e8193562-c3bb-4b48-a9bf-00bba5f2a213] [] [] "qvo7d18228b-47"
{} [] {} {} 1 [] []
02049620-d3b4-4ecf-9d9e-f8b40a039f64 0 false
[] 0 {} false
[c0a5afe4-b7a8-4230-b3bb-af11a33de14e] [] [] "qvoc64bd111-ef"
{} [] {} {} 1 [] []
d43963b0-be0b-4265-b0bb-4ebc928d3ee0 0 false
[] 0 {} false
[b33cdfe2-dab1-449e-b2be-ce85e2d0ac79] [] [] "qvoeea9263a-5e"
{} [] {} {} 1 [] []
QoS table
_uuid external_ids other_config queues type
----- ------------ ------------ ------ ----
Queue table
_uuid dscp external_ids other_config
----- ---- ------------ ------------
SSL table
_uuid bootstrap_ca_cert ca_cert certificate external_ids private_key
----- ----------------- ------- ----------- ------------ -----------
sFlow table
_uuid agent external_ids header polling sampling targets
----- ----- ------------ ------ ------- -------- -------
数据库表之间的关系如图所示









其中Open_vSwitch是根，结构如下：





这个表示ovs-vswitchd的配置，包含下面的几方面:

对bridge设备的配置，bridges指向bridge表，我们能看到的openvswitch的主要功能都是在bridge上实现的，在bridge表中详细叙述。

对本身的一些配置;

other_config : stats-update-interval ：将统计信息写入数据库的间隔时间

other_config : flow-restore-wait ：
为hot-upgrade使用的，如果设为true则不处理任何的包。一般使用的过程为，先停掉ovs-vswitchd，然后将这个值设为true，启
动ovs-vswitchd，这个时候不处理任何包，然后使用ovs-ofctl将flow table
restore到一个正确的状态，最后设置这个值为false，开始处理包

other_config : flow-limit ：在flow table中flow entry的数量

other_config : n-handler-threads ：用于处理新flow的线程数

other_config : n-revalidator-threads ：用于验证flow的线程数.

other_config : enable-statistics 是否统计

statistics : cpu 统计cpu数量，线程

statistics : load_average system load

statistics : memory 总RAM，swap

statistics : process_NAME ：with NAME replaced by a process name，统计memory size, cpu time等

statistics : file_systems：mount point, size, used

client request id:
也即cur_cfg和next_cfg，当一个client修改了数据库的之后，增加next_cfg，然后等待openvswitch应用这些修改，当
修改应用完毕，则cur_cfg =
next_cfg。如果我们打开/etc/openvswitch/conf.db文件，我们发现，随着我们队openvswitch的配
置，cur_cfg是不断++的

对SSL的配置：指向SSL表，主要配置private key, certificate(里面是public key)，已经CA的certificate

对ovsdb-server的配置，指向Manager表，ovs-vswitchd作为它的client，配置一下db connection的选项

在SSL表中，包含了经典的SSL connection的所有配置：

openvswitch本身的private key和public key对，其中public
key放在certificate中，并且需要CA使用自己的private
key进行签名，CA来担保这个certificate是合法的，为了验证这个CA签名，当然需要CA的public key，而CA的public
key是放在ca cert里面的，当然也需要被签名，被更高级的CA担保，或者自己担保自己。

bootstrap_ca_cert是一个boolean，如果是true，则每次启动的时候，都会向controller去拿最新的ca cert。





我们如果仔细观察ovsdb-server进程，据发现这些配置被使用了。

ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err
-vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach –monitor

我们还可以配置ovs-vswitchd和ovs-controller之间通过ssl相互通信，使用的是双向SSL互信

配置ovs-vswitchd，使用switch的private key(sc-privkey.pem)和public key(sc-cert.pem)，但是指定的是controller的CA Cert(cacert.pem)

cd /etc/openvswitch
sudo ovs-pki req+sign sc switch
sudo ovs-vsctl set-ssl \
/etc/openvswitch/sc-privkey.pem \
/etc/openvswitch/sc-cert.pem \
/var/lib/openvswitch/pki/controllerca/cacert.pem

配置并启动controller

cd /etc/openvswitch
sudo ovs-pki req+sign ctl controller

sudo ovs-controller -v pssl:6633 \
-p /etc/openvswitch/ctl-privkey.pem \
-c /etc/openvswitch/ctl-cert.pem \
-C /var/lib/openvswitch/pki/switchca/cacert.pem

这里配置的是controller的的private key(ctl-privkey.pem)和public key(ctl-cert.pem)，但是指定的是switch的CA Cert(cacert.pem)

这个SSL机制有关，当建立SSL连接的时候，



当client连接Server的时候，Server会将自己的certificate发给client，将来Client会用Server的 certificate来解密Server用private key加密的信息，然而这个certificate是否合法，需要CA进行验证，由于这个certificate是Server的，当然client要使 用Server的CA cert进行验证。同理Server会向client请求Client的certificate，来解密Client的private key加密的信息，则会用Client的CA cert进行验证。

当建立ssl连接后，我们可以看到下面的样子：

mininet@mininet:~$ sudo ovs-vsctl show
902d6aa3-6a0a-4708-a286-3301c8b36430
Bridge "s1"
Controller "ssl:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "s1"
Interface "s1"
type: internal
Port "s1-eth1"
Interface "s1-eth1"
Port "s1-eth2"
Interface "s1-eth2"
ovs_version: "2.0.1"

连接controller时某一个switch连接到某个controller.

然而连接manager时某一个openvswitch daemon连接到某个manager.

Manager表配置的是ovsdb-server的，他配置了ovsdb-server使用manager_options中的配置来监听端口，等待client来连接。





其中最重要的一项就是target：

ssl:ip[:port]：ovsdb-server会监听在ip的port上，协议为ssl

tcp:ip[:port]：会监听在ip的port上，协议为tcp

pssl:[port][:ip]：会监听在端口port上，协议为ssl

ptcp:[port][:ip]：会监听在端口port上，协议为ssl

可以通过下面的命令进行设置：

ovs-vsctl set-manager…

从架构图中我们看出，ovs-vswitchd是ovsdb-server的client，两者是通过unix domain sock /var/run/openvswitch/db.sock进行通信的。

ovs-vsctl也是ovsdb-server的客户端，默认情况下，ovs-vsctl是运行在ovsdb-server同一台机器上，也是通过/var/run/openvswitch/db.sock进行通信的。

我们看ovs-vsctl有参数--db，默认是unix:file，然而也可以是tcp:ip:port或者ssl:ip:port

这就使得ovs-vsctl在另外一台机器上，也能远程控制ovsdb-server.

我们有两台机器，一台16.158.165.153，我们设置manager

sudo ovs-vsctl set-manager ptcp:8881

$ sudo ovs-vsctl show
ab19a0f9-1dc8-44d0-ac00-a9687bb43fdd

Manager "ptcp:8881"

Bridge br-int

Port br-int

Interface br-int

type: internal

Port patch-tun

Interface patch-tun

type: patch

options: {peer=patch-int}

Port "qvoeea9263a-5e"

tag: 1

Interface "qvoeea9263a-5e"

Port "qvo7d18228b-47"

tag: 1

Interface "qvo7d18228b-47"

Port "qvoc64bd111-ef"

tag: 1

Interface "qvoc64bd111-ef"

Bridge br-ex

Port br-ex

Interface br-ex

type: internal

Port "eth1"

Interface "eth1"

Bridge br-tun

Port patch-int

Interface patch-int

type: patch

options: {peer=patch-tun}

Port br-tun

Interface br-tun

type: internal

ovs_version: "2.0.1"

另一台机器16.158.165.102，我们在上面运行

root@openstackcliu8:~# ovs-vsctl show

21c19aae-c278-4c65-9c6b-9b2c4d67b6dd

ovs_version: "2.0.1"

发现表是空的。

# ovs-vsctl --db=tcp:16.158.165.153:8881 show

ab19a0f9-1dc8-44d0-ac00-a9687bb43fdd

Manager "ptcp:8881"

Bridge br-int

Port br-int

Interface br-int

type: internal

Port patch-tun

Interface patch-tun

type: patch

options: {peer=patch-int}

Port "qvoeea9263a-5e"

tag: 1

Interface "qvoeea9263a-5e"

Port "qvo7d18228b-47"

tag: 1

Interface "qvo7d18228b-47"

Port "qvoc64bd111-ef"

tag: 1

Interface "qvoc64bd111-ef"

Bridge br-ex

Port br-ex

Interface br-ex

type: internal

Port "eth1"

Interface "eth1"

Bridge br-tun

Port patch-int

Interface patch-int

type: patch

options: {peer=patch-tun}

Port br-tun

Interface br-tun

type: internal

ovs_version: "2.0.1"

Open_vSwitch指向Bridge表





Bridge表无疑是最重要的表，所谓的virtual switch，多是用bridge来实现的。

Bridge有下面几项配置：

核心功能配置项

name

ports：指向Port表，

mirrors：指向mirror表

netflow

sflow

ipfix

flood_vlans：是一些vlan id，对于这些vlan id，mac address learning是不做的，每次寻找mac都进行arp

OpenFlow配置项：从架构图中我们可以看出，openvwitch的一个bridge可以通过openflow协议，被一个统一的controller管理的

Controller

flow_tables

fail_mode:

一旦一个bridge连到一个openflow controller，则flow table就由controller统一管理，如果连接断了

secure： 这个bridge会试图一直连接controller，并不自己建立flow table

standalone：一旦bridge三次连不上controller，就自己建立和管理flow table

datapath_id：

Spaning Tree配置

单纯创建一个bridge 很简单

ovs-vsctl add-br helloworld，但是里面的若干配置相对比较复杂。
http://www.cnblogs.com/popsuper1982/p/3800574.html