Bridged networking for libvirt with NetworkManager: 2014 / Fedora 21
2016-03-16 13:06
507 查看
https://www.happyassassin.net/2014/07/23/bridged-networking-for-libvirt-with-networkmanager-2014-fedora-21/
I’m not quite sure how, but I got sucked into spending the whole of today poking at various aspects of handling bridged networking with NetworkManager.
One of the most common uses of bridged networking is for virtualization: you set up a bridge for the host’s connection to your router and configure virtual machines to use that bridge, which allows them to connect to the router just as if they were real
physical machines that were plugged into it, they’ll grab their configuration from your router’s DHCP server, and virtual and metal systems can all talk to each other.
This is rather more convenient than libvirt’s default setup where the VM host more or less acts like a NAT router for all the virtual machines running on it. This works out of the box, but has limitations. The VMs can connect out to the internet and to other
systems on the same network as their host, but those systems and systems outside the local network can’t connect in to the guests without some messy
manual intervention. It’s sort of the same situation you have with relation to the public internet when you’re sitting behind your NAT router – you have to fiddle with stuff in the router settings in order to allow outside systems to connect in to servers
running on your machine.
Since more or less time immemorial, one of the first things you see in
any
set
of
instructions you happen to find for configuring bridging for libvirt is “disable NetworkManager, because it doesn’t work with bridges”.
Every few months I ask the NM devs what the status of this is, and get a sort of handwavy reply, and move on to something else.
But no more! Today I decided to actually poke about at it and see how it works.
Executive summary: yes, you actually can configure bridging for libvirt purposes using NetworkManager, and have it work properly. It’s not even that difficult. But there are some really evil gotchas.
Turn off or delete your existing wired network connection. In GNOME, run the Control Center ‘Network’ panel and slide the slider to Off. Edit its properties You can do it with nmcli or ifdown or KDE or whatever as well. If you only turn it off, best to
also set it not to start at boot: in GNOME, open its properties (with the weird cog icon), click
Identity, and uncheck Connect automatically.
Create a new bridge connection profile. In GNOME ‘Network’, click the +, then click Bridge. With
Create…
If you use DHCP, leave everything in the Editing Bridge connection X window you see at default, except click
Add next to the empty pane labelled “Bridged connections:”, leave the dropdown box at
Ethernet, and click Create…. If you need to customize your configuration at all – for static IP addressing, or whatever – do it here, in the
bridge’s properties, in the IPv4 Settings and IPv6 Settings tabs, before you click
Add. If you forget to do it now, don’t worry, you can always come back and edit the bridge’s properties later.
In the Editing bridgeX slave Y window you see, select your ethernet adapter in the drop-down labelled
“Device MAC address:”. Go to the General tab and check Automatically connect to this network when it is available.
Click Save… (in Editing bridgeX slave Y)
Click Save… (in Editing Bridge connection X)
Open a terminal, and run as root:
Run as root
a bug. At this point, your network should come back up again, maybe after a 30 second or so delay. If you look in
Configure your virtual machines to use the bridge as their Network source (in virt-manager, it’s one of the properties of the VM’s NIC, in the VM details page)
Profit!
If everything works, and it still works after a reboot, you might want to delete the original profile for your ethernet interface, to stop it confusing things.
During Fedora 21 development, two extra steps were needed between 9 and 10:
Run as root
Create a file
notorious, bug, which network.service has a workaround for, but NetworkManager does not.
but I have just tested that these are no longer needed with Fedora 21 Final, as a fix for that bug was included.
solve). But if you really want to try it, here’s what I recommend:
Run virt-manager
Right click on your host, and click Details
Go to the Network Interfaces tab
Click + to add an interface
Leave the type at Bridge and click Forward
Set the Start mode: to onboot
Do NOT check Activate now: ! Don’t do it!
Check the tick mark for your network interface in the “Choose interface(s) to bridge:” pane
Change IP settings: to manually configured, and set the appropriate configuration for your network (DON’T try copying the settings from the existing wired connection, that seems to be really broken)
Click Finish
Do steps 9, 10 and 11 from the NetworkManager instructions above
Now you can try running
Do steps 11, 12 and 13 from the NetworkManager instructions.
Don’t try actually activating the bridge from virt-manager. It uses
manually to actually interact with the connections (and see the bugs linked later).
traffic is routed wrongly – the connection observes the network traffic flow for a while to see what it should do.
This is part of a protocol called
STP which is apparently meant for complex enterprise-y networks with multiple bridges between network segments. It’s probably safe to simply turn it off. To configure this, edit the settings for the Bridge connection from the network configuration tool,
and on the Bridge tab, uncheck Enable STP (Spanning Tree Protocol).
Alternatively, you can reduce the delay to the minimum. To configure this, edit the settings for the Bridge connection from the network configuration tool, and on the
Bridge tab, set Forward delay and Hello time to 2. (Don’t try and set them to 0, or you may run into
a bug).
for a simple bridge in
9 through 11 from the NetworkManager instructions to make traffic flow correctly from the VMs, though.
A simple config with one bridge to one ethernet adapter basically consists of these files in
ifcfg-br1
ifcfg-em1 (or whatever your adapter is called, though the name of the file doesn’t actually matter any more)
All the ‘normal’ config options go in the bridge connection’s config, note (most of the settings aren’t strictly necessary, those are just typical ones from a Fedora install on my system; the UUID= line obviously will be some UUID or other that NM generated
for the connection, not xxx). The
The interface’s config just identifies the interface, says to start it on boot (assuming that’s what you want), and says it’s a bridge slave interface.
To NetworkManager these are two connections, and you want both of them to be active for the bridge to be running. If you set them both to ONBOOT=yes then the whole thing should just come up at boot time, or you can use
confuse things. If you have issues look in
the bridge working. At least set it ONBOOT=no.
If you create the setup using NetworkManager as described above, the bridge connection’s name will likely be “Bridge connection 1”, and the slave interface connection’s name will likely be “bridge0 slave 1”, or similar. In
As mentioned above, when everything’s working, the bridge connection / interface should have the IP address, and ‘brctl show’ should list the bridge with the slave interface in the column on the right. And
and your VMs should get IPs in the normal range for your router, and you should be able to connect between VMs and ‘regular’ systems.
Network control center panel does not show non-active bridge slave profiles (consequently, cannot activate bridges properly)
virt-manager errors in ‘ifup br1’ when creating and activating a bridged connection on NetworkManager system
Cannot bring up a bridge via ifup without causing an error (‘waiting for slaves before proceeding’)
Set up bridged connection, active slave connection, activate bridge -> active profile for interface switches from slave connection to ‘Wired connection’ profile
Fails copying a simple interface configuration to a bridge
I’m not quite sure how, but I got sucked into spending the whole of today poking at various aspects of handling bridged networking with NetworkManager.
One of the most common uses of bridged networking is for virtualization: you set up a bridge for the host’s connection to your router and configure virtual machines to use that bridge, which allows them to connect to the router just as if they were real
physical machines that were plugged into it, they’ll grab their configuration from your router’s DHCP server, and virtual and metal systems can all talk to each other.
This is rather more convenient than libvirt’s default setup where the VM host more or less acts like a NAT router for all the virtual machines running on it. This works out of the box, but has limitations. The VMs can connect out to the internet and to other
systems on the same network as their host, but those systems and systems outside the local network can’t connect in to the guests without some messy
manual intervention. It’s sort of the same situation you have with relation to the public internet when you’re sitting behind your NAT router – you have to fiddle with stuff in the router settings in order to allow outside systems to connect in to servers
running on your machine.
Since more or less time immemorial, one of the first things you see in
any
set
of
instructions you happen to find for configuring bridging for libvirt is “disable NetworkManager, because it doesn’t work with bridges”.
Every few months I ask the NM devs what the status of this is, and get a sort of handwavy reply, and move on to something else.
But no more! Today I decided to actually poke about at it and see how it works.
Executive summary: yes, you actually can configure bridging for libvirt purposes using NetworkManager, and have it work properly. It’s not even that difficult. But there are some really evil gotchas.
Setting up a bridge with NetworkManager
If you have a clean Fedora 20+ (I think – I tested with 21 and 22, but from reports I’ve read I think this all applies to F20 as well) system, and you just want to make it work, here is what you should do.Turn off or delete your existing wired network connection. In GNOME, run the Control Center ‘Network’ panel and slide the slider to Off. Edit its properties You can do it with nmcli or ifdown or KDE or whatever as well. If you only turn it off, best to
also set it not to start at boot: in GNOME, open its properties (with the weird cog icon), click
Identity, and uncheck Connect automatically.
Create a new bridge connection profile. In GNOME ‘Network’, click the +, then click Bridge. With
nm-connection-editor, click Add, set the dropdown to Bridge, and click
Create…
If you use DHCP, leave everything in the Editing Bridge connection X window you see at default, except click
Add next to the empty pane labelled “Bridged connections:”, leave the dropdown box at
Ethernet, and click Create…. If you need to customize your configuration at all – for static IP addressing, or whatever – do it here, in the
bridge’s properties, in the IPv4 Settings and IPv6 Settings tabs, before you click
Add. If you forget to do it now, don’t worry, you can always come back and edit the bridge’s properties later.
In the Editing bridgeX slave Y window you see, select your ethernet adapter in the drop-down labelled
“Device MAC address:”. Go to the General tab and check Automatically connect to this network when it is available.
Click Save… (in Editing bridgeX slave Y)
Click Save… (in Editing Bridge connection X)
Open a terminal, and run as root:
nmcli con show. You should see a connection whose name matches the ‘bridgeX slave Y’ profile you just created. Copy the UUID of that connection.
Run as root
nmcli con up (UUID), using the UUID you just copied. If you look at GNOME’s “Network” applet again, you’ll see a new connection suddenly appeared under “Wired”. The fact that it didn’t show up before and we had to use
nmclito turn it on is
a bug. At this point, your network should come back up again, maybe after a 30 second or so delay. If you look in
ifconfigor
ip addryou’ll see the IP address is tied to the bridge interface. If you look at
brctl showyou should see the bridge you created, with your network interface listed in the right-hand column.
Configure your virtual machines to use the bridge as their Network source (in virt-manager, it’s one of the properties of the VM’s NIC, in the VM details page)
Profit!
If everything works, and it still works after a reboot, you might want to delete the original profile for your ethernet interface, to stop it confusing things.
During Fedora 21 development, two extra steps were needed between 9 and 10:
Run as root
sysctl -p /usr/lib/sysctl.d/00-system.conf
Create a file
/etc/udev/rules.d/99-bridge.rulescontaining just this line:
ACTION=="add", SUBSYSTEM=="module", KERNEL=="bridge", RUN+="/usr/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/bridge". What we just did in these last two steps is deal with another, rather
notorious, bug, which network.service has a workaround for, but NetworkManager does not.
but I have just tested that these are no longer needed with Fedora 21 Final, as a fix for that bug was included.
Setting up a bridge with virt-manager
You may be able to successfully create a bridged network using virt-manager, with some care and a following wind, but I don’t recommend it, as there are some bugs that could leave you in a slightly messy state (though probably nothing a reboot wouldn’tsolve). But if you really want to try it, here’s what I recommend:
Run virt-manager
Right click on your host, and click Details
Go to the Network Interfaces tab
Click + to add an interface
Leave the type at Bridge and click Forward
Set the Start mode: to onboot
Do NOT check Activate now: ! Don’t do it!
Check the tick mark for your network interface in the “Choose interface(s) to bridge:” pane
Change IP settings: to manually configured, and set the appropriate configuration for your network (DON’T try copying the settings from the existing wired connection, that seems to be really broken)
Click Finish
Do steps 9, 10 and 11 from the NetworkManager instructions above
Now you can try running
nmcli con showas root and bringing up the bridge and slave profiles with
nmcli con up (UUID)for each, or you could try rebooting. Again, the network should come back up for the host when you get both the bridge and slave profiles active.
Do steps 11, 12 and 13 from the NetworkManager instructions.
Don’t try actually activating the bridge from virt-manager. It uses
ifupcommands, and I ran into various bugs with that (listed later in the post). If you’re going to use virt-manager, just use it to create the configs, but use nmcli or ifup
manually to actually interact with the connections (and see the bugs linked later).
Reduce the startup delay
By default the bridge will probably take 30 seconds to become active, each time it comes up (actually each time a slave connection comes up, it delays for 30 seconds). This is apparently intended for complex networks where ‘routing loops’ are possible iftraffic is routed wrongly – the connection observes the network traffic flow for a while to see what it should do.
This is part of a protocol called
STP which is apparently meant for complex enterprise-y networks with multiple bridges between network segments. It’s probably safe to simply turn it off. To configure this, edit the settings for the Bridge connection from the network configuration tool,
and on the Bridge tab, uncheck Enable STP (Spanning Tree Protocol).
Alternatively, you can reduce the delay to the minimum. To configure this, edit the settings for the Bridge connection from the network configuration tool, and on the
Bridge tab, set Forward delay and Hello time to 2. (Don’t try and set them to 0, or you may run into
a bug).
Background and details
So what’s actually going on here? Well, NetworkManager’s way of handling bridges is actually not very different from the way the old network service handled them. In fact, at the config file level it’s identical. If you already have correct configurationfor a simple bridge in
/etc/sysconfig/network-scriptsyou should be able to drop any NM_CONTROLLED=no lines, stop network.service, start NetworkManager.service, and find that NM brings up your bridge successfully. You’ll need to do steps
9 through 11 from the NetworkManager instructions to make traffic flow correctly from the VMs, though.
A simple config with one bridge to one ethernet adapter basically consists of these files in
/etc/sysconfig/network-scripts(on RH-ish distros):
ifcfg-br1
DEVICE=br1 ONBOOT=yes TYPE=Bridge BOOTPROTO=dhcp IPV6INIT=yes IPV6_AUTOCONF=no DHCPV6=no STP=yes DELAY=2 DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no NAME="Bridge br1" UUID=xxx BRIDGING_OPTS=priority=32768 PEERDNS=yes PEERROUTES=yes
ifcfg-em1 (or whatever your adapter is called, though the name of the file doesn’t actually matter any more)
DEVICE=em1 HWADDR=f4:6d:04:9a:1d:45 ONBOOT=yes BRIDGE=br1
All the ‘normal’ config options go in the bridge connection’s config, note (most of the settings aren’t strictly necessary, those are just typical ones from a Fedora install on my system; the UUID= line obviously will be some UUID or other that NM generated
for the connection, not xxx). The
STPand
DELAYoptions control the bits discussed under “Reduce the startup delay” above.
The interface’s config just identifies the interface, says to start it on boot (assuming that’s what you want), and says it’s a bridge slave interface.
To NetworkManager these are two connections, and you want both of them to be active for the bridge to be running. If you set them both to ONBOOT=yes then the whole thing should just come up at boot time, or you can use
nmcli con upto bring them up. Remember that NetworkManager can cope with the concept of there being multiple connections for a single device: you may well still have your original connection for the ethernet interface knocking around, and it may
confuse things. If you have issues look in
nmcli con showand see if you have more than one connection that’s for the ethernet interface, and if the wrong one is active. You probably should just get rid of the non-bridge connection once you have
the bridge working. At least set it ONBOOT=no.
If you create the setup using NetworkManager as described above, the bridge connection’s name will likely be “Bridge connection 1”, and the slave interface connection’s name will likely be “bridge0 slave 1”, or similar. In
/etc/sysconfig/network-scriptsthey’ll be named
ifcfg-Bridge_connection_1and
ifcfg-bridge0_slave_1. If you use virt-manager, it’ll use old-style interface names, and will actually overwrite the existing connection for your physical device (so at least you won’t have two knocking around and confusing things).
As mentioned above, when everything’s working, the bridge connection / interface should have the IP address, and ‘brctl show’ should list the bridge with the slave interface in the column on the right. And
sysctl -a | grep bridgeshould show:
net.bridge.bridge-nf-call-arptables = 0 net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0
and your VMs should get IPs in the normal range for your router, and you should be able to connect between VMs and ‘regular’ systems.
Bugs I found along the way
sysctl.conf / sysctl.conf.d settings not read when modules are loadedNetwork control center panel does not show non-active bridge slave profiles (consequently, cannot activate bridges properly)
virt-manager errors in ‘ifup br1’ when creating and activating a bridged connection on NetworkManager system
Cannot bring up a bridge via ifup without causing an error (‘waiting for slaves before proceeding’)
Set up bridged connection, active slave connection, activate bridge -> active profile for interface switches from slave connection to ‘Wired connection’ profile
Fails copying a simple interface configuration to a bridge
相关文章推荐
- Html5_canvas_01
- css切背景图片(background-position)
- gulp之文件合并以及整合html中的script和link
- Using HTML5 audio and video
- 使用HTML5 Web存储的localStorage和sessionStorage方式
- TouchPoint.js – 可视化展示 HTML 原型点击效果
- Java自定义缓冲区MyBufferedReader
- 百度前端技术学院 任务八:响应式网格(栅格化)布局
- CSS定位中的必须深究的常用技法
- jsp有哪些内置对象?作用分别是什么? 分别有什么方法?
- JavaScript 踩坑心得— 为了高速(上)
- git基本操作(fetch,pull,分支使用,初始化项目,恢复)
- JavaScript 踩坑心得— 为了高速(上)
- css 居中
- ace admin java 整合 开发 后台框架,aceadmin_HTML5
- 系统框架 springmvc mybatis Bootstrap html5 shiro maven SS
- hdfs格式化hadoop namenode -format错误
- ArcGIS API for JavaScript开发教程系列(一)之创建地图
- 微信视频开发jquery mobile
- node.js 的事件驱动