Linux下多网卡不同IP在同一网段的情况

今天在看Intel 1000 PT网卡的驱动时，有如下信息：

Multiple Interfaces on Same Ethernet Broadcast Network
echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter

google了一下，有人已经试验了这个，摘录如下：

http://blog.sina.com.cn/s/blog_544465b0010000kf.html

多网卡在同一网段内的问题(2005-11-23 23:38:42)

分类：网络安全

关键字：Strict Interface ARP, Multi-link, Multipath, Multiple network cards on same subnet problem

公司那个提供音乐下载的域名流量直逼 1Gbps，但是系统的连接却不高，服务器用的是DELL2850的，板载两个Gbe的网口，还插了两块Intel的Gbe网卡，因此打算通过多网卡的负 载均衡扩充一下系统的网络带宽。但是不想用多网卡的邦定，感觉那样不太灵活，因此决定采用Advanced Routing来解决这个问题。

查了一下 man ip，然后开始做了：
1) 给四个网口配置了四个地址
2) 在BIGIP上面把四个地址都加入到负载均衡
3) 用ip命令实现多网卡负载均衡
ip route replace default equalize scope global nexthop via 172.24.x.11 dev eth0 weight 1 nexthop via 172.24.x.11 dev eth1 weight 1 nexthop via 172.24.x.1 dev eth2 weight 1 nexthop via 172.24.x.11 dev eth3 weight 1

立即看到所有的流量被均衡到4个网口

但是出现了新的问题，大量的IP地址冲突信息出现在BIGIP的日志：
Nov 23 19:18:56 tc04 kernel: arp info overwritten for 172.24.x.30 by 00:14:22:1b:94:dc

Nov 23 20:30:24 tc04 kernel: arp info overwritten for 172.24.x.30 by 00:04:23:c0:5c:88

Nov 23 20:30:28 tc04 kernel: arp info overwritten for 172.24.x.30 by 00:14:22:1b:94:dc

Nov 23 21:03:14 tc04 kernel: arp info overwritten for 172.24.x.30 by 00:14:22:1b:94:dd

发现出现的几个产生IP冲突的MAC地址都是这台机器上面其他的网口的MAC，这就奇怪了。
通过tcpdump抓包，分析ARP信息发现是因为Linux响应ARP请求的时候，使用的IP地址并未被严格的限定，因此导致了这个问题。

很快 Google 到这篇文章：[PATCH] strict interface arp patch for Linux 2.4.2

然后就跑去 http://www.linuxvirtualserver.org/ 找到 http://www.linuxvirutalserver.org/docs/arp.html

发现没有什么可用的信息，然后就硬着头皮进入到 /proc/sys/net/ipv4/conf/all 目录看看有啥线索是解决2.6的

看到三个文件 arp_announce, arp_ignore, arp_filter
立即去 /usr/src/linux/Documentation/networking/ip-sysctl.txt 找到相关的说明：
arp_filter - BOOLEAN

1 - Allows you to have multiple network interfaces on the same

subnet, and have the ARPs for each interface be answered

based on whether or not the kernel would route a packet from

the ARP'd IP out that interface (therefore you must use source

based routing for this to work). In other words it allows control

of which cards (usually 1) will respond to an arp request.
0 - (default) The kernel can respond to arp requests with addresses

from other interfaces. This may seem wrong but it usually makes

sense, because it increases the chance of successful communication.

IP addresses are owned by the complete host on Linux, not by

particular interfaces. Only for more complex setups like load-

balancing, does this behaviour cause problems.
arp_filter for the interface will be enabled if at least one of

conf/{all,interface}/arp_filter is set to TRUE,

it will be disabled otherwise
arp_announce - INTEGER

Define different restriction levels for announcing the local

source IP address from IP packets in ARP requests sent on

interface:

0 - (default) Use any local address, configured on any interface

1 - Try to avoid local addresses that are not in the target's

subnet for this interface. This mode is useful when target

hosts reachable via this interface require the source IP

address in ARP requests to be part of their logical network

configured on the receiving interface. When we generate the

request we will check all our subnets that include the

target IP and will preserve the source address if it is from

such subnet. If there is no such subnet we select source

address according to the rules for level 2.

2 - Always use the best local address for this target.

In this mode we ignore the source address in the IP packet

and try to select local address that we prefer for talks with

the target host. Such local address is selected by looking

for primary IP addresses on all our subnets on the outgoing

interface that include the target IP address. If no suitable

local address is found we select the first local address

we have on the outgoing interface or on all other interfaces,

with the hope we will receive reply for our request and

even sometimes no matter the source IP address we announce.
The max value from conf/{all,interface}/arp_announce is used.
Increasing the restriction level gives more chance for

receiving answer from the resolved target while decreasing

the level announces more valid sender's information.
arp_ignore - INTEGER

Define different modes for sending replies in response to

received ARP requests that resolve local target IP addresses:

0 - (default): reply for any local target IP address, configured

on any interface

1 - reply only if the target IP address is local address

configured on the incoming interface

2 - reply only if the target IP address is local address

configured on the incoming interface and both with the

sender's IP address are part from same subnet on this interface

3 - do not reply for local addresses configured with scope host,

only resolutions for global and link addresses are replied

4-7 - reserved

8 - do not reply for all local addresses
The max value from conf/{all,interface}/arp_ignore is used

when ARP request is received on the {interface}

嘿嘿，通过：

echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore

or

echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter

立即搞定！

继续 Google 了一些信息，发现前面两个内核控制好像是在 2.6.5 实现的 :-)
推荐用前面的两个，arp_filter应该也能解决，感觉前面的两个是对arp_filter的更细节控制的实现。

然后还发现在inter pro/1000的网卡驱动的安装指南里面也有一节：
Multiple Interfaces on Same Ethernet Broadcast Network

------------------------------------------------------
Due to the default ARP behavior on Linux, it is not possible to have

one system on two IP networks in the same Ethernet broadcast domain

(non-partitioned switch) behave as expected. All Ethernet interfaces

will respond to IP traffic for any IP address assigned to the system.

This results in unbalanced receive traffic.
If you have multiple interfaces in a server, either turn on ARP

filtering by entering:
echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter

(this only works if your kernel's version is higher than 2.4.5),
or,
install the interfaces in separate broadcast domains (either

in different switches or in a switch partitioned to VLANs).

参考信息：
http://linux-ip.net/html/ether-arp.html
http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.arp_problem.html
http://www.ssi.bg/~ja/#arp_announce