memcache对hash一致性算法和取模算法详解

memcache是一套分布式的高速缓存系统，主要用来分担数据库压力。下载：

http://www.memcached.org/files/memcached-1.4.25.tar.gz

https://github.com/libevent/libevent/releases/download/release-2.0.22-stable/libevent-2.0.22-stable.tar.gz

依赖：

[root@martin tools]# yum install gcc make cmake autoconf libtool

[root@martin tools]# tar xf libevent-2.0.22-stable.tar.gz
[root@martin tools]# cd libevent-2.0.22-stable
[root@martin libevent-2.0.22-stable]# ./configure --prefix=/usr/local/libevent
[root@martin libevent-2.0.22-stable]# make && make install

安装：

[root@martin tools]# tar xf memcached-1.4.25.tar.gz
[root@martin tools]# cd memcached-1.4.25
[root@martin memcached-1.4.25]# ./configure --prefix=/usr/local/memcached-1.4.25 --with-libevent=/usr/local/libevent/
[root@martin memcached-1.4.25]# make && make install
[root@martin memcached-1.4.25]# cd /usr/local/
[root@martin local]# ln -sv memcached-1.4.25/  memcached

倒入环境变量

[root@martin local]# vim /etc/profile.d/memcached.sh
export PATH=/usr/local/memcached/bin:$PATH
[root@martin local]# .  /etc/profile.d/memcached.sh

启动：

[root@martin ~]# memcached -m 64 -p 11211 -u nobody -vvv
[root@martin ~]# memcached -m 64 -p 11211 -u nobody -d

memcached

-l IP：监听的地址；

-d: 运行为守护进程；

-u: 以指定的身份运行；

-m #: 缓存服务使用的内存空间；默认为64MB

-c #：最大并发连接数，默认为1024；

-p #: 监听的TCP端口，默认11211；

-U #：监听的UDP端口，0表示禁用；

-M：内存耗尽时返回错误而非删除缓存对象；

-f: 指定增长因子；默认为1.25；

-n: chuck起始大小，key+value+flags，默认为48；

php 编译memcached下载：

http://pecl.php.net/package/memcache

http://pecl.php.net/get/memcache-2.2.7.tgz

[root@martin tools]# wget http://pecl.php.net/get/memcache-2.2.7.tgz [root@martin tools]# tar xf memcache-2.2.7.tgz[root@martin tools]# cd memcache-2.2.7#memcache目录下配置configuer[root@martin memcache-2.2.7]# /usr/local/phpfpm/bin/phpize  --with-php-config=/usr/local/phpfpm/bin/php-config[root@martin memcache-2.2.7]# ./configure --with-php-config=/usr/local/phpfpm/bin/php-config[root@martin memcache-2.2.7]# make && make install#################Build complete.#Don't forget to run 'make test'.#Installing shared extensions:     /usr/local/phpfpm/lib/php/extensions/no-debug-non-zts-20131226/[root@martin demo]# vim /etc/php.iniextension=/usr/local/phpfpm/lib/php/extensions/no-debug-non-zts-20131226/memcache.so

常用操作：

# add key flag expire length#value#key 给值起一个名字   key最大250长度 （二进制协议支持65535个字节）#flag 标志 要求一个正整数  序列化成字符串 取出时反序列化  对象／数组／json等#expire 有效期#秒数第n秒后失效#时间戳，到指定的时间戳后消失#0不自动失效 (服务不重启 默认30天，也可能被挤出)#length 缓存的长度（字节为单位）#value  最大1Madd nba 1 0 5maidi#存储过的key不能再次存储  修改用replace#替换replace key flag expire length#参数和add完全一样 不单独写   要是key不存在  就不能写入#setset key flag expire length#相当于集合add  replace#取值get nbaVALUE nba 1 5maidi#删除delete nba   [time]#加 time后，被删除的key  n秒内不能再用，目的是让网站上页面缓存也代谢完毕#把值理解为32位无符号操作  范围在 ［0 － （2 ＊＊32  －1）］#incr 增长incr age 2  #age+2#decr 减少decr age 1 #age-1flush_all  #清空所有对象

状态说明：

statsSTAT pid 924                                       进程IDSTAT uptime 2820                                   服务器运行秒数STAT time 1466125979                               服务器当前unix时间戳STAT version 1.4.24                                服务器版本号STAT libevent 2.0.22-stable                        libevent版本STAT pointer_size 64                               64位服务器STAT rusage_user 0.034488                          进程累计用户时间STAT rusage_system 0.053985                        进程累计系统时间STAT curr_connections 10                           当前打开连接STAT total_connections 11                          曾打开连接数STAT connection_structures 11                      服务器分配的连接构造数STAT reserved_fds 20STAT cmd_get 15                                    ！执行get的次数STAT cmd_set 13                                    执行set的总数STAT cmd_flush 0                                   指向flush_all命令总数STAT cmd_touch 0STAT get_hits 10                                    ！get成功的次数STAT get_misses 5                                   get失败的次数STAT delete_misses 0                                delete未命中次数STAT delete_hits 1                                  delete命中次数STAT incr_misses 0                                  incr未命中次数STAT incr_hits 3                                    incr命中次数STAT decr_misses 0                                  decr未命中次数STAT decr_hits 1                                    decr命中次数STAT cas_misses 0                                   cas未命中次数STAT cas_hits 0                                     cas命中次数STAT cas_badval 0                                   使用擦拭次数STAT touch_hits 0STAT touch_misses 0STAT auth_cmds 0STAT auth_errors 0STAT bytes_read 522                                  读取总字节数STAT bytes_written 480                               写入总字节数STAT limit_maxbytes 67108864                         分配内存数（字节）STAT accepting_conns 1                               目前接受的连接数STAT listen_disabled_num 0STAT threads 4                                       线程数STAT conn_yields 0STAT hash_power_level 16STAT hash_bytes 524288STAT hash_is_expanding 0STAT malloc_fails 0STAT bytes 220                                        存储item字节数STAT curr_items 3                                     现存3个key：valueSTAT total_items 12                                   一共存过12个值STAT expired_unfetched 0STAT evicted_unfetched 0STAT evictions 0                                      为获取空间删除item的总数STAT reclaimed 0STAT crawler_reclaimed 0STAT crawler_items_checked 0STAT lrutail_reflocked 0END

以下一组实验观察hash算法 和取模算法对memcache命中率的影响：HashInterface.class.php

<?phpinterface hash{public function search ($key);public function addNode ($node);public function delNode ($node);}

Consistent.class.php

<?phpinclude 'HashInterface.class.php';class Consistent implements hash{private $nodes = array();  //服务器节点private $vritualNodes ;public function __construct($vritualNodes=64){$this->vritualNodes=$vritualNodes;}/*** 查找节点落在哪台服务器上* @param $key* @return mixed*/public function search ($key){$point = $this->_hashKey($key);//最小的node  起始点$node = current($this->nodes);foreach ($this->nodes as $k => $v){if ($point <= $k){$node = $v;break;}}return $node;}/*** 返回hash健*/private function _hashKey ($str){//把key转为32位符号整数return sprintf('%u', crc32($str));}/*** 添加主机* @param $node*/public function addNode ($node){for ($i = 0; $i < $this->vritualNodes; $i++)$this->nodes[$this->_hashKey($node . '-' . $i)] = $node;$this->_sortNode();}/*** 删除主机* @param $node*/public function delNode ($node){foreach ($this->nodes as $k => $v){if ($v == $node)unset($this->nodes[$k]);}}/*** 节点排序*/private function _sortNode (){ksort($this->nodes, SORT_REGULAR);}public function getNode (){var_dump($this->nodes);}}

Modulo.class.php

<?phpinclude 'HashInterface.class.php';class Modulo implements hash{private $nodes = array();  //服务器节点private $cnt = 0;      //节点个数public function search ($key){$key=$this->_hashKey($key)%$this->cnt;return $this->nodes[$key];}/*** 添加主机* @param $node*/public function addNode ($node){if(!in_array($node,$this->nodes)){$this->nodes[]=$node;$this->cnt +=1;}return true;}/*** 删除主机* @param $node*/public function delNode ($node){if(in_array($node,$this->nodes)){$key= array_search($node,$this->nodes);unset($this->nodes[$key]);//重组健名$this->nodes = array_merge($this->nodes);$this->cnt -=1;}return true;}/*** 返回hash健*/private function _hashKey ($str){//把key转为32位符号整数return sprintf('%u', crc32($str));}}

user.php

<?phpheader("Content-type: text/html; charset=utf-8");set_time_limit(0);function __autoload($class){require('./'.ucfirst(strtolower($class)).'.class.php');}$method=array('add','hitrate','search');$_GET['method']=isset($_GET['method'])?$_GET['method']:' ';if(!in_array($_GET['method'], $method))exit('need method param');//memcached服务器列表$memcacheServers=array();$memcacheServers['one']   = array('host'=>'127.0.0.1','port'=>'11211');$memcacheServers['two']   = array('host'=>'127.0.0.1','port'=>'11212');$memcacheServers['three'] = array('host'=>'127.0.0.1','port'=>'11213');$memcacheServers['four']  = array('host'=>'127.0.0.1','port'=>'11214');$memcacheServers['five']  = array('host'=>'127.0.0.1','port'=>'11215');//算法选择 Consistent Modulo$mode='Consistent';$mode='Modulo';$num=10000;$mode = new $mode();$mem=new Memcache();//存储数据if($_GET['method']=='add'){//添加节点foreach($memcacheServers as $k=>$v)$mode->addNode($k);//添加数据for($i=0;$i<$num;$i++){//计算存储数据的服务器$server=$memcacheServers[$mode->search('key'.$i)];//memcached连接 添加数据$mem->connect($server['host'],$server['port'],2);$mem->add('key'.$i,'value'.$i,0,0);usleep(3000);$mem->close();}exit('data add over');}//计算命中率if($_GET['method']=='hitrate'){$gets=0;  //请求$hits=0;  //命中$rtn=1;   //命中率 默认100%foreach($memcacheServers as $v){$mem->connect($v['host'],$v['port'],2);$rst = $mem->getstats();$mem->close();//0    var_dump($rst);$gets += $rst['cmd_get'];$hits += $rst['get_hits'];}if($gets>0)$rtn= $hits/$gets;echo $rtn;exit();}if($_GET['method']=='search'){foreach($memcacheServers as $k=>$v){$mode->addNode($k);}//模拟少一台服务器$mode->delNode('four');for($i=1;$i<$num;$i++){//$i=$i%$num;$server=$memcacheServers[$mode->search('key'.$i)];$mem->connect($server['host'],$server['port'],2);//$mem->get('key'.$i);if(!$mem->get('key'.$i))$mem->add('key'.$i,'value'.$i,0,0);$mem->close();usleep(3000);}echo "search over";exit;}

html/index.html

<!DOCTYPE HTML><html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><title>Memcached hitrate</title><script type="text/javascript" src="./js/jquery.min-2.1.4.js"></script><style type="text/css">${demo.css}</style><script type="text/javascript">$(function () {$(document).ready(function () {Highcharts.setOptions({global: {useUTC: false}});$('#container').highcharts({chart: {type: 'spline',animation: Highcharts.svg, // don't animate in old IEmarginRight: 10,events: {load: function () {// set up the updating of the chart each secondvar series = this.series[0];setInterval(function () {var x = (new Date()).getTime(), // current timey = parseFloat($.ajax({url:'../user.php?method=hitrate',async:false}).responseText);series.addPoint([x, y], true, true);}, 1000);}}},title: {text: 'Memcached  hit rate '},xAxis: {type: 'datetime',tickPixelInterval: 150},yAxis: {title: {text: 'hitrate'},plotLines: [{value: 0,width: 1,color: '#808080'}]},tooltip: {formatter: function () {return '<b>' + this.series.name + '</b><br/>' +Highcharts.dateFormat('%Y-%m-%d %H:%M:%S', this.x) + '<br/>' +Highcharts.numberFormat(this.y, 2);}},legend: {enabled: false},exporting: {enabled: false},series: [{name: 'Random data',data: (function () {// generate an array of random datavar data = [],time = (new Date()).getTime(),i;for (i = -19; i <= 0; i += 1) {data.push({x: time + i * 1000,y: 1});}return data;}())}]});});});</script></head><body><script src="./js/highcharts.js"></script><script src="./js/exporting.js"></script><div id="container" style="min-width: 310px; height: 400px; margin: 0 auto"></div></body></html>

实验数据：5台服务器，分别添加1w条数据，然后一台down的情况下，观察两种算法观察memcache的命中率hash_consistent数据 对命中率影响较小取模算法实验数据：影响较大：memcache雪崩现象：原因：一般是因为某个节点失效，导致其他节点的缓存命中率下降，缓存中缺的数据去数据库查询，短时间内，造成数据库崩溃，或者，是由于缓存周期性失效造成事故发生后，可能还会发生的事：重启db，期间又被压垮，但是缓存数据多了一些， 重启多次后，缓存重建完毕，db稳定运行解决方案：1.可以将失效时间 设置在一个范围内，不让其同时失效2.若同时失效，可以将失效时间设置在半夜，然后脚本刷新数据，建立缓存 （方案一更好）php可以对addServer 简单实现分布式配置默认是基于权重方式还可以在PHP.ini中,配置分布式一致性memcache.hash_strategy = consistent