问题描述
memcached作为互联网经常使用的缓存工具，大大减少了数据库的压力。但是当并发度较高时，可能会有多个请求都检测到expire，从而发起数据库请求，于是会对数据库造成冲击。
问题重现
当采用传统的设置expire time时，使用httpload工具并发100，在数据库请求函数里sleep 1秒钟。memcache封装里打印是从mc读取还是db读取。expire time设置为2秒。可以看到有重复的数据库请求：
[2016-03-26 10:17:13] NOTICE [1]: mc(127.0.0.1,/test.php) [2016-03-26 10:17:14] NOTICE [1]: mc(127.0.0.1,/test.php) [2016-03-26 10:17:14] NOTICE [1]: db(127.0.0.1,/test.php) [2016-03-26 10:17:14] NOTICE [1]: db(127.0.0.1,/test.php) [2016-03-26 10:17:15] NOTICE [1]: mc(127.0.0.1,/test.php) [2016-03-26 10:17:15] NOTICE [1]: mc(127.0.0.1,/test.php)
解决方法
function call_high($func, $args, $expire=60){/*{{{*/
// Calulate key
$key = md5(json_encode($func).json_encode($args));
// Get from memcached
return $this->get_high($key, $func, $args, $expire);
}/*}}}*/
// lock_expire should longer than db_func running time
function get_high($key/* string or array*/, $db_func/*mixed*/, $db_args/*mixed*/, $expire, $lock_expire=10){/*{{{*/
if (!$this->mem){
return false;
}
$val = $this->get($key);
// if(!$val){ // Maybe empty when FIRST get, otherwise, must something wrong, like the lock&work thread haven't refresh item before expire time. So we use a short lock expire time to allow more thread trying to refresh. }
if ( $val && (!isset($val['t']) || !isset($val['i'])) ){ // don't have expire index? must be wrong. For logic layer to process.
#Core_Log::get_instance()->notice("mc, no expire index");
return $val;
}
if ($val && ($val['t'] - $this->get_high_cal_time($expire) > time())){ // not expired
#Core_Log::get_instance()->notice("mc");
return $val['i'];
}
// expired, allowed one process to refresh it
if (!($this->get_high_lock($key, $lock_expire))){ // other process has refreshing it, so we quit with old data(or may be empty)
#Core_Log::get_instance()->notice("other lock, we quit");
return $val?$val['i']:false;
}
#Core_Log::get_instance()->notice("db");
// get from user func, and set into memcache
$db_info = call_user_func_array($db_func, $db_args); // may be empty, we also set into mc
$this->set_high($key, $db_info, MEMCACHE_COMPRESSED, $expire, $this->get_more_expire_time($expire));
$this->free_high_lock($key);
#Core_Log::get_instance()->notice("save and free lock");
return $db_info;
}
// These two functions should guarantee the refresh thread can do all work before realy expired.
// So the work_time < cal_time + more_time
protected function get_high_cal_time($expire){
return intval(min(10, max(0, $expire/5))); // [0, 10]
}
protected function get_more_expire_time($expire){
return intval(max(100, $expire*0.1)); // [100, ...)
}
function get_high_lock($key, $expire=100){
return $this->mem->add($key.self::HIGH_MC_LOCK_TAIL, 1, 0, $expire); // return true when not exists already, otherwise return false, use as mutex
//return $this->mem->increment($key.self::HIGH_MC_LOCK_TAIL, 1); //can't, because increment will return false, if elem not exists already
}
function free_high_lock($key){
return $this->del($key.self::HIGH_MC_LOCK_TAIL);
//return $this->mem->decrement ($key.self::HIGH_MC_LOCK_TAIL, 1);
}
function set_high($key, $val, $flag, $expire/*second*/, $more_expire=100){
if (!$this->mem){
return false;
}
// save expire time
$expire_time = time()+$expire;
$val = array('t'=>$expire_time, 'i'=>$val);
$expire += $more_expire; // need more expire time, because we don't want return null when get_high
return $this->set($key, $val, $flag, $expire);
}/*}}}*/基本思想就是利用mutex锁，保证仅有一个进程更新该item。
有几个点说明一下：
首次请求时，非refresh线程会返回为空，在业务逻辑上是否可接受？解决方法可以是等待一定时间后再次获取
如果refresh线程工作太久，导致memcached中相应item已经expire了，那么其他线程get from mc即为空，则也会尝试获取lock后从db读取。这里通过设置lock time、cal time和more expire time已解决，允许其他线程也请求db，这种情况一般是数据库阻塞导致的，允许重复读取可能会加大数据库压力，所以存疑。
如果memcached内存不足，导致item expire，则正常情况下也只有一个refresh线程从db读取
如果memcached内存不足，导致lock expire，则会有多个refresh线程从db读取（一般不会设置这么小的memcached吧？）
评论