Berkeley DB 源代码分析 (5) --- 事务锁模块

Locking Subsystem Learning Notes

0. locking API

__db_lput/__db_lget are txnal lock put/get, often __TLPUT is called instead,

and __TLPUT calls __db_lput internally. __db_lput will downgrade the lock

rathter than simply releasing it if the db supports dirty reads and the lock

is a write lock; in other circumstances it will release the lock.

__LPUT and __ENV_LPUT simply calls __lock_put, thus the lock is simply

released.

1. So far we only use intentional locks in cds mode. This is because we are not using row level locking in tds mode, we only lock pages in tds mode. (what about queue am?)

2. Need to provide row level locking API, like the existing __db_lget/__db_lput for page level locking. Because qam is using record locking now, it uses __db_lget with a DB_LOCK_RECORD flag to lock a record. However __db_lget can't be directly used because
it's locking a db-wide recno, so (dbfileuid, recno) is sufficient to be the lock obj, but in the new am we don't have global recno, we will lock (dbfileuid, pgno, in-page-indx).

qam does not lock pages, it always lock (dbfileuid, recno), so it does not

need intentional locking either.

1. So far we only use intentional locks in cds mode. This is because we are not using row level locking in tds mode, we only lock pages in tds mode. (what about queue am?)

2. Need to provide row level locking API, like the existing __db_lget/__db_lput for page level locking. Because qam is using record locking now, it uses __db_lget with a DB_LOCK_RECORD flag to lock a record. However __db_lget can't be directly used because
it's locking a db-wide recno, so (dbfileuid, recno) is sufficient to be the lock obj, but in the new am we don't have global recno, we will lock (dbfileuid, pgno, in-page-indx).

qam does not lock pages, it always lock (dbfileuid, recno), so it does not

need intentional locking either.

3. __db_ilock(DB_LOCK_ILOCK) is the lock object structure used inside DB to represent lock

info, all locks used by db internally are managed using this type e.g. the lock list.

We need to add a in-page-index to lock a row.

__db_lock is the struct in the lock region, used to store each lock's info

inside lock subsystem, to manage database locks;

DB_LOCK is used in a lock request used by outside caller to get/put a lock. it

contains offset pointer for us to find the corresponding internal __db_lock.

4.

the $db/lock/Design is obsolete.

Firstly, there are multiple lock table partitions, each of them has __db_lockpart structure,

containing partition specific info, and containing some of

the free lock objects and free locks. So getting a free lock object/lock can

be parallel in multiple partitions.

The main lock region has __db_lockregion structure, containing global info of the lock region; And it

contains the locker free list, lockers are not put into multiple partitions. It also contains the

locker/lock object hash tables, as expected. also it contains the lock table

partition array. Note that all the partitions live in the same region file as

the main lock region.

There are no direct bucket mutex for each bucket in lockobj hash table,

rather, we use lock partition's mutex to protect a specific bucket of

lockobjs. So if there are same number of partitions and lockobj buckets,

the concurrency of lockobj mgmt is not affected; However partition can be far

less than lockobjs and locks, it's by default 1 on single cpu systems and

10*cpu-number on multicpu systems. I think manually setting a big lock

partition number can promote concurrency, though the doc says otherwise.

And we use only one mutex to protect the entire lockers, so concurrency is

affected. Especially when there are a lot of short but frequent txns, e.g.

when mvcc is turned on; However this must be something we have to do to avoid other issues.

The stat info for lock region are divided into 3 parts, first the DB_LOCK_STAT

contains general stat info and lives in main lock region (__db_lockregion);

And we have an array of stats for lock object hash buckets(DB_LOCK_HSTAT) and

lives in main lock region too, there are same number of lockobj hash buckets

and lockobj stat structures;

lastly we have a DB_LOCK_PSTAT for each lock table partition (in

__db_lockpart).

The reason to divide stat into multiple parts is to bring up concurrency, when

reporting stat into, we integrate all the pieces into a DB_LOCK_STAT in

__lock_stat function.

Both locks and lockobjs have generation number, in order to allow reusing them

and at the same time avoid the issue that a previous use happen at the same

time with a reuse. Locker structs can be reused too but they don't need

generation numbers because each locker has a unique locker id.

5. __lock_region_mutex_count

total mutex needed in lock subsystem: max-number-of-locks +

number-of-partitions + 3. Each lock has a mutex for the thread of control to

block on; each parition has its mutex; lockregion, lockers mgmt and dead lock

detector needs 1 respectively.

6. __lock_region_size

So far we allocate more hash buckets than the things to be stored in the hash

table. this is true for locker/lockobj hash tables.

If the hash is good enough (simply modular operator % will be good enough), we

can put each locker and lockobj in a hash bucket, i.e. we never need to

traverse the bucket list for a locker/lockobj. Consequently there can be potentially a huge

number of hash bucket, and I think this is one of the reasons we don't use one

mutex for each hash bucket.

7. __lock_env_refresh

It frees all allocated memory space if using private env. From here we can see what

structures we allocated. The only issue I find is: we are only freeing free

lockers, in-use lockers are not freed. The in-use lockers normally are freed

when the locker is closed(txn end; db/dbc handle close), but may not be freed

if above close op does not execute.

8. __lock_region_init

Init shared lock region structure (__db_lockregion), allocate mutexes,

and init all members.

Allocate memory for locker/lockobj hash tables, lock partition array and lockobj hash bucket stat array, all lockers, locks and lockobjs, and link locks and lockobjs into free list of each partition evenly, linking lockers into a free list.

This function is only called when we are creating rather than joining a lock

region.

9. __lock_open

Open and init __db_locktab structure, which is the per-process struct for lock region. call __lock_region_init to create the lock region if needed.

10. __lock_fix_list -- sort lock req 's objs

this is the only function in use in file locklist.c. It arrange lockobjs in

the lockreq parameter of lock_vec so that they are grouped/sorted by file.

11. __lock_promote

Looks in a lockobj's waiter list WL for any locks L that doesn't conflict with any

locks in the same lockobj's holder list HL, and move such an lock L from WL to

HL, and mark L as DB_LSTAT_PENDING, and unlock the L->mtx_lock mutex so that the corresponding thread of control can continue executing in OS. And if the lockobj 's WL is empty and it is in a DD's dd_objs

list, remove it from the dd_objs since it won't block anyone.

It should be called whenever a lock becomes available so that some waiters may hold the lock and continue, (i.e. it is used as a scheduler), e.g. a lock in the lockobj's holder list is released, downgraded, or given back to the parent txn.

12. __lock_inherit_locks

Return a child txn's locks back to its parent txn. For each lock L the child

txn CT has, it L is also held by the parent txn T, then the L->refcount++;

else L is put into T's locker's lock list. Do a scheduling by calling

__lock_promote at the end so that any sibling txns waiting for any returned

locks can go on.

13. __lock_remove_waiter

Remove a lock from a lockobj's waiter list. if the lockobj has no locks in its

waiter list, remove it from dd's dd_objs list because it won't block any one.

At last, unlock the lock's mutex, so that any thread of control waiting for

the mutex can continue. A lock's mutex is used to suspend the thread of

control T for it to wait for the lock. if we want T to suspend, the code to

acquire the lock->mtx_lock is executed in T and thus T suspends, since the

lock->mtx_lock was created and mutex-locked. and whenever we want any thread

of control to go since they got this lock, we mutex-unlock the lock->mtx_lock,

and any such T does not mutex-unlock this mutex, so next time we can still

suspend such a T by having it acquire the lock->mtx_lock.

14. __lock_trade

transfer a lock from a locker A to another locker B, so that the lockobj A

locks now is locked by B.

15. __lock_allocobj and __lock_alloclock

__lock_allocobj Allocate lockobj from a partition other than the specified one P. we do so

because P does not have free lockobj space, and we want to find such a free

lockobj space from other partitions and move it to P's free list. Such an

action is called a lockobj 'steal' seen from the stat results.

Similarly we have __lock_alloclock. Free locks are also in partitions, so we

iterate each partition to find one free space for a lock, and steal it into

the current partition.

16. __lock_getobj

Find a specified lock obj from lockobj hash table. If the lockobj does not

exist yet and we will create one, first allocate a lockobj space from a partition P.

If P does not have free lockobj space, steal from other partitions; then init

the lockobj fields.

Note that lock subsystem allows external use, so the size

of a lockobj may not be standard (sizeof (__db_ilock)). so if the external

lock obj is larger than sizeof(db_ilock), we allocate memory from lock region and store the (off, size) into lockobj->lockobj. then we always copy bytes into either lockobj->lockobj.off or lockobj->objdata.

And we will need to free the region space when releasing a lockobj, if we

found lockobj->lockobj.size > sizeof(lockobj->objdata).

17. __lock_freelock

Optionally free a lock L from its locker's lock list, and optionally put it

into its partition's free lock list. The lock's mutex is destroyed.

Question: what does lock promotion mean?

A locker's info is stored in shared mem, and txns/db/dbc handles only store a

locker ID, when the locker is used inside locking subsystem, we find the

locker via __lock_getlocker and use it only inside locking subsystem.

18. __lock_put_pp, __lock_put, __lock_put_nolock, __lock_put_internal

__lock_put_internal is the ultimate funtion called. all the rest are wrappers.

__lock_put_internal releases a lock regarding/disregarding its reference count. Note that any lock/lockobj release will increment its generation to reuse it and prevent issues from the reuse.

It first unlink the lock from its lockobj's wait/holder list. If it was in holder list,

we call __lock_promote to schedule so that waiters can proceed. then if the

lockobj has no locks in both its watier and holder list, free the lockobj.

Finally call __lock_freelock to free the lock, it will be removed from its

locker's locklist.

19. __lock_get_pp, __lock_get, __lock_get_api, __lock_get_internal

__lock_get_internal is the ultimate function that does the tasks. It decides

whether a lock should be granted to the locker. If not, the thread of control

executing this function blocks on the requested lock's mutex. The locker and

lockobj are supposed to be both already created in the lock region.

Figure out if we can grant this lock or if it should wait.

By default, we can grant the new lock if it does not conflict

with anyone on the holders list OR anyone on the waiters list.

The reason that we don't grant if there's a conflict is that

this can lead to starvation (a writer waiting on a popularly

read item will never be granted). The downside of this is

that a waiting reader can prevent an upgrade from reader to writer,

which is not uncommon.

There are two exceptions to the no-conflict rule. First, if

a lock is held by the requesting locker AND the new lock does

not conflict with any other holders, then we grant the lock.

The most common place this happens is when the holder has a

WRITE lock and a READ lock request comes in for the same

locker.

If we do not grant the read lock, then we guarantee deadlock.

Second, dirty readers are granted if at all possible while

avoiding starvation, see below.

In case of conflict, we put the new lock on the end of the

waiters list, unless we are upgrading or this is a dirty reader in

which case the locker goes at or near the front of the list.

lock holders on a lockobj have compatible locks, so, if the locks are

DB_LOCK_READ, other locks are read locks too. that's why the grant_dirty

variable assignment only tests one lock.

locks can be downgraded from DB_LOCK_WRITE to DB_LOCK_WWRITE. a page can't be

read by a dirty reader when it's DB_LOCK_WRITE. when we are done writing the page

but txn not ended yet, and in dirty read txns, such action means the lock is

downgraded to DB_LOCK_WWRITE, and thus it can be read by dirty readers. such

behavior prevents reading an invalid page --- during the memory manipulations

to the page it is completely broken/invalid to other txns. For the same

reason, when a page is being dirty-read, it should not be write otherwise

there maybe memory errors, thus, dirty read lock conflicts with WRITE locks;

and it should not be normally-read because the page is dirty, but the conflict

matrix does not prevent so, just no locker does this.

when a locker get its requested lock, it's status is set to DB_LSTAT_PENDING.

and when the thread of control actually runs, it's stat will be set to

DB_LSTAT_HELD. when it's waiting for a lock it's stat is DB_LSTAT_WAITING.

DB_LOCK_UPGRADE is a flag modifying DB_LOCK_WRITE, meaning the locker wants to

upgrade from READ to WRITE lock; it's not a type of lock.

op flow:

1. decide to grant/wait a lock, and where to wait in the waiter list of the

lockobj;

2. create lock struct in lock region and wire up with locker/lockobj.

3. if granted the lock, continue, return; otherwise, do deadlock detect before

waiting for the lock(acquiring the lock's mutex, which will suspend the thread

of control since that mutex was acquired when the lock was created); when got

the lock and continue, handle time expires and other details.

Guidelines:

1. Reduce deadlocks

If a locker already locks a lockobj and it wants more locks on it, it should

get the locks ASAP, otherwise cyclic wait has a better chance to form.

so if the new lockreq doesn't conflict with any other holders on the lockobj,

it gets the lock, otherwise it wait as the first one and will get the lock

earliest.

also if the lockreq wants to upgrade to write lock, it should

have higher priority to proceed --- if it conflict with a holder, it's the 1st

in the waiter list; otherwise it get the lock, no need to be compatible with

all waiters. By doing so, lock upgrade can complete asap, otherwise it's more

likely to starve than read locks, read locks can starve lockers who want

to upgrade locks.

2. Avoid starvation

A lockreq should not conflict with any holders. and it shouldn't conflict with

any waiters (except the above 2 situations) so that waiters don't starve.

3. Promote throughput

For dirty readers(lockers that has DB_READ_UNCOMMITTED), they should get locks

asap, which is why they want to read dirty data. thus they will be put at the

1st or 2nd in the waiter list, after the write lock.

20. __lock_vec_pp, __lock_vec_pp, __lock_vec_api, __lock_vec

__lock_vec is the ultimate function that does the tasks, and it calls

__lock_put_internal and __lock_get_internal to do the work.

DB_LOCK_WAIT: an internal type of lock used to lock nothing, but wait for an

event to happen. such a lock is used as a condition var.