Berkeley DB 源代码分析 (2) --- Btree的实现 （1）

II. Type Dictionary

1. BTREE

The DB handle's DB->bt_internal structure, stores per-process and per-dbhandle

btree info and function pointers.

2. BTMETA

The btree meta page structure shared by all processes. It stores what's in the

btree's meta page, including all btree specific global(btree-db-wide) info and

common AM db-wide global info.

3. BTREE_CURSOR

It contains the common cursor fields defined in macro __DBC_INTERNAL which are shared

by all types of cursors, and mainly a page stack for btree searching.

In __dbc type, we use a pointer to the __dbc_internal type which is defined as the "base class" for all cursor types, but

actually we allocate memory for BTREE_CURSOR or other types, and cast to

specific cursor types before actually using them. We never directly use

__dbc_internal type.

III. Macro Dictionary

P_INIT: init a non-meta page.

DBC_LOGGING: Find via a cursor if using logging.

LCK_COUPLE: a parameter to __db_lget. If specified, in __db_lget we will first

release the lock then aquire the lock to the same lockobj with specified lock

mode.

DBC_DOWNREV: In replication it is allowd that the master have lower version DB lib than replicas.

So if the master uses DB versions older than the version which first

have latching support, replicas will notice this and set this flag to all its cursors, and replicas

will use traditional mutex locking rather than shared latches.

STD_LOCKING: Whether to use std locking, that is, the locking subsystem is started, and we are not using CDS, and the cursor is not sitting on an

off-page-duplicate apge/tree.

DB_MPOOL_TRY: a flag for __memp_fget, telling __memp_fget to try get a latch on the target page's buffer.

If the latch is not granted, return DB_LOCK_NOTGRANTED immediately without waiting.

IV. Function Dictionary

0. __bam_open

Open a btree database. It simply calls __bam_read_root after some config

checks.

1. __bam_read_root

get the btree db's metadata page and use info in it to init the BTREE structure of the DB->bt_internal. The meta data's info was filled in general

DB->open call before calling __bam_open.

2. __bam_metachk

Checks a btree meta data page validity.

3. __bam_init_meta

Init a meta data page's fields, i.e. the BTMETA structure's fields. Called

whenever a metadata page is created during btree db open procedures. For other

pages than meta pages, we use P_INIT to init them.

4. __bam_new_file

Routed from __db_new_file.

Create a btree db file by initializing its meta page and root page. Called

during db open process and routed from __db_new_file when db is a btree db.

The db may be in memory or not. For inmem db, we create the page from cache

and mark it dirty (mark this in __memp_fget rather than after actually writing

to it otherwise the page may get evicted before we had a chance to mark it.);

For on-disk db files, we don't use cache for now, rather, we put the page in

private memory to init, and directly write the pages into the db file using __fop_write.

when writing pages directly via __fop_writ/__fop_read, we should call the

internal common page in/out functions after got the page via __fop_read and

before writing the page via __fop_write. The __memp_fget/__memp_fput functions

call them too, as registered callbacks via __memp_pg. We have internal page in/out

callbacks for the 3 types of databases(btree, hash, queue), the internal page in/out functions mainly do

check summing and page header byte swap, so that database files created in

big-endian machines can be opened on little-endian machines, though the user

data are never swapped, so users need to make sure the bytes they get are correct.

There are AM specific work to do in internal page in/out functions, so we have

a __db_pgin/__db_pgout pair(placed in db/db_conv.c), in which they call AM specific pgin/out functions

like __bam_pgin/__bam_pgout (placed in btree/btree_conv.c, note the file name

convention).

The reason we use __fop_write here, is that at this point, the db is not fully

opened, it's not registered in the mpool region yet.

__memp_fget/put functions do not do logging, so before putting a dirty page

back to the cache, we should log changes; __fop_write logs the action, so no

need to do it in __bam_new_file.

QUESTION:

1. In this function we didn't lock the meta/root pages but use latches, why? don't we want txnal semantics?

2. Generally how do we guarantee txnal sementics when we release metapage

non-txnal locks immediately after use? (this is good for performance, but how to enture

consistence? ) examples are __bam_read_root, __bam_new_subdb, and __db_new.

These locks are not txnal, why can't they be replaced by latches?

In the DBMETA general meta info, only the "last_pgno", and "free", and

"key_count" and "record_count" can be updated, others are static fields. AM

specific parts have several more, for btree they are "root", "iv" and "chksum". So if

these fields don't require txnal locks, it's OK to release locks before txn

ends.

5. __bam_new_subdb

Routed from __db_init_subdb. It init the subdb's meta and root pages. It

locks the subdb's meta page during the entire function.

When this function is called, the db file is registered into the mpool so we

always use __memp_fget/put to read/write the page.

It calls __db_new to get a page.

Other than above, it's quite like __bam_new_file.

6. __db_new

__db_new prefer free pages in db file, and

falls back to allocating a new page by extending the db file. __db_new is

seldom called because it writes the db file's metadata page, which becomes a

bottle neck and is expensive, thus there can be many free pages but we are

extending the db file.

7. __db_free

Free a page and put it into the free list.

8. __bamc_close

See I.3.

9. __memp_dirty

Mark a page dirty.

10. __bamc_del

Mark the key/data pair with B_DELETE on the page containing it, and then

mark all cursors sitting on the key/data pair with C_DELETED via __bam_ca_delete.

But do not delete it yet or decrement the number of entries in the page, the k/d will

be deleted by the last cursor sitting on it if it is closed at this position. I think we should delete it if we find via __bam_ca_delete that

no other cursors are sitting on it.

Whenever we modify a page, we first lock the page via __db_lget, then get the page from cache via __memp_fget, then optionally mark the

page dirty via __memp_dirty, then log the action using various logging functions, followed by actually/effectively modifying the page.

Then we call __memp_fput to return the page back to the mpool, finally we release lock on the page.

QUESTION: how are key/data pairs deleted? this function only marks k/d

"deleted", but don't delete them from db pages. __bamc_close only deletes the

k/d it sits on when closed, but other k/d marked deleted by the cursur are not

even deleted when the cursor is closed. so when are all of them deleted from

page?

11. __bamc_count

When counting, consider B_DELETE items, don't count them.

12. __bamc_physdel & __bam_ditem

Physically delete a key/data pair, called when the last cursor sitting on the

deleted key/data pair is closed.

We call __bam_ditem twice to delete a key/data pair, and we log the op in

__bam_ditem. Following each __bam_ditem call, we call __bam_ca_di to adjust

other cursors of this database. We don't have a function to delete a k/d pair

from a btree leaf page at once, I think we should have such a function.

Internal btree pages only has a single structure to store the key and pageno,

they don't exist in pairs. actually except for btree leaf pages(P_LBTREE), all

other data items exist in single.

__bam_ditem alters the btree page's index array according to the type of btree

pages, and decrement the number of entries in the page, then calls __db_ditem to remove

the item from the page and log the action. or calls __db_doff to delete a opd overflow item.

from the overflow page and free the overflow page.

When deleting the last key/data pair from a btree leaf page, the page itself,

and potentially the stack of pages leading from root node to this leaf node

need to be deleted. So we note down the last key K by calling __db_ret to get

the k/d pair, and then delete this last k/d

pair by calling __bam_ditem twice, each followed by __bam_ca_di to adjust

cursors. Then, we search that last key K from root, when we complete the search,

we have in dbc->dbc_internal the stack/path of nodes leading to

this node, and we should delete several nodes in the stack --- imagine the leaf page P2's parent page P1

also has only one item, when we delete P2, we also delete P1's last item, thus

delete P1, and so on.

13. __bam_stkrel

Release pages in the search stack of the cursor, put each page back to mpool

and optionally unlock each page.

14. __bamc_get

the effective part of DBC->get.

According to the flags, dispatch calls to __bamc_prev, __bamc_next,

__bamc_search, or simply get page. The impl _DUP to is quite straigtforward,

by simply comparing adjacent keys; similarly for NO_DUP flags, it simply

iterate the k/d pairs with identical keys util got a different key.

15. __bamc_prev, __bamc_next

get from next/prev page, or from current page. alter DBC->dbc_internal's pgno

and indx. Note that in the 2 functions we may be on an opd page or a btree

ordinary leaf page.

The 2 functions plus __bamc_search only read data, they don't effectively

modify the page, so by default if we need to get another page, we read-lock

it, unless DB_RMW is set, and we would write-lock it.

the 2 funcs can skip empty pages, and deleted k/d pairs or key items in

btree internal pages.

QUESTION: Strangely enough, a k/d marked deleted is not physically deleted even when the

cursor moves away from it. so when is it deleted?