Freezing Your Tuples Off 之 vacuum_freeze_min_age
2016-01-01 19:34
323 查看
The vacuum_freeze_min_age setting determines the youngest XID which will be changed to FrozenXID on data pages which are being vacuumed anyway. The advantage of setting it low is that far more XIDs will already be frozen when the data page is finally evicted from memory. This is ideal from a maintenance perspective, as the data page may never need to be read from disk for freezing. The disadvantage of setting it too low is additional time and CPU used during vacuum, especially if the page ends up being vacuumed several times before it's written out.
The other disadvantage of setting it low, according to the pgsql-hackers list, is that you will have less data for reconstructive database forensics if your database gets corrupted. However, given PostgreSQL's very low incidence of corruption bugs, this is not a serious consideration when balanced against the very high cost of a vacuum freeze on a really large table.
Therefore, we want the setting to be low, but not so low that XIDs used a few minutes ago are expiring. This is where things get difficult and not auto-tunable in current Postgres. You really want XIDs to freeze after a few hours, or maybe a day if your application makes use of long-running transactions frequently. But Postgres has no single-step way to determine how many XIDs you use per hour on average.
The best way is to monitor this yourself. In recent versions of PostgreSQL, you can get this from pg_stat_database, which has the counters xact_commit and xact_rollback. If these are part of a monitoring scheme you already have in place (such as Nagios/Cacti, Ganglia or Munin), then you can look at the transaction rates using those tools. If not, you need to follow these three steps:
注:使用pg_stat_database中的xact_commit和xact_rollback字段计算xid耗费多少是不对的,因为这两个字段中包含了vxid。
关于vxid,可参见Understanding virtualxid && transactionid
1. Run this query, and write down the result:
2. Wait three or four hours (or a day, if you use long-running transactions)
3. Run the query again.
4. Subtract the number from the first query run from the second query run. If the 2nd number is lower, then you've wrapped around zero, and should try again.
5. Round up to the nearest multiple of 10.
So, as an example:
So my transaction burn rate is 1,061,936 for four hours, which I round to 1,000,000. I then set vacuum_freeze_min_age to 1000000. The approximate burn rate, 250,000 per hour, is also a handy figure to keep to figure out when XID wraparound will happen next (in about 1 year, if I was starting from XID 3).
注:此处指autovacuum_freeze_max_age设定为20亿。故会在一年后才会发生xid回卷。
Without doing the above, it's fairly hard to estimate a reasonable level, given that XID burn rate depends not only on the amount of write activity you're doing, but how you're grouping the writes into transactions. For example, in a data collection database, if you're doing each imported fact as a separate standalone INSERT, you could be burning a million XIDs per hour, but if you're batching them in batches of a thousand rows, that cuts you down to 1000 XIDs per hour. That being said, if you really don't have time to check, here's my rules-of-thumb settings for vacuum_freeze_min_age:
Low write activity (100 per minute or less): 50000
Moderate write activity (100-500 per minute): 200000
High write activity (500 to 4000 per minute): 1000000
Very high write activity (higher than 4000 per minute): 10000000
You'll notice that all of these are lower than the default which ships in postgresql.conf. That default, 100 million, is overly conservative, and means that preemtive freezing almost never happens on databases which run with the defaults. Also, the default of 100m is half of 200m, the default for autovacuum_freeze_max_age, meaning that even after you've completely vacuumed an entire table, you're left with many XIDs which are 50% of freeze_max_age old. This causes more wraparound vacuums than are necessary.
Also, to some degree, this isn't worth worrying about below 500 writes/minute, given that it takes 8 years to reach XID wraparound at that rate. Few PostgreSQL installations go 8 years without a dump/reload.
注:
1、可以使用下面的sql查询数据库最大frozenxid :
2、使用如下sql查询表大小大于1G,frozenxid 最大的前20个:
参考:
http://www.databasesoup.com/2012/10/freezing-your-tuples-off-part-2.html
The other disadvantage of setting it low, according to the pgsql-hackers list, is that you will have less data for reconstructive database forensics if your database gets corrupted. However, given PostgreSQL's very low incidence of corruption bugs, this is not a serious consideration when balanced against the very high cost of a vacuum freeze on a really large table.
Therefore, we want the setting to be low, but not so low that XIDs used a few minutes ago are expiring. This is where things get difficult and not auto-tunable in current Postgres. You really want XIDs to freeze after a few hours, or maybe a day if your application makes use of long-running transactions frequently. But Postgres has no single-step way to determine how many XIDs you use per hour on average.
The best way is to monitor this yourself. In recent versions of PostgreSQL, you can get this from pg_stat_database, which has the counters xact_commit and xact_rollback. If these are part of a monitoring scheme you already have in place (such as Nagios/Cacti, Ganglia or Munin), then you can look at the transaction rates using those tools. If not, you need to follow these three steps:
注:使用pg_stat_database中的xact_commit和xact_rollback字段计算xid耗费多少是不对的,因为这两个字段中包含了vxid。
关于vxid,可参见Understanding virtualxid && transactionid
1. Run this query, and write down the result:
SELECT txid_current();
2. Wait three or four hours (or a day, if you use long-running transactions)
3. Run the query again.
4. Subtract the number from the first query run from the second query run. If the 2nd number is lower, then you've wrapped around zero, and should try again.
5. Round up to the nearest multiple of 10.
So, as an example:
josh=# select txid_current(); txid_current -------------- 1000811 ... wait four hours ... postgres=# select txid_current(); sum --------- 2062747
So my transaction burn rate is 1,061,936 for four hours, which I round to 1,000,000. I then set vacuum_freeze_min_age to 1000000. The approximate burn rate, 250,000 per hour, is also a handy figure to keep to figure out when XID wraparound will happen next (in about 1 year, if I was starting from XID 3).
注:此处指autovacuum_freeze_max_age设定为20亿。故会在一年后才会发生xid回卷。
Without doing the above, it's fairly hard to estimate a reasonable level, given that XID burn rate depends not only on the amount of write activity you're doing, but how you're grouping the writes into transactions. For example, in a data collection database, if you're doing each imported fact as a separate standalone INSERT, you could be burning a million XIDs per hour, but if you're batching them in batches of a thousand rows, that cuts you down to 1000 XIDs per hour. That being said, if you really don't have time to check, here's my rules-of-thumb settings for vacuum_freeze_min_age:
Low write activity (100 per minute or less): 50000
Moderate write activity (100-500 per minute): 200000
High write activity (500 to 4000 per minute): 1000000
Very high write activity (higher than 4000 per minute): 10000000
You'll notice that all of these are lower than the default which ships in postgresql.conf. That default, 100 million, is overly conservative, and means that preemtive freezing almost never happens on databases which run with the defaults. Also, the default of 100m is half of 200m, the default for autovacuum_freeze_max_age, meaning that even after you've completely vacuumed an entire table, you're left with many XIDs which are 50% of freeze_max_age old. This causes more wraparound vacuums than are necessary.
Also, to some degree, this isn't worth worrying about below 500 writes/minute, given that it takes 8 years to reach XID wraparound at that rate. Few PostgreSQL installations go 8 years without a dump/reload.
注:
1、可以使用下面的sql查询数据库最大frozenxid :
select max(age(datfrozenxid)) from pg_database;
2、使用如下sql查询表大小大于1G,frozenxid 最大的前20个:
SELECT relname, age(relfrozenxid) as xid_age, pg_size_pretty(pg_table_size(oid)) as table_size FROM pg_class WHERE relkind = 'r' and pg_table_size(oid) > 1073741824 ORDER BY age(relfrozenxid) DESC LIMIT 20;
参考:
http://www.databasesoup.com/2012/10/freezing-your-tuples-off-part-2.html
相关文章推荐
- MacOS下质数的判断方法
- HTML简单介绍
- 【设计模式】简单工厂模式
- 资源路径
- 物体检测与识别-Faster RCNN
- php 图片上传安全探讨
- Java scanner split useDelimiter 用法小议
- Python Concurrent Programming
- Codeforces Round #332 (Div. 2) B. Spongebob an Joke (水)
- leetcode112---Path Sum
- MP3文件结构及编解码流程
- CI 系统搭建:Git、Gerrit与Jenkins
- C# SQL数据库学习时遇到到一些异常
- Qt之界面实现技巧
- zzuoj--10399--Turing equation(模拟)
- zzuoj--10399--Turing equation(模拟)
- 【LVS】负载均衡集群
- 有关内部类
- 按概率输出相应的随机数
- 数据源架构模式之活动记录