How to speed up insertion performance in PostgreSQL
2017-02-16 19:26
375 查看
Disable any triggers on the table
Drop indexes before starting the import, re-create them afterwards. (It takes much less time to build an index in one pass than it does to add the same data to it progressively, and the resulting index is much more compact).
Remove Foreign Key Constraints
If doing the import within a single transaction, it's safe to drop foreign key constraints, do the import, and re-create the constraints before committing. Do not do this if the import is split across multiple transactions as you might introduce invalid data.
If possible, use
If you can't use
Batch your inserts into explicit transactions, doing hundreds of thousands or millions of inserts per transaction. There's no practical limit AFAIK, but batching will let you recover from an error by marking the start of each batch in your input data.
Increase maintenance_work_mem:This will help to speed up CREATE INDEX commands and ALTER TABLE ADD FOREIGN KEY commands. It won't do much for COPY itself, so this advice is only useful when you are using one or both of the above techniques.
Use
Set a high
If and only if you don't mind losing your entire PostgreSQL cluster (your database and any others on the same cluster) to catastrophic corruption if the system crashes during the import, you can stop Pg, set
Run ANALYZE Afterwards
参考:
http://stackoverflow.com/questions/12206600/how-to-speed-up-insertion-performance-in-postgresql https://www.postgresql.org/docs/9.4/static/populate.html
Drop indexes before starting the import, re-create them afterwards. (It takes much less time to build an index in one pass than it does to add the same data to it progressively, and the resulting index is much more compact).
Change table to UNLOGGEDtable without indexes, then change it to logged and add the indexes.Unfortunately in PostgreSQL 9.4 there's no support for changing tables from
UNLOGGEDto logged. 9.5 adds
ALTER TABLE ... SET LOGGEDto permit you to do this.
Remove Foreign Key Constraints
If doing the import within a single transaction, it's safe to drop foreign key constraints, do the import, and re-create the constraints before committing. Do not do this if the import is split across multiple transactions as you might introduce invalid data.
If possible, use
COPYinstead of
INSERTs
If you can't use
COPYconsider using multi-valued
INSERTs if practical. Don't try to list too many values in a single
VALUESthough; those values have to fit in memory a couple of times over, so keep it to a few hundred per statement.
Batch your inserts into explicit transactions, doing hundreds of thousands or millions of inserts per transaction. There's no practical limit AFAIK, but batching will let you recover from an error by marking the start of each batch in your input data.
Increase maintenance_work_mem:This will help to speed up CREATE INDEX commands and ALTER TABLE ADD FOREIGN KEY commands. It won't do much for COPY itself, so this advice is only useful when you are using one or both of the above techniques.
Use
synchronous_commit=offand a huge
commit_delayto reduce fsync() costs. This won't help much if you've batched your work into big transactions, though.
INSERTor
COPYin parallel from several connections. How many depends on your hardware's disk subsystem; as a rule of thumb, you want one connection per physical hard drive if using direct attached storage.
Set a high
checkpoint_segmentsvalue and enable
log_checkpoints. Look at the PostgreSQL logs and make sure it's not complaining about checkpoints occurring too frequently.
If and only if you don't mind losing your entire PostgreSQL cluster (your database and any others on the same cluster) to catastrophic corruption if the system crashes during the import, you can stop Pg, set
fsync=off, start Pg, do your import, then (vitally) stop Pg and set
fsync=onagain. See WAL configuration. Do not do this if there is already any data you care about in any database on your PostgreSQL install. If you set
fsync=offyou can also set
full_page_writes=off; again, just remember to turn it back on after your import to prevent database corruption and data loss. See non-durable settings in the Pg manual.
Run ANALYZE Afterwards
参考:
http://stackoverflow.com/questions/12206600/how-to-speed-up-insertion-performance-in-postgresql https://www.postgresql.org/docs/9.4/static/populate.html
相关文章推荐
- How to speed up Remote Desktop Connection in Win7
- How to speed up Remote Desktop Connection in Win7
- How to Set Up SQL Server 2008 FileStream In the Cluster
- How to write the fast code/ high performance in C#
- How to cleanup orphaned Data Pump jobs in DBA_DATAPUMP_JOBS ?
- How to Set Up SSH Keys In Ubuntu
- How to attach to other process in VSTS performance profiler
- How To Automate Cleanup Of Dead Connections And INACTIVE 【如何自动清理ORACLE中的死连接和非活动会话】
- How to speed up and smooth over any load times on iPhone (preloading, animated loading screens etc)
- How to reduce CPU soft lock up in KVM guest
- How to avoid the "enumeral mismatch" error in eikmenup.h
- How To Automate Cleanup Of Dead Connections And INACTIVE Sessions [ID 206007.1]
- How to create a user in PostgreSQL
- How to use pop-up TextBox in Java ME
- How to monitor Web server performance by using counter logs in System Monitor in IIS
- How to choose the right data structure in different scenarios to bring out best performance?
- How to Hook Up a VS.NET 2005 Solution With CruiseControl.NET in a Few Minutes
- How to change Ethernet network card speed and duplex settings in ubuntu
- How to Set Up Your Own Home Seedbox (in Windows)
- How to speed up the MV refresh