python自然语言处理工具NLTK各个包的意思和作用总结
2013-11-07 23:40
513 查看
【转】http://www.myexception.cn/perl-python/464414.html
【原】Python NLP实战之一:环境准备
最近正在学习Python,看了几本关于Python入门的书和用Python进行自然语言处理的书,如《Python编程实践》、《Python基础教程》(第2版)、《Python自然语言处理》(影印版)。因为以前是学Java的,有着良好的面向对象的思维方式,所以刚开始看Python的语法,觉得Pyhon太随意了,很别扭,有不正规之嫌。而且,Python自己也正在向面向对象(OO)靠拢。但是,后来看到Python有强大的类库,尤其在自然语言处理方面有着强大的NLTK支持,我逐渐改变了对它的看法。不得不承认,Python非常简洁和清晰,很容易上手,对于有编程经验的人来说,可以快速编写程序来实现某个应用。下面是本人学习中的一些心得,与大家分享。
Python NLP实战之一:环境准备
要下载和安装的软件和资源有:
Python
PyYAML
NLTK
NLTK-Data
NumPy
Matplotlib
(一)下载地址和版本:
Python:http://www.python.org/getit/releases/2.7.2/ 版本:Python 2.7.2 (注:现在是2.7.3。Python已经发布3.3版了,之所以下载2.7,是因为2.x比较稳定,兼容的第三方软件多。Python官网提示:如果你不知道用哪个版本的话,就从2.7开始吧!)
PyYAML:http://pypi.python.org/pypi/PyYAML/ 版本:PyYAML 3.10 功能:YAML的解析工具
NLTK: http://www.nltk.org 版本:nltk-2.0.1 功能:自然语言工具包
NumPy: http://pypi.python.org/pypi/numpy 版本:numpy 1.6.1 功能:支持多维数组和线性代数
Matplotlib: http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.1.0/ 版本:matplotlib-1.1.0 功能:用于数据可视化的二维图库
安装都很简单,我是在Window下安装的。
(二)运行Python IDLE
Python安装完成后,运行Python集成开发环境IDLE:开始->所有程序->Python 2.7 ->IDLE (Python GUI),打开一个新的窗口,显示如下信息,表明安装成功。
(三)下载NLTK数据包
接下来,导入NLTK工具包,然后,下载NLTK数据源。
注意:在导入MLTK工具包时,如果显示如下信息,表明没有安装PyYAML。
按照(一)所列的地址下载、安装完PyYAML后,再打开Python IDLE,导入NLTK,执行nltk.download(),我的界面出现的是文字提示,书上和网上有同学说是图形界面,两者都可以吧。
选择d) Download,敲入d,再敲入l,然后按提示敲几次回车,显示的是将要下载的各种不同的数据包。
你可以选择敲入 all-corpora,或all,或book,我选的是all。保持网络畅通,下载可能需要一段时间。显示信息如下:
选择q,退出下载。
注1:在这个过程中,可以获取帮助,用h) Help,敲入:h。显示如下:
注2:下载之前,可以选择下载到本地的路径,选择c) Config,敲入:c,进入Config环境。显示默认的下载路径,如下:
选择d) Set Data Dir,敲入:d,键入新的下载路径:
如果不改变下载路径,直接退出设置环境。
返回到主菜单:
退出下载环境:
注3:如果上述不成功的话,你可以直接到 http://nltk.googlecode.com/svn/trunk/nltk_data/index.xml 去下载数据包,放到下载路径的目录下即可。
(四)测试NLTK数据包
导入nltk.book包中所有的东西:
显示如下,表明NLTK数据成功装载。
(五)开始NLP工作
运行《Python自然语言处理》(影印版)中的例子,检索含“monstrous”的句子,查询词居中显示:
再看几个例子。
查看语料库中的文本信息,直接敲它的名字:
查看与检索词类似的词语:
查看词语的分散度图:
注意到这里出错了,是因为找不到画图的工具包。按照提示,从(一)中所列的网站上下载、安装Matplotlib即可。我安装了NumPy和Matplotlib。显示如下图:
文本生成的例子:
查看文本中的词例(token)数:
查看文本中的词型(type)数,并按字母升序列出:
好了,准备工作已经做完了,以后就可以开始进行各种各样的NLP工作了,特别是对于中文(汉语)的自然语言处理工作,《Python自然语言处理》(影印版)并没有给出特别的篇幅。看来,这些工作还得我们自己想办法解决啊。
【原】Python NLP实战之一:环境准备
最近正在学习Python,看了几本关于Python入门的书和用Python进行自然语言处理的书,如《Python编程实践》、《Python基础教程》(第2版)、《Python自然语言处理》(影印版)。因为以前是学Java的,有着良好的面向对象的思维方式,所以刚开始看Python的语法,觉得Pyhon太随意了,很别扭,有不正规之嫌。而且,Python自己也正在向面向对象(OO)靠拢。但是,后来看到Python有强大的类库,尤其在自然语言处理方面有着强大的NLTK支持,我逐渐改变了对它的看法。不得不承认,Python非常简洁和清晰,很容易上手,对于有编程经验的人来说,可以快速编写程序来实现某个应用。下面是本人学习中的一些心得,与大家分享。
Python NLP实战之一:环境准备
要下载和安装的软件和资源有:
Python
PyYAML
NLTK
NLTK-Data
NumPy
Matplotlib
(一)下载地址和版本:
Python:http://www.python.org/getit/releases/2.7.2/ 版本:Python 2.7.2 (注:现在是2.7.3。Python已经发布3.3版了,之所以下载2.7,是因为2.x比较稳定,兼容的第三方软件多。Python官网提示:如果你不知道用哪个版本的话,就从2.7开始吧!)
PyYAML:http://pypi.python.org/pypi/PyYAML/ 版本:PyYAML 3.10 功能:YAML的解析工具
NLTK: http://www.nltk.org 版本:nltk-2.0.1 功能:自然语言工具包
NumPy: http://pypi.python.org/pypi/numpy 版本:numpy 1.6.1 功能:支持多维数组和线性代数
Matplotlib: http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.1.0/ 版本:matplotlib-1.1.0 功能:用于数据可视化的二维图库
安装都很简单,我是在Window下安装的。
(二)运行Python IDLE
Python安装完成后,运行Python集成开发环境IDLE:开始->所有程序->Python 2.7 ->IDLE (Python GUI),打开一个新的窗口,显示如下信息,表明安装成功。
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32 Type "copyright", "credits" or "license()" for more information. >>>
(三)下载NLTK数据包
接下来,导入NLTK工具包,然后,下载NLTK数据源。
>>> import nltk >>> nltk.download()
注意:在导入MLTK工具包时,如果显示如下信息,表明没有安装PyYAML。
>>> import nltk Traceback (most recent call last): File "<pyshell#0>", line 1, in <module> import nltk File "C:\Python27\lib\site-packages\nltk\__init__.py", line 107, in <module> from yamltags import * File "C:\Python27\lib\site-packages\nltk\yamltags.py", line 10, in <module> import yaml ImportError: No module named yaml
按照(一)所列的地址下载、安装完PyYAML后,再打开Python IDLE,导入NLTK,执行nltk.download(),我的界面出现的是文字提示,书上和网上有同学说是图形界面,两者都可以吧。
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32 Type "copyright", "credits" or "license()" for more information. >>> import nltk >>> nltk.download()
NLTK Downloader
---------------------------------------
d) Download l) List u) Update c) Config h) Help q) Quit
---------------------------------------
Downloader>
选择d) Download,敲入d,再敲入l,然后按提示敲几次回车,显示的是将要下载的各种不同的数据包。
Downloader> d Download which package (l=list; x=cancel)? Identifier> l Packages: [ ] maxent_ne_chunker... ACE Named Entity Chunker (Maximum entropy) [ ] abc................. Australian Broadcasting Commission 2006 [ ] alpino.............. Alpino Dutch Treebank [ ] biocreative_ppi..... BioCreAtIvE (Critical Assessment of Information Extraction Systems in Biology) [ ] brown_tei........... Brown Corpus (TEI XML Version) [ ] cess_esp............ CESS-ESP Treebank [ ] chat80.............. Chat-80 Data Files [ ] brown............... Brown Corpus [ ] cmudict............. The Carnegie Mellon Pronouncing Dictionary (0.6) [ ] city_database....... City Database [ ] cess_cat............ CESS-CAT Treebank [ ] comtrans............ ComTrans Corpus Sample [ ] conll2002........... CONLL 2002 Named Entity Recognition Corpus [ ] conll2007........... Dependency Treebanks from CoNLL 2007 (Catalan and Basque Subset) [ ] europarl_raw........ Sample European Parliament Proceedings Parallel Corpus [ ] dependency_treebank. Dependency Parsed Treebank [ ] conll2000........... CONLL 2000 Chunking Corpus Hit Enter to continue: [ ] floresta............ Portuguese Treebank [ ] names............... Names Corpus, Version 1.3 (1994-03-29) [ ] gazetteers.......... Gazeteer Lists [ ] genesis............. Genesis Corpus [ ] gutenberg........... Project Gutenberg Selections [ ] inaugural........... C-Span Inaugural Address Corpus [ ] jeita............... JEITA Public Morphologically Tagged Corpus (in ChaSen format) [ ] movie_reviews....... Sentiment Polarity Dataset Version 2.0 [ ] ieer................ NIST IE-ER DATA SAMPLE [ ] nombank.1.0......... NomBank Corpus 1.0 [ ] indian.............. Indian Language POS-Tagged Corpus [ ] paradigms........... Paradigm Corpus [ ] kimmo............... PC-KIMMO Data Files [ ] knbc................ KNB Corpus (Annotated blog corpus) [ ] langid.............. Language Id Corpus [ ] mac_morpho.......... MAC-MORPHO: Brazilian Portuguese news text with part-of-speech tags [ ] machado............. Machado de Assis -- Obra Completa [ ] pe08................ Cross-Framework and Cross-Domain Parser Evaluation Shared Task Hit Enter to continue: [ ] pl196x.............. Polish language of the XX century sixties [ ] pil................. The Patient Information Leaflet (PIL) Corpus [ ] nps_chat............ NPS Chat [ ] reuters............. The Reuters-21578 benchmark corpus, ApteMod version [ ] qc.................. Experimental Data for Question Classification [ ] rte................. PASCAL RTE Challenges 1, 2, and 3 [ ] ppattach............ Prepositional Phrase Attachment Corpus [ ] propbank............ Proposition Bank Corpus 1.0 [ ] problem_reports..... Problem Report Corpus [ ] sinica_treebank..... Sinica Treebank Corpus Sample [ ] verbnet............. VerbNet Lexicon, Version 2.1 [ ] state_union......... C-Span State of the Union Address Corpus [ ] semcor.............. SemCor 3.0 [ ] senseval............ SENSEVAL 2 Corpus: Sense Tagged Text [ ] smultron............ SMULTRON Corpus Sample [ ] shakespeare......... Shakespeare XML Corpus Sample [ ] stopwords........... Stopwords Corpus [ ] swadesh............. Swadesh Wordlists [ ] switchboard......... Switchboard Corpus Sample [ ] toolbox............. Toolbox Sample Files Hit Enter to continue: [ ] unicode_samples..... Unicode Samples [ ] webtext............. Web Text Corpus [ ] timit............... TIMIT Corpus Sample [ ] ycoe................ York-Toronto-Helsinki Parsed Corpus of Old English Prose [ ] treebank............ Penn Treebank Sample [ ] udhr................ Universal Declaration of Human Rights Corpus [ ] sample_grammars..... Sample Grammars [ ] book_grammars....... Grammars from NLTK Book [ ] spanish_grammars.... Grammars for Spanish [ ] wordnet............. WordNet [ ] wordnet_ic.......... WordNet-InfoContent [ ] words............... Word Lists [ ] tagsets............. Help on Tagsets [ ] basque_grammars..... Grammars for Basque [ ] large_grammars...... Large context-free and feature-based grammars for parser comparison [ ] maxent_treebank_pos_tagger Treebank Part of Speech Tagger (Maximum entropy) [ ] rslp................ RSLP Stemmer (Removedor de Sufixos da Lingua Portuguesa) [ ] hmm_treebank_pos_tagger Treebank Part of Speech Tagger (HMM) Hit Enter to continue: [ ] punkt............... Punkt Tokenizer Models Collections: [ ] all-corpora......... All the corpora [ ] all................. All packages [ ] book................ Everything used in the NLTK Book ([*] marks installed packages)
你可以选择敲入 all-corpora,或all,或book,我选的是all。保持网络畅通,下载可能需要一段时间。显示信息如下:
Download which package (l=list; x=cancel)? Identifier> all Downloading collection 'all' | | Downloading package 'abc' to C:\Documents and | Settings\lenovo\Application Data\nltk_data... | Unzipping corpora\abc.zip. | Downloading package 'alpino' to C:\Documents and | Settings\lenovo\Application Data\nltk_data... | Unzipping corpora\alpino.zip. | Downloading package 'biocreative_ppi' to C:\Documents and | Settings\lenovo\Application Data\nltk_data... | Unzipping corpora\biocreative_ppi.zip. | Downloading package 'brown' to C:\Documents and | Settings\lenovo\Application Data\nltk_data... | Unzipping corpora\brown.zip. | Downloading package 'brown_tei' to C:\Documents and | Settings\lenovo\Application Data\nltk_data... | Unzipping corpora\brown_tei.zip. | Downloading package 'cess_cat' to C:\Documents and | Settings\lenovo\Application Data\nltk_data... | Unzipping corpora\cess_cat.zip. | Downloading package 'cess_esp' to C:\Documents and | Settings\lenovo\Application Data\nltk_data... | Unzipping corpora\cess_esp.zip. | Downloading package 'chat80' to C:\Documents and ... ... ... | Downloading package 'book_grammars' to C:\Documents and | Settings\lenovo\Application Data\nltk_data... | Unzipping grammars\book_grammars.zip. | Downloading package 'sample_grammars' to C:\Documents and | Settings\lenovo\Application Data\nltk_data... | Unzipping grammars\sample_grammars.zip. | Downloading package 'spanish_grammars' to C:\Documents and | Settings\lenovo\Application Data\nltk_data... | Unzipping grammars\spanish_grammars.zip. | Downloading package 'basque_grammars' to C:\Documents and | Settings\lenovo\Application Data\nltk_data... | Unzipping grammars\basque_grammars.zip. | Downloading package 'large_grammars' to C:\Documents and | Settings\lenovo\Application Data\nltk_data... | Unzipping grammars\large_grammars.zip. | Downloading package 'tagsets' to C:\Documents and | Settings\lenovo\Application Data\nltk_data... | Done downloading collection 'all' --------------------------------------- d) Download l) List u) Update c) Config h) Help q) Quit --------------------------------------- Downloader>
选择q,退出下载。
Downloader> q True
注1:在这个过程中,可以获取帮助,用h) Help,敲入:h。显示如下:
Downloader> h Commands: d) Download a package or collection u) Update out of date packages l) List packages & collections h) Help c) View & Modify Configuration q) Quit --------------------------------------- d) Download l) List u) Update c) Config h) Help q) Quit --------------------------------------- Downloader>
注2:下载之前,可以选择下载到本地的路径,选择c) Config,敲入:c,进入Config环境。显示默认的下载路径,如下:
Downloader> c Data Server: - URL: <http://nltk.googlecode.com/svn/trunk/nltk_data/index.xml> - 3 Package Collections Available - 74 Individual Packages Available Local Machine: - Data directory: C:\Documents and Settings\lenovo\Application Data\nltk_data --------------------------------------- s) Show Config u) Set Server URL d) Set Data Dir m) Main Menu --------------------------------------- Config>
选择d) Set Data Dir,敲入:d,键入新的下载路径:
Config> d New Directory> D:\nltk_data --------------------------------------- s) Show Config u) Set Server URL d) Set Data Dir m) Main Menu --------------------------------------- Config>
如果不改变下载路径,直接退出设置环境。
New Directory> q Cancelled! --------------------------------------- s) Show Config u) Set Server URL d) Set Data Dir m) Main Menu --------------------------------------- Config>
返回到主菜单:
Config> m --------------------------------------- d) Download l) List u) Update c) Config h) Help q) Quit --------------------------------------- Downloader>
退出下载环境:
Downloader> q True
注3:如果上述不成功的话,你可以直接到 http://nltk.googlecode.com/svn/trunk/nltk_data/index.xml 去下载数据包,放到下载路径的目录下即可。
(四)测试NLTK数据包
导入nltk.book包中所有的东西:
>>> from nltk.book import *
显示如下,表明NLTK数据成功装载。
>>> from nltk.book import * *** Introductory Examples for the NLTK Book *** Loading text1, ..., text9 and sent1, ..., sent9 Type the name of the text or sentence to view it. Type: 'texts()' or 'sents()' to list the materials. text1: Moby Dick by Herman Melville 1851 text2: Sense and Sensibility by Jane Austen 1811 text3: The Book of Genesis text4: Inaugural Address Corpus text5: Chat Corpus text6: Monty Python and the Holy Grail text7: Wall Street Journal text8: Personals Corpus text9: The Man Who Was Thursday by G . K . Chesterton 1908 >>>
(五)开始NLP工作
运行《Python自然语言处理》(影印版)中的例子,检索含“monstrous”的句子,查询词居中显示:
>>> text1.concordance('monstrous') Building index... Displaying 11 of 11 matches: ong the former , one was of a most monstrous size . ... This came towards us , ON OF THE PSALMS . " Touching that monstrous bulk of the whale or ork we have r ll over with a heathenish array of monstrous clubs and spears . Some were thick d as you gazed , and wondered what monstrous cannibal and savage could ever hav that has survived the flood ; most monstrous and most mountainous ! That Himmal they might scout at Moby Dick as a monstrous fable , or still worse and more de th of Radney .'" CHAPTER 55 Of the Monstrous Pictures of Whales . I shall ere l ing Scenes . In connexion with the monstrous pictures of whales , I am strongly ere to enter upon those still more monstrous stories of them which are to be fo ght have been rummaged out of this monstrous cabinet there is no telling . But of Whale - Bones ; for Whales of a monstrous size are oftentimes cast up dead u >>>
再看几个例子。
查看语料库中的文本信息,直接敲它的名字:
>>> text1 <Text: Moby Dick by Herman Melville 1851>
>>> text2 <Text: Sense and Sensibility by Jane Austen 1811>
查看与检索词类似的词语:
>>> text1.similar('monstrous') Building word-context index... abundant candid careful christian contemptible curious delightfully determined doleful domineering exasperate fearless few gamesome horrible impalpable imperial lamentable lazy loving
>>> text2.similar('monstrous') Building word-context index... very exceedingly heartily so a amazingly as extremely good great remarkably sweet vast
查看词语的分散度图:
>>> text4.dispersion_plot(['citizens','democracy','freedom','duties','America']) Traceback (most recent call last): File "<pyshell#12>", line 1, in <module> text4.dispersion_plot(['citizens','democracy','freedom','duties','America']) File "C:\Python27\lib\site-packages\nltk\text.py", line 454, in dispersion_plot dispersion_plot(self, words) File "C:\Python27\lib\site-packages\nltk\draw\dispersion.py", line 25, in dispersion_plot raise ValueError('The plot function requires the matplotlib package (aka pylab).' ValueError: The plot function requires the matplotlib package (aka pylab).See http://matplotlib.sourceforge.net/ >>>
注意到这里出错了,是因为找不到画图的工具包。按照提示,从(一)中所列的网站上下载、安装Matplotlib即可。我安装了NumPy和Matplotlib。显示如下图:
文本生成的例子:
>>> text3.generate() Building ngram index... In the six hundredth year of Noah , and Epher , and I put the stone from the field , And Ophir , and laid him on the morrow , that thou dost overtake them , and herb yielding seed after him that curseth thee , of a tree yielding fruit after his kind , cattle , and spread his tent , and Abimael , and Lot went out . The LORD God had taken from our father is in the inn , he gathered up his hand . And say ye moreover , Behold , I know that my
查看文本中的词例(token)数:
>>> len(text3) 44764
查看文本中的词型(type)数,并按字母升序列出:
>>> sorted(set(text3)) ['!', "'", '(', ')', ',', ',)', '.', '.)', ':', ';', ';)', '?', '?)', 'A', 'Abel', 'Abelmizraim', 'Abidah', 'Abide', 'Abimael', 'Abimelech', 'Abr', 'Abrah', 'Abraham', 'Abram', 'Accad', 'Achbor', 'Adah', 'Adam', 'Adbeel', 'Admah', 'Adullamite', 'After', 'Aholibamah', 'Ahuzzath', 'Ajah', 'Akan', 'All', 'Allonbachuth', 'Almighty', 'Almodad', 'Also', 'Alvah', 'Alvan', 'Am', 'Amal', 'Amalek', 'Amalekites', 'Ammon', 'Amorite',...]
好了,准备工作已经做完了,以后就可以开始进行各种各样的NLP工作了,特别是对于中文(汉语)的自然语言处理工作,《Python自然语言处理》(影印版)并没有给出特别的篇幅。看来,这些工作还得我们自己想办法解决啊。
相关文章推荐
- 【Python】基于PyUnit的自动化单元测试框架,PyUnitGUI
- Hello Python! - 记本人首个Python小程序
- Python Cookbook学习记录 ch6_3_2013/11/7
- Python Cookbook学习记录 ch6_2_2013/11/7
- python 中 input 和 raw_input 的区别
- python 字符串格式化转换类型
- python中__get__,__getattr__,__getattribute__的区别
- Python学习记录----语法学习
- Python学习记录----类型匹配(转)
- python基础学习笔记5--对象
- 【python】 读写文件
- 【python】 字符串转小写(含汉字等时仍work)
- 【python】 编码转换
- Python自动化备份系统及网站
- Python自动化备份系统及网站
- python 2.4 不支持 try-except-finally
- Python:数组、队列及堆栈的使用(list用法)--转
- [Python] modify *.c -> *.java script
- python excel handle
- Python strip lstrip rstrip使用方法