python Beautiful soup网页解析-星座网
2017-11-01 15:00
309 查看
beautiful-soup是用来解析网页的利器。
我们先打开一个网页
code=200表示访问成功
from urllib.request import urlopen
url='http://www.xzw.com/astro/virgo/?appid=bds'
response=urlopen(url)
response.getcode()
content=response.read()
接下来,我们可以用beautiful-soup来解析这个网页,生成一个beautiful-soup对象
bs_obj=BeautifulSoup(urlopen(url), 'html.parser',from_encoding='utf-8')
然后可以用bs对象的find方法来查找节点,我们通过传入节点的标签和属性作为参数,返回节点的list
bs_obj.find_all('a',{'target':'_blank','hidefocus':'true'})
[<a class="icon0" hidefocus="true" href="/astro/aries/" target="_blank" title="白羊座"></a>,
<a class="icon1" hidefocus="true" href="/astro/taurus/" target="_blank" title="金牛座"></a>,
<a class="icon2" hidefocus="true" href="/astro/gemini/" target="_blank" title="双子座"></a>,
<a class="icon3" hidefocus="true" href="/astro/cancer/" target="_blank" title="巨蟹座"></a>,
<a class="icon4" hidefocus="true" href="/astro/leo/" target="_blank" title="狮子座"></a>,
<a class="icon5" hidefocus="true" href="/astro/virgo/" target="_blank" title="处女座"></a>,
<a class="icon6" hidefocus="true" href="/astro/libra/" target="_blank" title="天秤座"></a>,
<a class="icon7" hidefocus="true" href="/astro/scorpio/" target="_blank" title="天蝎座"></a>,
<a class="icon8" hidefocus="true" href="/astro/sagittarius/" target="_blank" title="射手座"></a>,
<a class="icon9" hidefocus="true" href="/astro/capricorn/" target="_blank" title="摩羯座"></a>,
<a class="icon10" hidefocus="true" href="/astro/aquarius/" target="_blank" title="水瓶座"></a>,
<a class="icon11" hidefocus="true" href="/astro/pisces/" target="_blank" title="双鱼座"></a>]
对于每个节点,我们可以获取它的名称,属性,文字
a /astro/aries/ 白羊座
a /astro/taurus/ 金牛座
a /astro/gemini/ 双子座
a /astro/cancer/ 巨蟹座
a /astro/leo/ 狮子座
a /astro/virgo/ 处女座
a /astro/libra/ 天秤座
a /astro/scorpio/ 天蝎座
a /astro/sagittarius/ 射手座
a /astro/capricorn/ 摩羯座
a /astro/aquarius/ 水瓶座
a /astro/pisces/ 双鱼座
Children 节点
如果我们想拿到div标签后面的所有a标签,可以用node.children
for child in bs_obj.find('div', {'class':'a-nav'}).children:
print(child)
<a class="icon0" hidefocus="true" href="/astro/aries/" target="_blank" title="白羊座"></a>
<a class="icon1" hidefocus="true" href="/astro/taurus/" target="_blank" title="金牛座"></a>
<a class="icon2" hidefocus="true" href="/astro/gemini/" target="_blank" title="双子座"></a>
<a class="icon3" hidefocus="true" href="/astro/cancer/" target="_blank" title="巨蟹座"></a>
<a class="icon4" hidefocus="true" href="/astro/leo/" target="_blank" title="狮子座"></a>
<a class="icon5" hidefocus="true" href="/astro/virgo/" target="_blank" title="处女座"></a>
<a class="icon6" hidefocus="true" href="/astro/libra/" target="_blank" title="天秤座"></a>
<a class="icon7" hidefocus="true" href="/astro/scorpio/" target="_blank" title="天蝎座"></a>
<a class="icon8" hidefocus="true" href="/astro/sagittarius/" target="_blank" title="射手座"></a>
<a class="icon9" hidefocus="true" href="/astro/capricorn/" target="_blank" title="摩羯座"></a>
<a class="icon10" hidefocus="true" href="/astro/aquarius/" target="_blank" title="水瓶座"></a>
<a class="icon11" hidefocus="true" href="/astro/pisces/" target="_blank" title="双鱼座"></a>
<em class="todayastro"></em>
更多的还有find_parent, find_next_sibling等,用法都差不多,可以查看文档
我们先打开一个网页
code=200表示访问成功
from urllib.request import urlopen
url='http://www.xzw.com/astro/virgo/?appid=bds'
response=urlopen(url)
response.getcode()
content=response.read()
接下来,我们可以用beautiful-soup来解析这个网页,生成一个beautiful-soup对象
bs_obj=BeautifulSoup(urlopen(url), 'html.parser',from_encoding='utf-8')
bs_obj.title<title>处女座2018年运势,处女座男生女生,处女座男生女生最配的星座,处女座的男人,Virgo-星座是按阴历还是阳历,十二星座速配查询 - 星座屋</title>
然后可以用bs对象的find方法来查找节点,我们通过传入节点的标签和属性作为参数,返回节点的list
bs_obj.find_all('a',{'target':'_blank','hidefocus':'true'})
[<a class="icon0" hidefocus="true" href="/astro/aries/" target="_blank" title="白羊座"></a>,
<a class="icon1" hidefocus="true" href="/astro/taurus/" target="_blank" title="金牛座"></a>,
<a class="icon2" hidefocus="true" href="/astro/gemini/" target="_blank" title="双子座"></a>,
<a class="icon3" hidefocus="true" href="/astro/cancer/" target="_blank" title="巨蟹座"></a>,
<a class="icon4" hidefocus="true" href="/astro/leo/" target="_blank" title="狮子座"></a>,
<a class="icon5" hidefocus="true" href="/astro/virgo/" target="_blank" title="处女座"></a>,
<a class="icon6" hidefocus="true" href="/astro/libra/" target="_blank" title="天秤座"></a>,
<a class="icon7" hidefocus="true" href="/astro/scorpio/" target="_blank" title="天蝎座"></a>,
<a class="icon8" hidefocus="true" href="/astro/sagittarius/" target="_blank" title="射手座"></a>,
<a class="icon9" hidefocus="true" href="/astro/capricorn/" target="_blank" title="摩羯座"></a>,
<a class="icon10" hidefocus="true" href="/astro/aquarius/" target="_blank" title="水瓶座"></a>,
<a class="icon11" hidefocus="true" href="/astro/pisces/" target="_blank" title="双鱼座"></a>]
对于每个节点,我们可以获取它的名称,属性,文字
Node.name | 节点的名称 |
Node['href'] | 节点的属性 |
Node.get_text() | 节点的文字 |
node_list = bs_obj.find_all('a',{'target':'_blank','hidefocus':'true'}) for node in node_list: print(node.name,node['href'],node['title'],node.get_text())
a /astro/aries/ 白羊座
a /astro/taurus/ 金牛座
a /astro/gemini/ 双子座
a /astro/cancer/ 巨蟹座
a /astro/leo/ 狮子座
a /astro/virgo/ 处女座
a /astro/libra/ 天秤座
a /astro/scorpio/ 天蝎座
a /astro/sagittarius/ 射手座
a /astro/capricorn/ 摩羯座
a /astro/aquarius/ 水瓶座
a /astro/pisces/ 双鱼座
Children 节点
如果我们想拿到div标签后面的所有a标签,可以用node.children
for child in bs_obj.find('div', {'class':'a-nav'}).children:
print(child)
<a class="icon0" hidefocus="true" href="/astro/aries/" target="_blank" title="白羊座"></a>
<a class="icon1" hidefocus="true" href="/astro/taurus/" target="_blank" title="金牛座"></a>
<a class="icon2" hidefocus="true" href="/astro/gemini/" target="_blank" title="双子座"></a>
<a class="icon3" hidefocus="true" href="/astro/cancer/" target="_blank" title="巨蟹座"></a>
<a class="icon4" hidefocus="true" href="/astro/leo/" target="_blank" title="狮子座"></a>
<a class="icon5" hidefocus="true" href="/astro/virgo/" target="_blank" title="处女座"></a>
<a class="icon6" hidefocus="true" href="/astro/libra/" target="_blank" title="天秤座"></a>
<a class="icon7" hidefocus="true" href="/astro/scorpio/" target="_blank" title="天蝎座"></a>
<a class="icon8" hidefocus="true" href="/astro/sagittarius/" target="_blank" title="射手座"></a>
<a class="icon9" hidefocus="true" href="/astro/capricorn/" target="_blank" title="摩羯座"></a>
<a class="icon10" hidefocus="true" href="/astro/aquarius/" target="_blank" title="水瓶座"></a>
<a class="icon11" hidefocus="true" href="/astro/pisces/" target="_blank" title="双鱼座"></a>
<em class="todayastro"></em>
更多的还有find_parent, find_next_sibling等,用法都差不多,可以查看文档
相关文章推荐
- python Beautiful Soup 抓取解析网页
- Python通过解析网页实现看报程序的方法
- Python解析本地和web端网页
- Python网页抓取工具Beautiful Soup面面观!
- python爬虫解析网页编码问题
- 利用Python抓取和解析网页
- python爬虫:不同解析网页方式添加报头
- Python网页解析利器BeautifulSoup安装使用介绍
- Python 网页解析和URL下载
- 利用Python抓取和解析网页(一)
- Python中的HTMLParser、cookielib抓取和解析网页、从HTML文档中提取链接、图像、文本、Cookies(二)
- python解析网页时的错误(EOF in middle of construct)
- python网页解析利器——BeautifulSoup
- 用Python做网页抓取与解析入门笔记
- Python实现抓取网页并且解析的实例
- 利用Python抓取和解析网页(1)
- Python抓取和解析网页
- HTMLParser解析网页,提取链接地址、标题名称,并插入数据库 分类: python 小练习 HTMLParser 2014-02-19 09:57 519人阅读 评论(0) 收藏
- Beautiful Soup 示例解析html文件(python)
- 利用Python和Beautiful Soup抓取网页内容