关于java爬虫与python爬虫

前言

很多人说学习数据挖掘，先从爬虫入手。接触了大大小小的项目后，发现数据的获取是数据建模前的一项非常重要的活儿。在此，我需要先总结一些爬虫的流程，分别有python版的以及java版的。

url请求

java版的代码如下：

public String call (String url){
String content = "";
BufferedReader in = null;
try{
URL realUrl = new URL(url);
URLConnection connection = realUrl.openConnection();
connection.connect();
in = new BufferedReader(new InputStreamReader(connection.getInputStream(),"gbk"));
String line ;
while ((line = in.readLine()) != null){
content += line + "\n";
}
}catch (Exception e){
e.printStackTrace();
}
finally{
try{
if (in != null){
in.close();
}
}catch(Exception e2){
e2.printStackTrace();
}
}
return content;
}

python版的代码如下：

# coding=utf-8
import chardet
import urllib2

url = "http://www.baidu.com"
data = (urllib2.urlopen(url)).read()
charset = chardet.detect(data)
code = charset['encoding']
content = str(data).decode(code, 'ignore').encode('utf8')
print content

正则表达式

java版的代码如下：

public String call(String content) throws Exception {
Pattern p = Pattern.compile("content\":\".*?\"");
Matcher match = p.matcher(content);
StringBuilder sb = new StringBuilder();
String tmp;
while (match.find()){
tmp = match.group();
tmp = tmp.replaceAll("\"", "");
tmp = tmp.replace("content:", "");
tmp = tmp.replaceAll("<.*>", "");
sb.append(tmp + "\n");
}
String comment = sb.toString();
return comment;
}
}

python的代码如下：

import re
pattern = re.compile(正则)
group = pattern.findall(字符串)