您的位置:首页 > 编程语言 > Python开发

将爬取到的数据(用Python)写入PostgreSQL数据库

2016-03-08 12:29 696 查看
我想砍人。

这么简单,我特么搞了整整一天。从昨天下午到今天11:40.

WTF!

以下是代码(目的想把爬取到的数据存入PostgreSQL):

class PostgreSQLPipeline(object):
def process_item(self,item, spider)
#sql_desc="INSERT INTO postgresql_1(fullname, username, organization, mail, joined,followers,starred,following,popular_repos,popular_repos_download,popular_repos_star,popular_repos_info,home_page)values(item['fullname'], item['username'], item['organization'], item['mail'],item['joined'],item['followers'],item['starred'],item['following'],item['popular_repos'],item['popular_repos_download'],item['popular_repos_star'],item['popular_repos_info'], item['home_page'])"
conn = psycopg2.connect(database="mypg", user="postgres", password="student", host="127.0.0.1", port="5432")
try:
cur=conn.cursor()
#self.conn.query(sql_desc)
#cur.execute("INSERT INTO ewrrw values(dict(item));")
cur.execute("""INSERT INTO postgresql_1
(fullname, username, organization, mail, joined, followers, starred, following, popular_repos, popular_repos_download, popular_repos_star, popular_repos_info, home_page)
VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s);""",
(item['fullname'],
item['username'],
item['organization'],
item['mail'],
item['joined'],
item['followers'],
item['starred'],
item['following'],
item['popular_repos'],
item['popular_repos_download'],
item['popular_repos_star'],
item['popular_repos_info'],
item['home_page']),)

conn.commit()
log.msg("Data added to PostgreSQL database!",
level=log.DEBUG,spider=spider)

except Exception,e:
print 'insert record into table failed'
print e

finally:
if cur:
cur.close()
conn.close()
return item


为什么搞了一天,。。。我哭啊

心痛,因为蠢呗。

以下7个错误 一直在出现,测试了近70次!!!才成功的将爬取到的数据存入PostgreSQL(前提先在数据库mypg里创建postgresql_1表):

1.insert record into table failed...

2.LINE 2: INSERT INTO postgresql_1(fullname, username,...

3.ProgrammingError: syntax error at or near "fullname"...

4.value too long for type character varying(100)...

5.psycopg2.InterfaceError: connection already closed...

6.psycopg2.InternalError: current transaction is aborted, commands ignored until end of transaction block...

7.psycopg2.ProgrammingError: column "item" does not exist
LINE 3:                 values(item['fullname'], item['username'], i...


写这个是因为我看到这些错误我快疯了,还好我有时间 慢慢的试啊试 试啊试 终于OK了。

内容来自用户分享和网络整理,不保证内容的准确性,如有侵权内容,可联系管理员处理 点击这里给我发消息