Ideas for creating a friendfeed like feed aggregator system
2009-04-03 17:30
393 查看
I was invited to join a meeting about implement a friendfeed like feed system. Here are some ideas about requirement and architecture, which I typed on my BlackBerry during the meeting.
Like the friendfeed, The product can import external RSS, so we separate the system into two parts. The rss crawl system and the feed pubsub system. The pubsub system has no responsbility to grab the feed from source. The feed itself only save feed summary and url.
We decided to use the INBOX approach, which will push the published feed to be saved in all subscriber’s data table. More information of how this work can refer to Scaling a Microblogging Service – Part I.
User’s homepage is an aggregation result. We choose to return a limited recent real-time date, no infinity pagination. But the user’s own feed(user’s profile page) may have a bigger date range.
The unsubscibe logic have two choice, delete or keep the history data from one’s inbox. We decide to keep them.
If the feed source had been deleted, do we need to delete all references in all subscriber’s inbox? If need delete, each feed push to the pubsub system need to have a unique resource id. Another problem is after the source updated whether to publish a new feed or update the current feed?
How to manage the group(QUN in Chinese) feed, deliver to all member’s inbox? Or share a group inbox?
How to impl the feed comment logic, publish the comment to feed system or design a standalone comment system. We prefer to use a standalone comment system which doesn’t publish the comment back to the feed system.
Every feed has a media type, such as text, video, image so the subscribe API can only retrieve a limited media type (text for mobile device). And a feed may have tags.
The read/unread count is easy to implement. But the load is heavy. (QQ / QQzone may has such logic.)
Need open API for 3rd party client(like twitter client), and RSS feed, may have OAuth integration.
The storage may like friendfeed’s mysql schema (see How FriendFeed uses MySQL to store schema-less data) or use Amazon simpledb.
May add support for XMPP Publish-Subscribe, or PEP(Personal Eventing Protocol) for pushing realtime time to users in the future.
Like the friendfeed, The product can import external RSS, so we separate the system into two parts. The rss crawl system and the feed pubsub system. The pubsub system has no responsbility to grab the feed from source. The feed itself only save feed summary and url.
We decided to use the INBOX approach, which will push the published feed to be saved in all subscriber’s data table. More information of how this work can refer to Scaling a Microblogging Service – Part I.
User’s homepage is an aggregation result. We choose to return a limited recent real-time date, no infinity pagination. But the user’s own feed(user’s profile page) may have a bigger date range.
The unsubscibe logic have two choice, delete or keep the history data from one’s inbox. We decide to keep them.
If the feed source had been deleted, do we need to delete all references in all subscriber’s inbox? If need delete, each feed push to the pubsub system need to have a unique resource id. Another problem is after the source updated whether to publish a new feed or update the current feed?
How to manage the group(QUN in Chinese) feed, deliver to all member’s inbox? Or share a group inbox?
How to impl the feed comment logic, publish the comment to feed system or design a standalone comment system. We prefer to use a standalone comment system which doesn’t publish the comment back to the feed system.
Every feed has a media type, such as text, video, image so the subscribe API can only retrieve a limited media type (text for mobile device). And a feed may have tags.
The read/unread count is easy to implement. But the load is heavy. (QQ / QQzone may has such logic.)
Need open API for 3rd party client(like twitter client), and RSS feed, may have OAuth integration.
The storage may like friendfeed’s mysql schema (see How FriendFeed uses MySQL to store schema-less data) or use Amazon simpledb.
May add support for XMPP Publish-Subscribe, or PEP(Personal Eventing Protocol) for pushing realtime time to users in the future.
相关文章推荐
- 删除svn隐藏目录SHELL命令(for windows and unix-like system)
- 删除svn隐藏目录SHELL命令(for windows and unix-like system)
- WCF Error RANT: An error occured creating the configuration section handler for system.serviceModel/
- Howto create a System.Linq.Expressions.Expression for Like
- What are the best practices for building something like a News Feed?
- Hack Like a Pro: Using Nexpose to Scan for Network & System Vulnerabilities
- what-is-the-best-components-stack-for-building-distributed-log-aggregator-like
- where would you like to install the boot loader for your system?
- 微软企业库中数据库访问代码问题(System.InvalidOperationException: Parameter '@FileContent' exceeds the size limit for the sql_variant datatype.)
- Practical Ruby Projects: Ideas for the Eclectic Programmer
- ORA-00119: invalid specification for system par...
- Pro Visual Studio Team System with Team Edition for Database Professionals, Second Edition
- No system images installed for this target
- Creating Web Pages All-in-One Desk Reference For Dummies
- Acegi Security System for Spring 书摘(II)
- Android 导入工程出现 unable to get system library for the project
- Distributed system for fun and profit笔记1
- 创业公司的点子创意——Ideas for Startups
- Error 270 for archive log file ORA-00270: error creating archive log Error 1033 received logging on
- c3po异常:Creating emergency threads for unassigned pending tasks!