At BackType, we use several real-time protocols to index content and conversations as well as provide access to our services and data. We recently added PubSubHubbub support to our feeds with the help of Superfeedr. Superfeedr does real-time feed parsing in the cloud; in doing so, they provide publisher services and setup a hub for BackType.

PubSubHubbub is an open protocol for distributed Publish/subscribe communication on the internet. The protocol extends the Atom and RSS protocols for data feeds. The main purpose is to provide near-instant notifications of change updates, which would improve on the typical situation where a client periodically polls the feed server at some arbitrary interval.

Implementation

1. Subscriber polls Publisher's feeds. The feed contains a forward link to the Hub.
   • We added this forward link to our feeds:

     <link="http://www.w3.org/2005/Atom" rel="hub" 
     href="http://backtype.superfeedr.com/"/>

   • Subscribers can then discover the hub: ^[1]

     import push_subscriber
     feed_url = 'http://feeds.backtype.com/mm'
     hub_url, self_url = push_subscriber.get_links(feed_url)

2. Subscriber POSTs subscription request to the Hub. The request contains the endpoint URL where the hub should POST new updates.

   push_subscriber.subscribe_topic(hub_url, self_url, subscribe=True)

3. Hub POSTs to the endpoint URL to verify the request was authentic; Subscriber responds with confirmation to the Hub.
4. Publisher notifies Hub about updates by POSTing feed URLs to the Hub; Hub pulls the feed again to find new entries: ^[2]

   import pubsubhubbub_publish
   hub_url = 'http://backtype.superfeedr.com/single'
   feed_url = 'http://feeds.backtype.com/mm'
   pubsubhubbub_publish(hub_url, feed_url)

5. If the Publisher does not inform the Hub, the Hub will periodically poll the Publisher's feed for new updates.
   • We already support Simple Update Protocol for our feeds, which allows services like FriendFeed to fetch users' updates in near real-time
   • Superfeedr has implemented SUP over PubSubHubbub
6. When Hub receives new update to feed X, it POSTs the update to the Subscriber's endpoint URL.
7. If feed X has multiple subscribers, the Hub sends updates to all of them. This reduces load on the Publisher.

Conclusion

The real-time web is still in it's infancy, so we're sure to see this protocol and others fleshed out more. At BackType we're excited to be among the early adopters of many of these technologies, and we will continue to push the limits so we can deliver better experiences to users.

[1] A PubSubHubbub subscriber library: http://github.com/adamac/PuSH-Subscriber/blob/master/push_subscriber.py
[2] A publisher client: http://code.google.com/p/pubsubhubbub/source/browse/trunk/publisher_clients/p...