Rob Kaye has a good write-up of Kellan Elliott-McCrae and Evan Henshaw-Plath (Rabble)'s interesting XMPP talk at OSCON '08. There were a couple of things worth commenting on:
Rabble presented using XMPP for FireEagle, Yahoo!'s new personal geolocation service that allows users to provide their current location to other users. For a few users and a few updates you can paginate the data stream into RSS/atom feeds. But once you have more than a few users and frequent updates a paginated stream cannot keep up. What if a user publishes more updates than can an RSS feed can capture? Updates get lost -- and for applications using FireEagle missing an update presents a critical flaw. Using a system like XMPP, FireEagle can rely on Jabber to deliver all the updates -- exactly what Jabber was meant to do.
Pagination of feeds is worth a mention here: Google Reader, for example, uses this for delivering a continuous list of items to the browser, and so do OAI and PubMed's EUtils for sequentially retrieving chunks of items from a long list of results. All it takes is a key that fixes the initial query, and 'start' and 'end' parameters that determine which items to retrieve. This solves the "What if a user publishes more updates than can an RSS feed can capture?" problem nicely.
As for "FireEagle can rely on Jabber to deliver all the updates", as I understand it XMPP (unlike HTTP's response packets) has no mechanism for confirming delivery of a message. There might be an XEP for this that I haven't seen, though.
Kellan also applied XMPP/PubSub to Flickr and how a Flickr update "Firehose" might work. If Flickr sends a ~2k an atom enriched packet for each new public picture posted at a rate of 60 updates a second, it would take roughly a megabit of traffic. Even a normal DSL line can handle one mbit of traffic, so the network effects are manageable on this level, compared to the polling system that FriendFeed uses. (Kellan also points out that FriendFeed is not doing anything wrong at all -- the current web service centric model is simply insufficient for this type of service.)
Fine, if Flickr's only sending a firehose stream of XMPP packets to one listener (FriendFeed), but what happens when there are 1000 listeners, or more? Flickr's XMPP server has to send a copy of the packets to each of the listeners' XMPP servers, which each have to be able to handle the incoming data. It's not unmanageable, but does present some problems.
The open and mature infrastructure that Rabble and Kellan found to use for this service is Jabber. Jabber has 10 years of experience of passing messages around the internet and has been embraced by many companies including Google.
Except Google might well be moving away from XMPP to a binary protocol for Google Talk on mobile devices, in Android, to reduce the communication overhead. While XMPP support will probably get back into Android, maybe a binary protocol (not this one) is better suited to high-volume notifications?
Comments
All fields are optional, email address will not be shown; no HTML, URLs are automatically hyperlinked.
Regarding delivery confirmation, IQ queries are ACKed with an IQ stanza of type "result". Also there exists XEP-0198: Stanza Acknowledgements (I never used it).
And ZLib compression (and TLS) reduces bandwidth consumption a lot, although it requires more CPU.
XMPP vs. REST was a recurring theme at OSCON. The finer details are a little beyond my understanding, but I am still not completely convinced.
Regarding XMPP and message delivery, because XMPP is not a direct client-server protocol, but arbitrated by an intermediating server, it's the server's job to detect when delivery fails and store that message for later. Offline message storage is a core feature of XMPP servers.
If the sender can talk to the server, then the client need not worry about eventual delivery. And if the receiver can (eventually) talk to the server then the messages will be delivered (or dropped if they are too old, as defined by the server).
As for distributing messages to a list of recipients, pub-sub solves the scaling problem by again pushing the load off onto the server. The sender targets one resource and that resource then broadcasts to the rest of the distribution list. Of course, I know you know that, but XMPP is particularly adept at low latency and low overhead message passing.
I'd like to supply some numbers as food for thought. Take them as a data point, and something to compare against. Last Monday, July 21, 2008 (for historical reference) PINES (the Georgia state-wide resource sharing consortium running on the Evergreen ILS) saw just under 22 million API calls cross its XMPP servers. Each call causes at least two XMPP messages, one for the request and one for the response, and that is in the simplest case. Some responses generate multiple messages, both status and data. Several APIs require the creation of a stateful session, generating at least three more XMPP messages. It's not unreasonable to estimate the number of XMPP messages at greater than 50 million. The CPU power required of the ejabberd server to pass these messages around is approximately 2 fully utilized Xeon cores at 2.4GHz during peak hours.
Incidentally, these are the main reasons why we chose XMPP as the native message passing protocol for the Evergreen backend. It's very low overhead, extremely reliable and because it is connection oriented it has much lower latency and startup cost than HTTP for high-traffic environments.
--miker
"This solves the "What if a user publishes more updates than can an RSS feed can capture?" problem nicely."
how does it?
Isn't this about pull vs push? I think what he meant was for aggregation. If you are relying on RSS for aggregation and the RSS producer produces lots of items, only last items will show up on the RSS feed. and as a consumer, you may miss some items behind your last check and the previous one since there is no history.
but with a push model, you don't have to check periodically as all the updates will come to you as soon as published.