Comments on: Where’s the Web in “Real Time Web”?

By: mikepk

mikepk — Thu, 30 Apr 2009 00:08:09 +0000

Phil, interesting tech I'll take a look. With the caveat that I haven't looked very deeply at it, in terms of infrastructure, I don't think liberator gives us the whole package. Liberator seems strongly suited for broadcast to endpoints of the content network. I could envision industrial strength comet servers for content distribution, but then an alternate internal distributed update mechanism I think could be done with more direct, application specific, protocols. What I'm exploring now is more of the idea of a RTW stack where the top level protocol is embedded in the content payload along with authorization. Basically make the updates and keys be transport agnostic. This gives you a lot of interesting abilities, like for not-so-real time updates you might use something like email as a transport for bulk messages. This would allow utilizing existing distributed infrastructure and addressing, and then possibly having member machines move the content along on whatever transport makes the most sense (XMPP, comet to the client etc…). It could also act as a strong bootstrap, rather than force the network to adapt to the new requirements, utilize existing infrastructure even if it's not perfect. It's just the start of an idea so I'm still exploring the ramifications.

By: Phil Leggetter

Phil Leggetter — Wed, 29 Apr 2009 22:54:59 +0000

Ok, I'm a bit late responding to this article. I bookmarked it a while back and just got around to reading over it. I a software engineer working for a company who have components that facilitate the distribution of data across the web in real-time. I’m hopeful that we have the real-time web silver bullet. The section of our website that details streaming push server is here: http://www.caplin.com/caplinplatform/?curart_id=36

The server is a highly scalable comet server (Liberator) that supports thousands of concurrent connections, manages connection problems, load-balancing, clustering (basically, it's highly scalable) and a whole host of other features that I think are exactly what the real-time web need. The really good news is that there is a free version available: http://www.freeliberator.com

I'm presently trying to pushing to encourage people to use our components in areas other than the financial sector since the Liberator is an excellent piece of kit that could be put to so many other uses. It can be used to distribute real-time updates across the Internet to a number of clients (we have APIs for Java, .NET, JavaScript/Ajax). Obviously the standard clients are web pages, .NET Web Form apps and Java applications but this doesn’t have to be the case. A web page example that I really like isthe Twitter website; they’ve experienced numerous problems and even today I saw the fail whale due to high load. By using Liberator and SL4B (the JavaScript API) there would be less frantic hitting of F5 to reload the page putting strain on the Twitter servers, updates would simply be pushed to clients as soon as they become available. Liberator also caches content taking additional strain from the Twitter servers.

I recently wrote a post about What is the Real-Time web which I think is relevant here: http://blog.caplin.com/2009/04/20/what-is-the-r...

Has anybody tried out Free Liberator? Does anybody think that this could be the silver bullet for the real-time web? If no, what is it missing?

By: mikepk

mikepk — Wed, 29 Apr 2009 20:08:09 +0000

By: Phil Leggetter

Phil Leggetter — Wed, 29 Apr 2009 18:54:59 +0000

By: mikepk

mikepk — Thu, 16 Apr 2009 21:56:27 +0000

Bram, XML is what we have now. Its primary problem (with regards to real time) is that agents have to continuously poll to find out if there's anything new to consume. It's inefficient and wasteful use of computing resources, especially if you intend to track more than a handful of sources. For any significant number of sources a polling paradigm just won't scale (and keep near-real time data). Imagine a long-tail of several thousand blogs that you want to filter for content but be able to get the data in near real time. Even Google, with it's massive resources, can only promise feed updates (for google reader) in about a 2 hour window. That's not good enough (IMHO) to be called real time.

By: Bram Pitoyo

Bram Pitoyo — Wed, 15 Apr 2009 03:24:23 +0000

“Ping networks, XMPP, update streams, FriendFeed’s SUP protocol, all try to improve update efficiencies but all seem to suffer from specific shortcomings.”

What about using technologies like XML to deliver information?

As someone who uses a Social Intelligence Dashboard to consume and filter a large number of content sources (of varying timescale: short ones like Twitter, and long ones like blog posts), RSS or ATOM seems to me like an easy and near-real time way to do it.

But this carries two inherent limitation: 1) RSS, while universal, isn’t rich. 2) Pass these files through any third party filtering and splicing service (like Yahoo!Pipes, PostRank, etc.) and the latency increases.