Pipelines: Plumbing for the next web

APIs are a great way of developers and hackers being able to access data and content from one provider. But with the trend of the “mash-up” has come the ability to join two or more providers together to the benefit of the user rather than the providers. This level of interoperability means people can start offering automation and new business opportunities by “chaining” services together. As many of us look towards the social benefits of a somewhat provider centralised Web 2.0, its easy to see how our single provider habits can be broken by user generated pipelines.

Like Unix Pipelines, a user generated pipeline defines how content and data goes through a series of pipes. Unlike UNIX pipelines these pipes can be a series of remote or local web-services, applications, transformers. A simple example could be, uploading a photo from your mobile phone to Flickr.com, then that same photo magically appears on your friends doorstep processed, nicely cropped with a related personal message with no more time or effort required from yourself. That sounds like magic to a user but is perfectly workable to a developer with enough time.

The Flickr example I gave works on an application being authorised to access a certain picture on Flickr. Flickr already has this feature in its API and many other services use this to provide services to there users. So in this example Preloadr.com are instructed to receive the picture and do the default image enhancement which there famous for. After the preloadr is finished the picture is passed on to delivr.net which can create postcards and send them to a person on request.

The developer would normally use multiple available APIs to create this task. Those services and applications which are not available will struggle to survive in the new ecosystem. Open services would be happy with the traffic and the business. The user would be happy with the time saving and convince. So ideally everyone will be happy.

This is not a new concept but how we manage this has existed in the domain of Apple-scripter’s, Perl and Python hackers. Automator by Apple is an example of this, but fails to gain momentum due to its proprietary nature to the OSX operating system.

Touchstone (currently in private alpha) is a next generation desktop RSS aggregator which has the concept of filters which can trigger things to happen like a email to a friend or an automatic post to del.icio.us. It has many plugin’s and the ability to add more, but its uncertain if the filters can be shared and are non proprietary.

I’m proposing that a series of pipelines will be ultimately definable, non-proprietary and shareable by anyone who can install and run a browser. If correctly executed a whole eco-system could grow out of this decentralised user driven behavior. I believe XML Pipeline Definition Language is the key to making this happen, but currently lacks vision.