The Classic Way

First, lets look at Twitter as it is right now. Yes, Twitter is a social application and as such, each of the network’s nodes, each user, is connected to a certain number of other nodes. The apparent structure of the service is thus the one of a peer-to-peer network or a decentralized network. However, from a technology point of view, this is not the case. Twitter is a fully centralized network.

To do whatever action you want to do, you have to go through the company’s servers. Every single request, whether it is through the Twitter public API or directly on the website, has to be processed by Twitter. All the clients do is parse the server’s response, all the hard work is done on Twitter’s side.

Why is this bad? Because Twitter’s users check often for new messages since they post a lot of messages. Also, Twitter’s open API led to the creation of many applications based on the service, the worst for the company being clients and automated services. First, a lot of clients, to stay up to date, will make frequent requests to the servers to see if something new was posted. This wouldn’t be too bad, for example, with a blog service where people only post once in a while, because the servers could use a cache system to reduce unnecessary processing. However, with its thousand of new tweets per minute (and growing), a caching system is hardly useful for Twitter. The second problem is automated applications such as WordPress plugins or blog update services. These service also make a lot of requests, every time a blog page is loaded in the case of WordPress and at a certain rate for the update services. Again, each request sent by the service has to be processed on Twitter’s servers, thus requiring a lot of processing power without the user even seeing the little Twitter bird.

The Efficient Way

So how can we solve this problem? Well, since Twitter is, by its social nature, decentralized, why not try to decentralize the system? This is what peer-to-peer applications such as BitTorrent and Skype have been doing for a while, and it seems to work well. Starting with the second generation of peer-to-peer networks, decentralization became a key idea. This culminated with the integration of Distributed Hash Tables (DHT) in peer-to-peer protocols and clients. From Wikipedia:

Distributed hash tables (DHTs) are a class of decentralized distributed systems that provide a lookup service similar to a hash table: (key, value) pairs are stored in the DHT, and any participating node can efficiently retrieve the value associated with a given key. Responsibility for maintaining the mapping from keys to values is distributed among the nodes, in such a way that a change in the set of participants causes a minimal amount of disruption.

So how does this apply to Twitter? Well, with its minimalist structure, Twitter is the perfect candidate for a DHT system. DHTs use key value pairs for data management and Twitter is just that, a key value system. The key is your username, and the value your tweets. What makes this an even better solution, is the simplicity of the service. Compared with Facebook activity or blog posts, tweets doesn’t include complex data such as images or text formatting. It is all about text, and small pieces of text. This not only makes storing user’s data easy, but it also requires less space. A simple XML file of an average size could be use to store a user’s full history of tweets and a list of all of his followers/following.

The idea of DHTwitter is to create an additional layer of abstraction, this time between the Twitter API and the client or application used by the end user. This API would include the overlay network and the keyspace partitioning specifications needed to establish the DHT. Once this is done, the API would redirect requests to the Peer Cloud to see if the information is available before hitting Twitter’s servers. If the user wants to publish a tweet, then all the API has to do is to publish it in the Peer Cloud as well as on Twitter’s servers to be accessible from everyone, even people not using DHTwitter. By sending the information part by part and by sending the latest tweets first, transfer speed would be less of a problem. Also, keeping a user’s friends close to him in the overlay network would make request processing in the Peer Cloud faster.

This would seriously reduce the load on Twitter’s server by distributing the lookup and parsing processing throughout the Peer Cloud. It would reduce the number of server hits by connecting the users together, in the same way they connect in the social layer.

Possible Implementation

For now, this is only an idea. I am not planning on coding an implementation on my own in the near future. However, if someone is interested, please let me know, I would love to participate in the development. The biggest challenge in implementing DHTwitter is the lack of libraries for DHTs. Full system specifications exist; Kademlia, Pastry and Tapestry(Chimera) are good examples. However, there is a lack of solid implementation. Librairies are practically nonexistent and most practical implementations are hard coded into existing applications, such as Vuze or LimeWire. Again, if you thought of a way this could be implemented, please let me know.

First of all, I have to say that I love Twitter. I think it is a really useful tool for the rapid spread of what I would call bleeding edge knowledge. It is easy to find and spread the latest news in a variety of domains, mostly through links to interesting blog posts. My fears are for Twitter Inc, the corporation behind the service.

Lately, Twitter has been opening its API like no proprietary web service before. Back in September, when I started working with their different developer tools, the only option available to me was to use an API key to make requests using a limited sets of functions. Today, with the integration of OAuth to the API and the development of other useful tools, connecting to Twitter has never been easier. It got to the point where the user has dozens of alternative to the official website to connect with its fellow twitterers, some of which are way more advanced than the original web interface or target specific aspects of the Twittersphere. Based on recent statistics, less than 75% of the users post their updates via the web client. The question is: is this a good thing?

From the point of view of the user, it is the best thing that could happen. Opening the API to developers worldwide means faster and better innovation. Twitter’s users are prosumers that don’t passively use the end product, but also develop it based on their needs. The peer pioneering behind Twitter lead to the creation of whole new services, like Twitpic, that integrates seamlessly to the main service via the open API. This means more alternative for the end user and, as a result, a better experience.

The conclusion is the same from the service’s point of view. By opening itself to the world, Twitter gains new features and users everyday, making it more useful and purposeful. Clients like Tweetdeck and Seesmic Desktop make it easy to reorganize and mash the data available on the Twittersphere, keeping only the parts that are relevant. You can easily find what is most discussed by visiting Twitscoop or automatically post with Twitterfeed. It is these services that make the Twitter service more than a 140 characters blog.

The problem is with Twitter Inc. All these API requests have to be managed by some central system, a system that happens to be owned and operated by Twitter Inc. Every request has to go through one of Twitter’s server, creating operation and maintenance costs that they have to pay to stay online. They could have tried to payoff their bills with ads like other Web 2.0 companies, but not only do they don’t display ads, but they soon may not even display any content at all. As I said before, less than 75% of the users use the official web interface. That means that more than 25% of the users don’t even bother going on the Twitter website and this number is growing. By decentralizing a system that, in is nature, is centralized, Twitter Inc is facing the risk of becoming a simple content provider for more advanced clients or at least, to lose potential profits to someone else. Even worse is the fact that its problem, the widespread availability of the content, is also its strength. Even identi.ca, a free and open source micro-blogging platform, seems to have a better business plan with its status.net initiative. So how can Twitter be worth $700 millions? Hype. But hype is not enough in the long run. With a loss of control over the distribution of its content, Twitter will have to find alternative way to start making money or it will become part of the past.

We plan to build Twitter, Inc into a successful, revenue-generating company that attracts world-class talent with an inspiring culture and attitude towards doing business.

About Twitter

I hope for them that this is not just a dream.

At the end of the competition, the Yahoo! team ended up with 24 projects to judge. Each team had to present his project in less than five minutes. After 2 hours of presentations and 20 minutes of deliberation between the judges came the announcement of the winners. This is where I learned that Globami finished third overall. Not bad for a first year student.

Thanks to the Yahoo! team, especially to Rasmus and Paul, for this great week.