For those interested in getting a job there, here are a couple of tips I learned from my interviews. First, do stuff on you own. Everybody does the school projects, that doesn’t make you special. Don’t go over the “reverse a linked list” or “how do I find anagrams” algorithms before your interview, you would be losing your times. They come up with unusual questions that are not too hard, but that require some knowledge of basic algorithm concepts (e.g. dynamic programming) and that test your problem solving skills, not your memory. If your want to train for these, Project Euler is a good place to start. Also, make sure to visit their programming puzzles page.

The result is called JobU and can be found here: http://renaudbourassa.com/projects/jobu/.

We won 4th place (when really, we should have won at least 3rd). The first place went to Docuvine, a really interesting web application that mimics Google Wave. Here are some picture of the event.

For more pictures of the event, see Rasmus Lerdorf or Jamie Lockwood flickr photostream.

And also, the Warriors defeated the Windsor Lancers 49-0. Let’s go Warriors!

I thus decided to implement these simple, but oh so useful data structures. Sure, you could reply that ruby arrays already implement all the functionalities provided by theses data structures. True, but when you are dealing with linked lists of millions of elements, having to relocate the whole array in memory to add an element can prove to be a lenghty task. Also, finding a memory chunk big enough to hold the whole array may prove challenging. Ruby arrays grow pretty fast (~1.5x) as can be seen from the following code snippet from array.c (ruby 1.9.1p243) and can rapidly become hard to store in memory.

if (idx >= ARY_CAPA(ary)) {
    long new_capa = ARY_CAPA(ary) / 2;

    if (new_capa < ARY_DEFAULT_SIZE) {
        new_capa = ARY_DEFAULT_SIZE;
    }
    if (new_capa >= ARY_MAX_SIZE - idx) {
        new_capa = (ARY_MAX_SIZE - idx) / 2;
    }
    new_capa += idx;
    ary_resize_capa(ary, new_capa);
}

And finally, theses data structures implement functionalities that are badly supported by arrays such as deleting or adding an element in the middle of a list.

The stack and queue implementation are separated from the linked list implementation to keep them as simple as possible. The linked list implementation is more extensive and is also used to implement the sorted linked list data structure. The next data structure to be added to Batfish will probably be graphs and it should include a number of basic graph algorithms.

Trie

A trie is a data structure that is used to store an associative array where the array’s keys are strings. It has the same structure as any other tree, except that keys are not stored in nodes. Instead, each edge has a character associated with it and you browse the trie by going down the edges, one character at a time, until you reach the end of the key. The node you reach this way contains the value associated with the key.

Tries have several advantages over binary search trees. First, the complexity of trie lookup is O(L) where L is the length of the key while it is of O(n) where n is the number of elements in the tree for a BST. It also takes less space since different keys overlap. It also have advantages over hash tables. First, the keys, in a trie, are ordered, which makes it a useful data structure to use to store a dictionary. It can also lead to faster lookup depending on the hash function and considering that collisions are possible with string hashes.

More informations on tries can be found here.

BK-Tree

A BK-tree is a useful data structure for nearest neighbor lookup in discrete metric spaces. A metric space is any space that obeys the following rules, where d(a,b) is the distance between a and b.

The later is also known as the triangle inequality. It basically states that there is no shorter way to go from a point to another than the direct way. Examples of discrete metric spaces, that is, where the distances are integers, are the real numbers or the levenshtein distance between strings.

The BK-Tree is constructed by measuring the distance between the value to insert and every node, going down the edges corresponding to the distance at each node. Once an unregistered distance at a node is calculated for that node, the value is attached to it. The lookup process works by going down each edges in the distance to the node ± the lookup treshold range until a node with a distance equal to the treshold value or less is found. It is thus possible to find all the nodes within a certain distance of a value without going through each nodes. However, the larger the threshold, the more nodes you have to visit.

For more information on BK-trees, you can read the following article.

In their book Wikinomics: How Mass Collaboration Changes Everything, Don Tapscott and Anthony D. Williams describe a new business model, Business 2.0, that takes advantage of the new realities of the digital age. Their theory sits on four main principles: being open, peering, sharing and acting globally. They then use them to describe seven business strategies to harness the power of mass collaboration.

The Four Principles

The four principles are at the core of the Wikinomics theory. These ideas contrast with the principles of previous business models which promote a closed vision of business. Conventional wisdom says companies should protect their intellectual property fiercely, they should keep research and development in-house and keep control over every steps of the development cycle of their products. Wikinomics is all about enterprises opening themselves to the world and as such, stands on different principles.

The first is being open, but openness has a lot of different meanings. Openness could mean openness in research. With science and technology evolving faster than ever before, it is not possible for a single company to do everything in-house. If it wants to stay competitive, it has to open its boundaries and harness ideas from the outside. Openness could also means open standards. This is something that is common in the software world where standards like XML make it easier for companies to develop compatible products. It could also mean transparency. Disclosure of pertinent information to interested parties like shareholders or partners can only lead to improvements. This is also true with costumers, where more information can lead to better feedback and creation of new ideas via product customization.

The idea of peering severely contrast with the common idea that corporation should be highly hierarchized to be functional. Peer production community have been around for a while already. You can find them everywhere in the Open Source movement and on social networks. However, as our society evolves, this idea makes more and more sense in a business environment. Peering succeeds where hierarchical management fails by leveraging self-organization. By harnessing ideas and opinions from a wider group of people, companies open themselves to more opportunities.

Conventional wisdom says company should control and protect their assets as much as they can. However, sharing may end up being more profitable. Sharing crucial R&D information can prove profitable if it leads to the discovery of the missing piece of the puzzle. This leads to win-win interactions between companies, such as pharmaceutical companies that win from having a better knowledge of the human genome by giving away some of their discoveries in this field in return of other companies discoveries. This could also lead to the licensing of assets that are unused, but that can prove useful to someone else. This can also be extended to the field or computer science where distributed systems can share the computing load between users in order to reduce the total overload on a company’s servers.

The last principle, but not the least, is acting globally. Globalization is not new, it has been there since the end of the Second World War with the introduction of the GATT. However, the phenomenom have accelerated lately with collaboration over long distance made possible by the widespread usage of new means of communications such as the Internet. This allows company to tap into a bigger pool of human and physical resources. However, companies still tend to act locally in multiple places by opening branches that perform the same work in parallel. Acting globally means more than that. It means breaking down geographical boundaries and act as a trully global company by having global workforces that work together, by putting forward global processes that minimize redundancies between different locations and usually, by having a global IT platform that allows for easy collaboration and sharing between distant locations seamlessly.

Seven Business Strategies to Harness Mass Collaboration

Theory is always fun, but knowing how to apply it is usually better. Economics being more of an applied science than a pure one, every business models needs its share of practical strategies and processes. Here is a brief overview of seven business strategies described in details by Don Tapscott and Anthony D. Williams in their book to help enterprises harness the opportunities offered by mass collaboration.

“Peer Pioneers” or how thousands of dispersed volunteers can harness the power of peering and sharing to create fast, fluid, and innovative projects that outperform those of the largest and best-financed enterprises. Peer producers apply Open Source principles to create products usually made of bits. Good examples of this are the Linux community or the Apache project and how companies like IBM collaborate with them.

“Ideagoras” or how marketplaces for ideas, inventions, and uniquely qualified minds enable big companies such as Procter & Gamble to tap global pools of highly skilled talent more than ten times the size of their own workforce. These Ideagoras give companies access to a global marketplace that they can use to extend their problem-solving capacity. Good examples of this are InnoCentive and yet2.com.

“Prosumers” or the increasingly dynamic world of customer innovation, where a new generation of producer consumers considers the “right to hack” its birthright. These communities working around a common product can be an incredible source of innovation if companies give customers the tools they need to participate in value creation. A good example of this is LEGO and their LEGO Factory initiative.

“New Alexandrians” or how a new science of sharing will rapidly accelerate human health, turn the tide on environmental damage, advance human culture, develop breakthrough technologies, and even discover the universe all the while helping companies grow wealth for their shareholders. The New Alexandrians, scientists, researchers and enterprises from all around the world, are ushering in a new model of collaborative science that will lower the cost and accelerate the pace of technological progress in their industries. A good example of this is Intel and its industry-university collaboration program.

“Platform for Participation” or how smart companies are opening up their products and technology infrastructures to create an open stage where large communities of partners can create value, and in many cases, create new businesses. These platforms create a global stage where large communities of partners can create value and, in many cases, new businesses in a highly synergistic ecosystem. Good examples of this are Twitter and its open API or Facebook and the Facebook applications platform.

“Global Plant Floor” or how even manufacturing-intensive industries are giving rise to planetary ecosystems for designing and building physical goods, marking a new phase in the evolution of mass collaboration. The plant floors harness the power of human capital across borders and organizational boundaries to design and assemble physical things. A good example of this is Boeing and the development process of its 787 Dreamliner.

“Wiki Workplace” or how mass collaboration is taking root in the workplace and creating a new corporate meritocracy that is sweeping away the hierarchical silos in its path and connecting internal teams to a wealth of external networks. These new workplaces increase innovation and improve morale by cutting across organizational hierarchies in all kinds of unorthodox ways. A good example of this is the Geek Squad program.

Wrapping It Up

Overall, Wikinomics is a pretty good book for everyone interested in the world of business. It poses a new look on subject that evolves fast and it brings new solutions to problems that used to look impossible to solve.

The Classic Way

First, lets look at Twitter as it is right now. Yes, Twitter is a social application and as such, each of the network’s nodes, each user, is connected to a certain number of other nodes. The apparent structure of the service is thus the one of a peer-to-peer network or a decentralized network. However, from a technology point of view, this is not the case. Twitter is a fully centralized network.

To do whatever action you want to do, you have to go through the company’s servers. Every single request, whether it is through the Twitter public API or directly on the website, has to be processed by Twitter. All the clients do is parse the server’s response, all the hard work is done on Twitter’s side.

Why is this bad? Because Twitter’s users check often for new messages since they post a lot of messages. Also, Twitter’s open API led to the creation of many applications based on the service, the worst for the company being clients and automated services. First, a lot of clients, to stay up to date, will make frequent requests to the servers to see if something new was posted. This wouldn’t be too bad, for example, with a blog service where people only post once in a while, because the servers could use a cache system to reduce unnecessary processing. However, with its thousand of new tweets per minute (and growing), a caching system is hardly useful for Twitter. The second problem is automated applications such as Wordpress plugins or blog update services. These service also make a lot of requests, every time a blog page is loaded in the case of Wordpress and at a certain rate for the update services. Again, each request sent by the service has to be processed on Twitter’s servers, thus requiring a lot of processing power without the user even seeing the little Twitter bird.

The Efficient Way

So how can we solve this problem? Well, since Twitter is, by its social nature, decentralized, why not try to decentralize the system? This is what peer-to-peer applications such as BitTorrent and Skype have been doing for a while, and it seems to work well. Starting with the second generation of peer-to-peer networks, decentralization became a key idea. This culminated with the integration of Distributed Hash Tables (DHT) in peer-to-peer protocols and clients. From Wikipedia:

Distributed hash tables (DHTs) are a class of decentralized distributed systems that provide a lookup service similar to a hash table: (key, value) pairs are stored in the DHT, and any participating node can efficiently retrieve the value associated with a given key. Responsibility for maintaining the mapping from keys to values is distributed among the nodes, in such a way that a change in the set of participants causes a minimal amount of disruption.

So how does this apply to Twitter? Well, with its minimalist structure, Twitter is the perfect candidate for a DHT system. DHTs use key value pairs for data management and Twitter is just that, a key value system. The key is your username, and the value your tweets. What makes this an even better solution, is the simplicity of the service. Compared with Facebook activity or blog posts, tweets doesn’t include complex data such as images or text formatting. It is all about text, and small pieces of text. This not only makes storing user’s data easy, but it also requires less space. A simple XML file of an average size could be use to store a user’s full history of tweets and a list of all of his followers/following.

The idea of DHTwitter is to create an additional layer of abstraction, this time between the Twitter API and the client or application used by the end user. This API would include the overlay network and the keyspace partitioning specifications needed to establish the DHT. Once this is done, the API would redirect requests to the Peer Cloud to see if the information is available before hitting Twitter’s servers. If the user wants to publish a tweet, then all the API has to do is to publish it in the Peer Cloud as well as on Twitter’s servers to be accessible from everyone, even people not using DHTwitter. By sending the information part by part and by sending the latest tweets first, transfer speed would be less of a problem. Also, keeping a user’s friends close to him in the overlay network would make request processing in the Peer Cloud faster.

This would seriously reduce the load on Twitter’s server by distributing the lookup and parsing processing throughout the Peer Cloud. It would reduce the number of server hits by connecting the users together, in the same way they connect in the social layer.

Possible Implementation

For now, this is only an idea. I am not planning on coding an implementation on my own in the near future. However, if someone is interested, please let me know, I would love to participate in the development. The biggest challenge in implementing DHTwitter is the lack of libraries for DHTs. Full system specifications exist; Kademlia, Pastry and Tapestry(Chimera) are good examples. However, there is a lack of solid implementation. Librairies are practically nonexistent and most practical implementations are hard coded into existing applications, such as Vuze or LimeWire. Again, if you thought of a way this could be implemented, please let me know.