I thus decided to implement these simple, but oh so useful data structures. Sure, you could reply that ruby arrays already implement all the functionalities provided by theses data structures. True, but when you are dealing with linked lists of millions of elements, having to relocate the whole array in memory to add an element can prove to be a lenghty task. Also, finding a memory chunk big enough to hold the whole array may prove challenging. Ruby arrays grow pretty fast (~1.5x) as can be seen from the following code snippet from array.c (ruby 1.9.1p243) and can rapidly become hard to store in memory.

if (idx >= ARY_CAPA(ary)) {
    long new_capa = ARY_CAPA(ary) / 2;

    if (new_capa < ARY_DEFAULT_SIZE) {
        new_capa = ARY_DEFAULT_SIZE;
    }
    if (new_capa >= ARY_MAX_SIZE - idx) {
        new_capa = (ARY_MAX_SIZE - idx) / 2;
    }
    new_capa += idx;
    ary_resize_capa(ary, new_capa);
}

And finally, theses data structures implement functionalities that are badly supported by arrays such as deleting or adding an element in the middle of a list.

The stack and queue implementation are separated from the linked list implementation to keep them as simple as possible. The linked list implementation is more extensive and is also used to implement the sorted linked list data structure. The next data structure to be added to Batfish will probably be graphs and it should include a number of basic graph algorithms.

Trie

A trie is a data structure that is used to store an associative array where the array’s keys are strings. It has the same structure as any other tree, except that keys are not stored in nodes. Instead, each edge has a character associated with it and you browse the trie by going down the edges, one character at a time, until you reach the end of the key. The node you reach this way contains the value associated with the key.

Tries have several advantages over binary search trees. First, the complexity of trie lookup is O(L) where L is the length of the key while it is of O(n) where n is the number of elements in the tree for a BST. It also takes less space since different keys overlap. It also have advantages over hash tables. First, the keys, in a trie, are ordered, which makes it a useful data structure to use to store a dictionary. It can also lead to faster lookup depending on the hash function and considering that collisions are possible with string hashes.

More informations on tries can be found here.

BK-Tree

A BK-tree is a useful data structure for nearest neighbor lookup in discrete metric spaces. A metric space is any space that obeys the following rules, where d(a,b) is the distance between a and b.

The later is also known as the triangle inequality. It basically states that there is no shorter way to go from a point to another than the direct way. Examples of discrete metric spaces, that is, where the distances are integers, are the real numbers or the levenshtein distance between strings.

The BK-Tree is constructed by measuring the distance between the value to insert and every node, going down the edges corresponding to the distance at each node. Once an unregistered distance at a node is calculated for that node, the value is attached to it. The lookup process works by going down each edges in the distance to the node ± the lookup treshold range until a node with a distance equal to the treshold value or less is found. It is thus possible to find all the nodes within a certain distance of a value without going through each nodes. However, the larger the threshold, the more nodes you have to visit.

For more information on BK-trees, you can read the following article.

If you are interested in implementing any other functionality or website, don’t hesitate to fork the project on github. Here is the code of the Download superclass.

# The Download superclass
class Download
 attr_accessor :url, :name

 # Initializes a download
 def initialize(url, opts)
 @url = url
 @opts = opts
 @file = nil
 end

 # Tests the url
 def test
 begin
 Net::HTTP.get_response(URI.parse(url)).value
 rescue
 return false
 end
 return true
 end

 # Executes the download
 def execute
 self.set_file
 self.download
 end

 # Sets the file for the download
 def set_file
 @file = @url
 end

 # Downloads the file
 def download
 cmd = "wget -c --user-agent="#{@opts[:header]['User-Agent']}""
 cmd += " -q" if !$verbose
 cmd += " --directory-prefix="#{@opts[:dir]}"" if @opts[:dir]

 IO.popen(cmd + " #{@file}"){|f|}
 end
end

My solution: RSDownloader. It is a small ruby script I wrote to easily download any number of files from RapidShare. The script graps all the wait times directly from the site, thus optimizing your time. It starts the downloads (approximately) as soon as the countdowns reach 0. It uses wget to get the files so it is for Linux and motivated MacOSX users only. You can get the script from my github or here.

Edit: Since I keep making change to the source code, I decided to take it off this post. The latest “stable” version should always be available on my github.

Now back to calculus.

We also wrote a backward program for extra points. We are one of the only team who got it to work. Here is the video of it.

So this is it for this project. The only thing remaining is adding the final touches to the final report and send it. If we can do as good with the final report as we did with the preliminary design report, this project would be nearly perfect. Well done Team Nerdcore!

I also wrote a directory view script to display the files of projects that are not web projects. It uses PHP to retrieve the file list and a jQuery script I wrote to reorder the files without reloading the whole page every time. It also more visually appealing than the classic Apache directory view.

Also, I have no plan to add any other networks now. If you have a request for a specific network, just post it here and I will see what I can do.