BTW, if don’t know what Taiko is or have never heard Taiko, I refer you to the mighty Wikipedia and the mightier YouTube:

I’m on a temporary hiatus from Taiko right now, but I had an amazing experience the few months I spent with SF Taiko Dojo.

Yes, there are rules and etiquettes. But in a society where anything goes and freedom rules and any kind of “discipline” is often frowned upon, SFTD was almost refreshing. In many ways, it was reminiscent of the Gurukul system in ancient India.

Taiko itself is a wonderful art form. There is something powerful about a Taiko performance. A single drum is an excellent percussion device, but in a group, Taiko takes a life of its own. Like most art forms, you can pick up the basics real quick. But to go deep into Taiko, you need time, patience and a lot of hard work. The veterans at SFTD have been playing for 10-15 years and still learning.

Needless to add, Taiko is also a fantastic full body workout. It is a combination of dance, drumming, music and more. The classes are fun, but you do need serious commitment if you want to become an advanced Taiko player. The folks in the adult beginners class are a merry bunch. Before our first class, I was extremely anxious, trying to memory numerals in Japanese from Wikipedia and worried whether I’ll be able to keep up with everyone. There was help every step of the way. The class won’t stop for you, but it will not leave you behind either :)

But the best part of SFTD is the opportunity to learn from Sensei Tanaka. His accomplishments in the world of Taiko are well known, so I won’t enlist them here. What surprised me was the humility and generosity and the energy he brings with him, even after doing this for more than four decades. He could easily delegate the adult beginners class to one of his many advanced students; yet he still routinely teaches the class himself, ever so patient and understanding. Better yet, his expertise in Taiko is matched only by his wistful humor.

So if you are in the Bay Area and are looking for some inspiration, do checkout San Francisco Taiko Dojo.

Related posts:

1. Test post using ScribeFire (please ignore)
2. Happy Republic Day
3. History of Computing

var sys = require('sys');
var http = require('http');
var url = require('url');
var path = require('path');

function search() {
  stdin = process.openStdin();
  stdin.setEncoding('utf8');
  stdin.on('data', function(term) {
    term = term.substring(0, term.length - 1);
    var google = http.createClient(80, 'ajax.googleapis.com');
    var search_url = "/ajax/services/search/web?v=1.0&q=" + term;
    var request = google.request('GET', search_url, {
      'host': 'ajax.googleapis.com',
      'Referer': 'http://floatingsun.net',
      'User-Agent': 'NodeJS HTTP client',
      'Accept': '*/*'});
    request.on('response', function(response) {
      response.setEncoding('utf8');
      var body = ""
      response.on('data', function(chunk) {
        body += chunk;
      });
      response.on('end', function() {
        var searchResults = JSON.parse(body);
        var results = searchResults["responseData"]["results"];
        for (var i = 0; i < results.length; i++) {
          console.log(results[i]["url"]);
        }
      });
    });
    request.end();
  });
}

search();

This program (also available as a gist) reads in search terms on standard input, and does a Google search on those terms, printing the URLs of the search results.

I was quite surprised (and a bit embarrassed) at how long it took me to get this simple program working. For instance, it took me the better part of an hour to realize that when I read something from stdin, it includes the trailing newline (as the user hits ‘Enter’). Earlier, I was using the input as-is for the search term, and that was leading to a 404 error, because the resulting URL was malformed.

Debugging was also harder, as expected. Syntax errors are easily caught by V8, but everything else is still obscure. I’m sure some of the difficulty is because of my lack of expertise with Javascript. But at one point, I got this error:

events:12
        throw arguments[1];
                       ^
Error: Parse Error
    at Client.ondata (http:881:22)
    at IOWatcher.callback (net:517:29)
    at node.js:270:9

I still haven’t figured out exactly where that error was coming from. Nonetheless, it was an interesting exercise. I’m looking forward to writing some non-trivial code with node.js now.

Related posts:

1. What is node.js?
2. Ajax galore!
3. Google Reader archiving less?

dbShards

As the name suggests, dbShards is all about sharding. Sharding, also known as partitioning, is the process of distributing a given dataset into smaller chunks based some policy. AFAIK, the term “shard” was popularized recently by Google even though the concept of partitioning is at least a few decades old. Most distributed data management systems implement some form of sharding by necessity, since the entire data set will not fit in memory on a single node (if it would, you should not be using a distributed system). And therein lies the USP of dbShards — it brings sharding (and with it, performance and scalability) to commodity, single-node databases such as MySQL and Postgres.

So how does it work? Well, dbShards acts as a transparent layer sitting in front of multiple nodes running MySQL, lets say. Transparent, because they want to work with legacy code, meaning no or minimal client side modifications. Inserting new data is pretty simple: dbShards using a “sharding key” to route an incoming tuple to the appropriate destination. Queries are a bit more complex, and here the website is skimpy on details. Monash’s post mentions that join performance is good when sharding keys are the same — this is not a surprise. I’m not interested in what other kinds of query optimizations are in place. When data is partitioned, you really need a sophisticated query planner and optimizer that can minimize data movement and aggregation, and push down as much computation as possible to individual nodes.

I found the page on replication intriguing. I’m guessing when they say “reliable replication”, they mean “consistent replication” in more common parlance (alternative, that dbShards supports strong consistency, as opposed to eventual or lazy consistency). This particular bit in the first paragraph caught my eye: “deliver high performance yet reliable multi-threaded replication for High Availability (HA)”. I’m not sure how to read this. Are they implying that multi-threaded replication is typically not performant? And usually you do NOT want threading for high availability, because a single thread can still take the entire process down. The actual mechanism for replication seems like a straightforward application of the replicated state machine approach.

But making a replicated state machine based system scale requires very careful engineering, otherwise it is easy to hit performance bottlenecks. I’d be very interested in knowing a bit more about the transaction model in dbShards and how it performs on larger systems (tens to hundreds of nodes).

The pricing model is also quite interesting. I think it is the first vendor I know of that is pricing on CPU and not storage (their pricing is $5,000 per year per server). I think this is indicative of the target customer segment as well — I would imagine dbShards works well with a few TBs of data on machines with a lot of CPU and memory.

Related posts:

1. What is node.js?
2. Thoughts on the Rupee symbol
3. Toying with node.js

Udaan is one of those rare movies where it seems like the crew had an intense clarity about the movie they wanted to make, and that is exactly what they did. They did not make it for the money, they did not make it to please a broad audience, they did not make it to please the critics — they made it, because that was what they wanted to show.

I’m not going to talk about the story or the characters much, just Google those things if you are interested. Instead, I want to talk about an analogy.

Any good designer knows the importance of whitespace, whether in layout or typography. Architects have long understood that negative or empty space is just as (or perhaps more) important as filled space. Watching Udaan was a good reminder that good moments in a movie need their space as well.

I didn’t feel rushed as I saw the movie; it felt a bit slow at times, but there was no hurry to get to the end. There are several scenes that are made poignant by the lack of dialog. The same goes for the music. Amit Trivedi has done an outstanding job with the background score as well as the soundtrack. The lyrics (by Amitabh Bhattacharya) are fabulous and are fittingly given their space in the songs — Amit makes sure that the music recedes and does not overwhelm so you can pay attention to the words. But when the voices take a break, the music that fills in the gaps is just as good.

As my wife observed, “this movie has craft.”

Related posts:

1. Movies: H2G2, KFH
2. The Good, The Bad and The Ugly
3. What are movies for?

Courtesy http://www.flickr.com/photos/ericrobinson/

A quick Google search on “how to buy a new car” led me straight to the very comprehensive CarBuyingTips.com. It is probably a great resource for many people. But after spending a few hours clicking through the numerous links on there, I almost felt exhausted. There was way too much (redundant) information, perhaps badly organized and overall just not very easy to consume. As they say, hindsight is 20/20. So, with the experience of having just purchased a new vehicle, here is my attempt at a concise, five step guide to buying a new car.

1. Figure out what you want: You should know exactly what make and model you want, down to the last detail — this includes the interior color, upholstery and exterior color, as well as any other options and accessories. The more precise you are in what you want, the better off you will be. My first impression while researching new cars was that I could get whatever configuration I wanted — if the dealership doesn’t have it in stock, they’ll simply order it in. Unfortunately, most dealers will only work with what stock they have. So checking for availability is critical. Go ahead and schedule those test drives, but let the dealers know upfront that you are not looking to buy just yet. The dealers will ask for your contact information though, so be prepare for a barrage of emails and phone calls from them, until you’ve made your purchase.
2. Get the numbers: Once you have identified the configuration you want, find your car on Edmunds.com. Edmunds will give you the invoice price of your car. Go ahead and add all the options and select the colors to get a final estimated invoice price. The more informed you are, the better your chances are when negotiating with dealers and making an informed decision.
3. Get a quote from CarsDirect: Buying a car online these days is not only possible, but highly recommended. You save the hassle of driving to dealerships, wasting time over the phone etc. Start your hunt for the best price by getting a quote from carsdirect.com. They partner with local dealerships and have very competitive pricing. My experience with CarsDirect was fantastic and I’d have definitely bought a car from them had a local dealer not given me a much better deal.
4. Get quotes from local dealers: Open a spreadsheet, fire up your browser and start calling your local dealerships. Ask for the new car sales department and let them know exactly what configuration you are looking for. Ask them for price and availability. Always ask for out-of-the-door price, including taxes and rebates. This way there will be fewer surprises on the final bill. Make a point to let them know that you are talking to other dealers. Jot down the dealers quote in the spreadsheet (add the CarsDirect quote here as well). This process can take some time because you may not be able to reach them in the first attempt and there might be some back and forth while they get back to you with details. I recommend setting aside 2 slots of 2 hrs each for these phone calls.
5. Decide and Buy: Once you have all competing quotes, you can make your decision. The final decision will probably depend not just on the price, but other factors such as availability, location of the dealership, your experience with the dealership etc. If you finance your car, most car companies typically have their own financing arm which usually provides great APRs. If not, talk to your bank. For the final paperwork, you should make a visit to the dealership. Be sure to read the fine print and know exactly what service (if any) the dealer will provide above and beyond the warranty and services provided by the manufacturer.

That’s pretty much it! I read a lot of horror stories online about swindling and cheating in dealerships. My personal experience with at least the Toyota dealers in the Bay Area was pretty good. Most of them were very straightforward and to the point. They did not want to waste their time or mine, and did not try to pressurize or hoodwink me into a bad deal. You might also find this guide useful.

Related posts:

1. IBNLive : Decision on quota is final: Arjun

If you follow the world of Javascript and/or high-performance networking, you have probably heard of node.js. If you already grok Node, then this post is not for you; move along. If, however, you are a bit confused as to exactly what Node.js is and how it works, then you should read on.

The node.js website doesn’t mince words in describing the software: “Evented I/O for V8 JavaScript.” While that statement is precise and captures the essence of node.js succinctly, at first glance it did not tell me much about node.js. I did what anyone interested in node.js should do: downloaded the source and started playing around with it.

So what exactly is node.js? Well, first and foremost it is a Javascript runtime. Think of your web browser; how does it run Javascript? It implements a Javascript runtime and supports APIs that make sense in the browser such as DOM manipulation etc. Javascript as a language itself is fairly browser agnostic. So node.js is yet another runtime for Javascript, implemented primarily in C++.

Because node.js focuses on networking, it does not support the standard APIs available in a browser. Instead, it provides a different set of APIs (with fantastic documentation). Thus, for instance, HTTP support is built into node.js — it is not an external library.

The other salient feature of node.js is that it is event driven. If you are familiar with event driven programming (ala Python Twisted, Ruby’s Event Machine, the event loop in Qt etc), you know what I’m talking about. The key difference though is that unlike all these systems, you never explicitly invoke a blocking call to start the event loop — node.js automatically enters the event loop as soon as it has finished loaded the program. A corollary is that you can only write event driven programs in node.js, no other programming models are supported. Another consequence of this design choice is that node.js is single-threaded. To exploit CPU parallelism, you need to run multiple node.js instances. Of course, there are several node.js modules and projects already available to address this very issue.

To implement a runtime for Javascript, node.js first needs to parse the input Javascript. node.js leverages Google’s V8 Javascript engine to do this. V8 takes care of interpreting the Javascript so node.js need not worry about syntactical issues; it only need to implement the appropriate hooks and callbacks for V8.

node.js claims to be extremely memory efficient and scalable. This is possible because node.js does not expose any blocking APIs. As a result, the program is completely callback driven. Of course, any kind of I/O (disk or network) will eventually block. node.js does all blocking I/O in an internal thread pool — thus even though the application executes in a single thread, internally there are multiple threads that node.js manages.

Overall, node.js is very refreshing. The community seems great and there is a lot of buzz around the project right now, with some big companies like Yahoo starting to use experiment with node.js. node.js is also driving the “server side Javascript” movement. For instance, Joyent’s Smart platform allows you to write your server code in Javascript, which they can then execute on their hosted platforms.

Finally, no blog post about node.js is complete without an example of node.js code. Here is a simple web server:

Related posts:

1. Toying with node.js
2. Some thoughts on dbShards
3. Browsers are the new cross platform GUI tookit

I was quite thrilled when I found out that SIGCOMM 2010 is going to be in Delhi this year!! While some of these conferences are known to pick “exotic” venues, it is also an encouraging nod from the academic community. This is probably the first top-tier systems/networking conference to be held in India and I hope the local universities in and around Delhi will take advantage of this opportunity.

It is shaping up to be an exciting year for Delhi, with the Commonwealth Games coming up soon after SIGCOMM. A glance at the organizing committee for this year suggests that Geoff Voelker might have been involved in pushing for this venue :)

I personally am not very fond of Delhi, however. Out of curiosity, I looked over the local information page on the SIGCOMM 2010 web site and found myself both agreeing with and a bit disappointed at some of the tips listed on that page:

• Keep your wallet/passports in an inner pocket of a jacket or shirt. Avoid keeping it in the rear pocket, especially while moving around in crowded places or in public transport like buses or Delhi Metro.
• Never follow the advice of taxi or cab drivers regarding your stay and travel in the city. Always take assistance from “May I Help You” counter and other assistance cell of the government like Delhi Tourism, DTC, etc.
• Don’t travel alone late nights, especially female travelers. If you are getting late, take proper private cab or arrange a pick up

I do hope all visitors to SIGCOMM have a fantastic stay and that more and more conferences choose India.

Related posts:

1. On blind reviews
2. DEMO.com Conferences > Demofall 2006
3. India, here I come!

Here’s a simple snippet to add Webfonts support to your Thematic child themes. Just add it to the functions.php:

Related posts:

1. Thematic: the final word in WordPress themes
2. Amarok ate your stats?
3. WP-Dokuwiki 0.2

I had previously written about the Rupee symbol here. The ToI seems to suggest that #4 is the likely candidate. According to the article, the symbol should have been decided by now, but I haven’t found any information to that effect. Does anyone know the final outcome? I can’t say I like any of the finalists that much, but I guess out of the five #4 is probably one of the better ones.

Related posts:

1. Thoughts on the Rupee symbol

Some observations:

• If you have not filed your tax return for years, the IRS should be coming after you and auditing you! The IRS should relentlessly pursue you for your unpaid taxes.
• I’m sure there are a lot of folks out there who are hurting and are not able to pay their taxes for legitimate reasons. But the ad does not convey that at all. It almost sounds like “Hey, I’m going to tell you how you can get away from the IRS and not pay your taxes”
• Everyone knows how targeted television advertising is. For instance, if you watch Comedy Central, you’ll see a lot of ads around dating websites, calling services for singles etc. I see the the TaxMasters ad on CNN all the time. Wonder what that says about CNN’s audience.
• This ad sucks because it doesn’t really engage the audience, the value proposition is not clear to me at least and there is absolutely no creativity.
• But I gotta hand it to TaxMasters — I was pleasantly surprised to discover their blog and their sense of humor.

More to come.

Related posts:

1. How is TV viewership measured?

In March 2009, the Indian government (specifically, the Finance Ministry) announced a contest to design a symbol for the Indian Rupee. Sometime in April, the Ministry put out a press release listing all the eligible applicants; there were around 2300 eligible candidates it seems.

At some point after that, images of a few of the designs started surfacing:

Rupee Symbols

As is the case with most Indian Government websites, the Financial Ministry website is a disaster. There is very little useful information there, there is no way to search for information. Case in point — I was not able to find any information about the design contest on their website. The image above is the result of a Google search.

Couple of thoughts on the designs above (note that I do not know if these are even actual candidates. I’m assuming they are):

• It isn’t entirely clear to me why we need a symbol in the first place. Sure, writing ‘$’ is probably nicer than writing ‘USD’, but ‘Rs’ isn’t all that bad.
• Any symbol for a currency should be really simple to draw. Simple. You should be able to draw it by hand in a few strokes. How many of the above designs do you find that simple?
• Even if we pick a symbol, for it to actually start getting used, it has to be readily available on all computing platforms. Does Unicode have a provision for adding new symbols?

In December 2009, the Economic Times reported that the Ministry had shortlisted five finalists. Really? Wow. Again, no information to be found from the Ministry itself. It would have been pretty amazing (and easy to set up) if the Ministry had set up a public poll and asked Indian citizens which symbol they liked best.

Does anyone know what happened to the Indian rupee symbol design contest? I couldn’t find anything on Google after December 2009.

Related posts:

1. Rupee symbol finalists
2. Conscientious Bollywood
3. Some thoughts on dbShards

Such a simple app, but such a fascinating output. An easy way to create computer generated art! Couple of observations:

• I have a dual-monitor setup at work. I use the left monitor for email for browsing and the right monitor for code. The mouse patterns clearly reflect this usage pattern. I tend to rest the mouse roughly equally on the both the monitors.
• I was very intrigued by the fact that most of the mouse motions are very smooth. Most curves almost look parabolic. There are very few jerks and jittery lines. Once again, nature seems poetic even in the most chaotic and random actions.

Related posts:

1. How to simulate Compiz’s screenshot plugin in KDE4
2. Andrew Morton moves to Google
3. Apple’s new toys

Photo by http://www.anna-om-line.com/

In general, I find myself on (long) email threads containing a different subset of people for different occasions (birthdays, anniversaries etc) several times a year. The email threads quickly become long and unwieldy. People keep adding other people as the thread progresses, and the only way the new adds can figure out whats going on is looking at the content of future emails. There is no way for anyone to go back and read all the discussion so far.

That got me thinking, wouldn’t it be great to have a service that provide throw away mailing lists? Hear me out. Here’s how the service would work:

• To start a new mailing list, I simply send an email to newlist@mycoolservice.com. In the email, I also include a list of email addresses I want to seed the list with.
• The service sends me back the address of a newly created throw away list. This could be of the form some-random-number@googlegroups.com.
• For all practical purposes, this is exactly like any other mailing list (or Google Group). We can add more members, search the messages etc.
• Start your discussion and let the thoughts flow.
• …
• At some point, the purpose behind the list will cease to exist (successful surprise, for instance). Needless to add, further discussions on the topic will also cease.
• You forget you even created this mailing list. After the mailing list has been idle for some time (say two weeks), the service automatically deletes the mailing list. Any future messages to that address will bounce back saying that the list has been deleted, please contact the administrator.

Does anyone else think this could be useful?

Related posts:

1. Faking from address
2. Unsung heroes of the FLOSS world
3. A tale of status messages

First, I’m really happy to see Google at least trying to become more transparent. Second, I was very intrigued to see that India is in the top 3! Just in the last 6 months of 2009 India made over 1000 data requests to Google and upwards of 140 removal requests. Here’s the current breakdown:

• 1061 data requests
• 142 removal requests
  • 77.5% of removal requests fully or partially complied with.
  • 2 Blogger
  • 1 Book Search (court order)
  • 2 Geo (except Street View)
  • 119 orkut
  • 1 SMS Channels
  • 2 Web Search
  • 15 YouTube

Thankfully, bulk of the removal requests (119 out of 142) are for Orkut and given the spammy state of Orkut, I’m not really surprised. I’m more interested in the data requests, but right now that (possibly inaccurate) number is all Google can tell us.

Here’s another weird thing, Brazil made 291 removal requests of which 119 were for Orkut. India also made 119. Co-incidence? Perhaps. Or may be it was a common set of bad URLs that both countries ended up getting rid of. Of course, more than 75% of Orkut users are either from Brazil or from India, so there’s definitely some connection here.

All in all, another interesting data source.

Related posts:

1. Business networking
2. How Google killed Urchin
3. Google Analytics — the deep web

Groovy Gurus is a bunch of gastronomically inclined city dwellers (and then some) who’ve chosen to spend a couple of Thursdays each month – the 2nd and 4th to be precise – dining at a yet-undiscovered venue (at least for the purposes of the Thursday dinners), picked by a rotating designatory in the group. San Francisco has such a wide range of cuisines served up by restaurants that often reflect the unique character of their neighborhoods that it would be an opportunity lost to not give the lot of them a try.

Tonight is the third GG meetup. Follow our journey on Groovy Gurus.

Meanwhile, here are some of the other names we thought of before settling on Groovy Gurus:

• Tangy Tuesday
• Water, no ice
• Masala Chai
• Tuesday Tarkari
• Mangal Munch
• The Fortnight Bite
• स्वाद की बहार, महीने में दो बार

Related posts:

1. Forrest Tuesday
2. What is with sororities?
3. Monsoon mayhem

First, I was a bit surprised and disappointed that the top hit for a Google search for “india census 2011″ does not point to an Indian government website for the census. Instead, most of the top hits are from various news stories. An interesting aside here — check out the same search carried on google.co.in; the results are quite different. In fact, I found it quite interesting that the censusindia.gov website is the top hit for the US search, but not in the India search.

Second, the census website is pretty shabby. Forget about SEO, typography, accessibility — the folks who are building these websites are still living in 1999. Someone needs to tell them about good web design. Design and usability issues apart, the content itself is lacking in many places; even incorrect in some places. Consider this page:

• the breadcrumb style navigation is only for appearances — the links don’t really work
• It says “Census 2001″!!!!
• the <title> anchor says “Evolution of Data Processing”. There’s no indication of the title on the page itself. Instead, it says “Preparation for the 2011 census.”
• Best viewed in 1024×768? Wow.
• None of the foreign language links at the bottom of the page work. It would be trivial to link to a Google-translated page.
• The header image or text don’t link to the Census home page. This is really basic stuff. I wonder if any of the developers building the website have actually used it.
• Most of the links on the Census home page are “under construction.”

Anyways, I don’t want to waste this post on nitpicking. After all, the issues at stake here are considerably more serious than web design. Without further ado, here are the key facts:

• This is the 15th census. The first one took place in 1872.
• I couldn’t get any information about how the previous censuses were conducted, but this time around the Indian Government is aiming to record each and every one of the 1.2 billion individuals. Thats a really lofty goal.
• This Census is also being used to piggy back another significant project: the construction of a National Public Registry (NPR). The NPR will contain some key stats about everyone in India. The NPR will also be used to generate biometric IDs for all Indians at some point.
• For the first time in Census history, some NGOs will be involved in the training phase.
• A lot of geo-spatialization technologies are in play. The only article I could find was this.
• Another first: this Census will be collecting some data on basic sanitation, drinking water etc.
• The 2001 census employed roughly 2 million “enumerators” (data collection personnel). This census will undoubtedly require more.

The FAQ has several interesting nuggets:

The information collected about individuals is kept absolutely confidential. In fact this information is not accessible even to Courts of law.

Once this database has been created, biometrics such as photograph, 10 fingerprints and probably Iris information will be added for all persons aged 15 years and above.

The issue of Cards will be done in Coastal Villages to start with. After this the coastal Towns will be covered and so on till the entire country is covered.

35 questions relating to Building material, Use of Houses, Drinking water, Availability and type of latrines, Electricity, possession of assets etc. will be canvassed.

Like I said earlier, these are lofty goals and this is a massive effort. But there’s so little information out there. I have a lot of questions and concerns. For one, the construction of the NPR poses a huge privacy risk. How is the Government going to ensure that corrupt officials don’t sell out the data to whoever pays the right price? What kind of audit policies and procedures are in place such that a neutral third-party can attest to the integrity of the entire process? Who all within the government will be able to look up my data? If the Government is going to store the biometric data as well, aren’t they constructing a gold-mine of identity-theft ammunition?

Digging around for more information, I found this fascination presentation (linked off of the United Nations Statistics Division). The PPT claims that it was prepared by someone from within the Census organization, but I can’t vouch for its authenticity.

I really hope that a lot more information on the technology behind the Census comes out in the coming months. The news channels in India should take some time out from reporting on tabloid/gossip issues and focus some energy on something that will impact every single Indian.

Related posts:

1. India blocks blogs?
2. Thoughts on the Rupee symbol
3. Bombay blasts

I really like the tagline on our logo: big data. fast insights.

But leaving the marketing aside, what does it mean really? What is all the hoopla about big data analytics?

The way I look at things, a few key observations here are:

1. Data is increasing. This is almost self-evident, so I won’t bother with presenting any evidence.
2. Data is driving businesses more than ever. Whether it is search, advertising, insurance, finance, health care, governance — data is becoming an integral part of more and more business processes.
3. Finally, data movement is slow. And I mean really really slow, compared to our processing and memory speeds. Once you go into the range of hundreds of terabytes of petabytes of data, you really don’t want to keep moving around that data into isolated silos for doing analytics.

Clearly, none of these observations is particularly new or insightful. However, I do think some of the implications of these observations are quite powerful and were new at least for me. For instance, (3) implies that once you have accumulated a lot of data in one place (imagine hundreds of TB or more), it is extremely difficult and time consuming to move that data around. This, in turn, means that more often than not, data is likely to reside in a single place.

Traditionally, it was not uncommon to have a large data warehouse that would be the repository of all data. Then smaller data sets could be carved out from this master data set (also known as data marts) as required. This approach is becoming increasingly unfeasible. Carving out 100TB data marts from a 1PB data warehouse is simply not going to scale.

At the same time, it is clear that a one-size-fits-all approach to data storage and analysis is not practical either. Some data sets naturally lend themselves to a relational data model, while others might be more suited to unstructured processing (Hadoop) or document oriented processing (CouchDB or MarkLogic) or graph analysis (Neo4J) and so on. Forcing a single model or access mechanism down all customers’ throat is not tenable.

So what would the ideal platform for big data analytics look like? One that allows you to store and access data in various ways, seamlessly.

Related posts:

1. Google Analytics — the deep web
2. Interesting Analytics
3. Big Data Summit

Image via Wikipedia

I’m a Linux guy; Windows was never my thing honey
Apple seemed interesting, but required too much money

I have ideological problems with Apple too,
What with all the DRM and hardware lock-in they do.

But people are crazy about Apple, and I used to wonder why,
I had a dream: to own Apple products that I didn’t have to buy.

A few months back my wife gifted me an iPhone, bro!
And then at work I got the new Macbook Pro!!

Thus suddenly I was an Apple user,
Sure, some people called me a sore loser.

Allow me to share my early experiences,
Some accolades and some grievances.

I’ll try to keep a neutral tone,
Shall focus on the Mac and not the iPhone.

Integration, integration, integration!
The attention to detail gives a wonderful sensation.

User experience is the key,
Excellent design is for all to see.

They’ve taken care of the enterprises,
Exchange support, Google integration — no surprises.

It’s by far the best laptop I’ve ever used,
The hardware is slick, the software is smooth.

Image via CrunchBase

But boy do I hate iTunes,
It’s so broken it should be called Looney Tunes.

Try connecting multiple iPhones to the same device,
Or plug your iPhone in another laptop (poor advice).

Sync is threatening, sounds like a bully.
“I shall sync or destroy”, that just sounds silly.

The Terminal app should aspire higher,
No 256-color support leaves much to desire.

Keyboard shortcuts are hard to find,
Change them? you must be out of your mind!

“Features” like “Spaces” are overrated,
More like awaited, belated and deflated.

I prefer iTerm over Terminal and Adium for chat,
Chrome over Safari, and this over that.

I’m certainly not blown away,
But a Mac is convenient, I have to say.

Related posts:

1. To Apple or not to Apple
2. Big Brother Apple
3. Horse Drawn Carriage

But the Make model is rife with problems. Here are a few of them:

• Make requires the use of its own domain specific language — this is, in general, not a good idea. Have you looked at any sizable project’s Makefile lately? Its hard to understand, and harder to modify.
• In the same vein, autoconf/automake are notoriously hard to use. Bear in mind that these tools are supposed to make your life easier.
• Makefile are so hard to write and extend that several popular build systems today are essentially Makefile generators. A good example is CMake.
• Make relies heavily on file timestamps to detect changes.
• Make is slow.
• Makefile are not modular. Recursive Make is especially evil.
• …

I recently began work on a new pet project. As is usually the case, I spent a lot more time figuring out what tools and libraries I would use for my project, than in actually writing any code for the project :) Part of the investigation was to survey the state of the art in build systems. At work, we started using SCons for most of our build, which was already a huge improvement over Make. But SCons has its own set of issues.

One of the nicest features in SCons is that build files are regular Python files. This provides enormous flexibility and immediate familiarity. Unfortunately, the SCons documentation leaves much to be desired. I still don’t quite understand the execution model of SCons very well. For instance, I know how to extend SCons to support cross-compilation for multiple platforms. However, I don’t really understand why those modifications work — there’s quite a bit of black magic that goes on behind the scenes. As a concrete example, there are several magic variables such _LIBDIRFLAGS that have strange powers.

Waf

After some more looking around, I discovered Waf. And now that I’ve played around with it a little bit, I’m happy to say that it is the most pleasant build system I’ve ever used. Things I really like about Waf:

• The execution model just makes sense to me. You typically build a project in phases: there’s a configure phase, to sort out dependencies, tools etc; there’s the actual build phase; and then there’s the install phase. It is not uncommon to have a ‘dist’ step as well, to prepare the source for distribution. Waf understands these operations as first class entities. There is a very strong notion of workflow built into Waf.
• Comprehensive documentation. Check out the Waf book and the wiki.
• Waf has a very strong task model. There is a much stronger notion of dependencies (powered by content hashes, not timestamps). Waf also enforces that all generated code lands up in a separate “build” directory, so your source tree always remains clean.
• Using waf is a breeze — there are no big dependencies, no packages to install, no bloated software to include with your code. Just a single 80kb script.
• Progress indication and colored output is built in, not an after thought. Like SCons, Waf build files are regular Python files.
• Waf is fast. Faster than SCons.

Of course, Waf is not perfect. Coming from a Make/SCons world, I sorely miss the ability to build specific targets. Yes there are ways to achieve this in Waf, but they are all clumsy. The API documentation (and the source itself) are a bit hard to parse.

Related posts:

1. gooLego: Google’s software building blocks
2. darcs status?
3. Experiences with Google App Engine

I thought I had attained screen nirvana… until I found tmux (hat tip xed). So what exactly is tmux?

tmux is intended to be a modern, BSD-licensed alternative to programs such as GNU screen. Major features include:

• A powerful, consistent, well-documented and easily scriptable command interface.
• A window may be split horizontally and vertically into panes.
• Panes can be freely moved and resized, or arranged into one of four preset layouts.
• Support for UTF-8 and 256-colour terminals.
• Copy and paste with multiple buffers.
• Interactive menus to select windows, sessions or clients.
• Change the current window by searching for text in the target.
• Terminal locking, manually or after a timeout.
• A clean, easily extended, BSD-licensed codebase, under active development.

And how is tmux better than screen? Thats question #1 in the FAQ:

tmux offers several advantages over screen:

- a clearly-defined client-server model: windows are independent entities which may be attached simultaneously to multiple sessions and viewed from multiple clients (terminals), as well as moved freely between sessions within the same tmux server;

- a consistent, well-documented command interface, with the same syntax whether used interactively, as a key binding, or from the shell;

- easily scriptable from the shell;

- multiple paste buffers;

- choice of vi or emacs key layouts;

- an option to limit the window size;

- a more usable status line syntax, with the ability to display the first line of output of a specific command;

- a cleaner, modern, easily extended, BSD-licensed codebase.

There are still a few features screen includes that tmux omits:

- builtin serial and telnet support; this is bloat and is unlikely to be added to tmux;

- wider platform support, for example IRIX and HP-UX, and for odd terminals.

Related posts:

1. Tools I use: screen
2. Center screen on current line
3. darcs status?