Floating Sun » google

gooLego: Google’s software building blocks

Diwaker Gupta — Fri, 25 Dec 2009 05:00:31 +0000

: Image via Wikipedia

Over the past few years, Google has open sourced several projects that provide some commonly used building blocks in any large software project. Some of them I was aware of since when they were launched (like protobufs), while others I discovered only recently. I couldn’t find any location where all the projects were listed together and combing through Google Code looking for them was painful, so I’m putting together a list myself. Hope some of you find it useful.

protobufs: Platform agnostic messages. Critical for any distributed system. Note that protobufs only provide message serialization/deserialization (for various languages). An important missing piece is an RPC framework built on top of them. There are several projects attempting to build one using protobufs, but none of them are robust or mature enough for production use.
style guide: The importance of a style guide is probably understated. It is not about what is the “right” style — it is about consistency. While people may have different opinions, if everyone follows the same style, the code becomes much more readable and maintainable. Google maintains style guides for C++ and Python.
config flags: Another important building block for all command line programs.
logging: Self-evident. Google’s logging library supports various log levels and other useful macros.
core dumper: A very nifty library — it allows you to dump core from within a running application. Extremely useful for debugging production systems.
perftools: An extremely useful library for measuring and monitoring performance of programs. By simply linking against perftools, your application gets a much better malloc, heap checking, visual CPU profile of various routines (via graphviz), visualization of memory usage etc.
googlemock: A framework to quickly build mock objects — useful for testing.
googletest: Google’s C++ unit testing framework, built on top of xUnit. Integrates well with googlemock.

Of course, this is not an exhaustive list. There are numerous other open source projects from Google, some of them probably much more bigger and visible than the ones listed above — such as Wave, Go, GWT etc. If there’s a project that is a software building block that I missed out, do chime in the comments below.

Google (Contacts, Mail, Talk) confusion

Diwaker Gupta — Tue, 11 Aug 2009 22:07:08 +0000

: Image via CrunchBase

When Gmail first introduced the ability to import Contacts, I prompty exported my addressbook from KAddressbook. And then I mostly forgot about it, until recently. In the meantime, Google happily kept adding “suggested” contacts to my addressbook.

I decided to revisit my Google Contacts after reading some blog posts about new functionality. Sure enough, Contacts now even has its own URL (google.com/contacts). I figured this was a good time to clean out the contact and start from scratch with a clean list not polluted by the automatic suggested contacts. So I went ahead and deleted all the contacts and re-imported them from my desktop address book.

Surprisingly, there are weird interactions between my Google Contacts, and my Google Talk buddy list. A lot of people on my buddy list silently disappeared, without any kind of message or confirmation from either GMail, Talk or Contacts. And since then, my attempts to add back all the deleted buddies has failed miserably. Every time I add someone to my list, they show up just fine, but if I log out and log back in, they are usually not there.

What is even worse, this behavior is non-determinstic. Some additions persist across multiple sessions, while others are more ephemeral. I still don’t know exactly what the interaction between these three properties is, but it is very confusing. Google should clarify this more — what exactly is the impact of modifying my Contacts on things like Google Talk etc?

GAFYD slowness

Diwaker Gupta — Tue, 31 Mar 2009 05:18:11 +0000

I have been using Google Apps For Your Domain (GAFYD) for my floatingsun.net email for a while now (earlier I was using the email setup at my hosting provider, but moved away because of the lack of adequate spam filtering). In the beginning, it was just a joy and everything was nice and peachy.

However, over time, the service has been gradually deteriorating. Since the past few weeks I have noticed a significant increase in latency. Meaning that if I open mail.google.com side-by-side with mail.google.com/a/floatingsun.net, my “regular” gmail account loads up much, much faster than my floatingsun.net account. This despite the fact that my regular gmail account has at least 100x the messages on my floatingsun.net account. And in fact, there have been several occassions in the recent past where it doesn’t load at all, or fails with a server error.

Not to mention that IMAP access has been horrendous recently. Throughout the day, email takes forever to open in my IMAP client (it opens up relatively faster on the web interface) and I get frequent disconnections from the server.

I don’t mean to beat up on Google. I must admit that I am on the free plan, so I really have no reason to complain. But gmail is also free. I have a feeling Google is deliberately imposing some kind of quality-of-service differentiation between paid Google Apps accounts vs free ones. I am keeping an eye on the status dashboard — it says no issues but my IMAP is still flaky. Is anyone else seeing poor performance on free GAFYD accounts?

Google Chrome

Diwaker Gupta — Mon, 01 Sep 2008 17:52:44 +0000

Its all over the blogosphere anyways: http://blogoscoped.com/google-chrome/ Related posts: Google in talks to buy YouTube Google Page Creator Google buys Writely! Related posts:

uBoggle is a featured application!

Diwaker Gupta — Fri, 15 Aug 2008 21:08:42 +0000

[[http://uboggle.appspot.com|uBoggle]] is now appearing as the **featured application** on the [[http://appengine.google.com|Google AppEngine]] [[http://appgallery.appspot.com|Application Gallery]]. Thank you for your votes, and thanks to the appgallery editors!

And I have a screenshot for proof and posterity :)

uBoggle is the featured application on App Gallery

Experiences with Google App Engine

Diwaker Gupta — Tue, 15 Jul 2008 01:53:36 +0000

I’ve been playing around with [[http://appengine.google.com|Google AppEngine]] for the past two weeks, and the experience has been mixed so far. First, the good:

* really easy to build something simple and get started.
* no need to worry about scaling, backup, replication etc. I haven’t verified this obviously, but at least thats the claim.
* the integration with Google accounts is nice.
* good documentation, lots of sample code available.
* dev server really helps with most of the development.
* the sort of restrictive resource usage limits (see below) forced us to think carefully about our code and heavily optimize certain operations to make them work on GAE.

{{ http://code.google.com/appengine/images/appengine_lowres.jpg}}

And now, the bad:
* too many limits: 1 million is their favorite number. No files over 1MB, no request should take more than 1 million CPU cycles (whatever that means) and who knows what other limits they impose internally. While developing, this was the biggest barrier for us. Things would randomly fail, and then our application would be disabled for several hours.
* The dev server doesn’t replicate the constraints in production. So everything would run fine and dandy locally, and the minute we upload, it would fail. Since we can only debug in production, and our application exceeded the quota every time we ran it, debugging was extremely slow and painful.
* local data store is excruciatingly slow. But this is not that critical, since it is only for testing anyways.
* even the remote data store is very flaky and slow at times. Any query involving more than a few hundred elements exceeds the quota.
* the bulk uploader is very useful, but again it is really really slow. If you want to upload anything in “bulk”, you’ll have a hard time. The parameters have to be chosen carefully as well. Even for very simple data models involving 3-5 fields (mostly strings), we had to reduce the batch size to 2-4 to make it work. And despite that we got a few HTTP 500 errors while uploading.

But its been fun so far. Hopefully most of these issues will get ironed out moving forward. As for what we are building? That will have to wait for another post ;-)

Google Reader auto sort

Diwaker Gupta — Tue, 22 Apr 2008 22:32:36 +0000

[[http://reader.google.com|Google Reader]] offers several options for [[http://www.google.com/support/reader/bin/answer.py?answer=69980&topic=12012|sorting feed items]]. After having played around with the “auto-sort” for several months now, I am reverting back to “Sort by newest”.

{{ http://www.google.com/googlereader/images/logo_reader.gif|Google Reader}}

The problem is that the auto-sort mode is a little too simplistic. Here’s what it does in their own words:

//This works by prioritizing subscriptions with fewer items. So, with this setting, your friend’s blog with one item a month will not be drowned out by higher volume sites such as the New York Times because we’ll raise the blog to the top.//

The general idea behind auto-sort is good, but unfortunately the execution hasn’t evolved at all to become smarter. For instance, some blogs I read haven’t been updated in a while. And I’m really not interested in the stuff they wrote some months back. So I never read those few old posts and yet they continue to hang around at the top of my feeds, which gets annoying quickly.

Ideally, the auto-sort should also take into account my reading trends (they obviously collect all this data, so might as well use it). In my case, what I really want the auto-sort to do is this: if there are some old posts and I’m consistently choosing not to read them, then perhaps they don’t need to be raised to the top any more. If I need to find them, I can always do so. In fact, I wouldn’t even mind if the old posts were raised to the top of the list once in a while.

An even smarter auto-sort will also take into account my reading habits. If there’s an infrequently updated blog that I read religiously, then I definitely don’t want to miss even an old post, no matter what. Similarly, old posts from an inactive blog that I have stopped following should be given less weight.

How do you sort your feeds?

Come on Yahoo! dikha de!!

Diwaker Gupta — Tue, 26 Feb 2008 04:34:27 +0000

I’ve always felt a little sorry for [[http://yahoo.com|Yahoo!]] (and I find it ironic that even for such a statement, I need to use the exclamation point). They always seem to be living in the shadow of Google, some times to no fault of theirs. Sure, they have made their share of mistakes, but I think the tech circles, and particularly the media give Y! much less credit than it deserves. And thus I’ve been following the Microhoo saga with some interest, and with a feeling of resignation ([[http://news.yahoo.com/fc/Business/Microsoft_Yahoo|full coverage]]).

{{ http://farm3.static.flickr.com/2214/2234037367_2a77f57641_m.jpg|Microsoft’s hostile takeover bid}}

It would be sad if the merger/acquisition does go through (which I think it will, eventually). Meanwhile, while the long drawn battle plays itself out, I can’t help but wonder why Y! failed to leverage some of its really valuable assets. Honestly, some of their assets have incredible value in them. To some extent I do blame the media (or Yahoo’s PR). I don’t believe that Google does //all// the innovation, nor that all their products are superior to the competition. But still, even if someone in Google sneezes, it gets Dugg and Slashdotted and every one just goes hyper. In this post I’ll discuss some of these issues.

First off, some of the good stuff (I’m not going to mention the usual suspects like Y’s traffic numbers or their share in the web-mail and IM markets):

* Yahoo! is a major supporter and contributor in [[http://hadoop.apache.org|Hadoop]]: an open source implementation of [[http://google.com|Google's]] [[wp>MapReduce|MapReduce]]. Complaints of Yahoo playing catch up and “too little too late” apart (I will address them in another post), I do think this is a timely and much needed development, both for Yahoo and the industry in general. A cursory look at the [[http://wiki.apache.org/hadoop/PoweredBy|list of places using Hadoop]] is enough to give an idea of the kind of enabler this platform is. An entire community and several other projects are mushrooming around Hadoop including [[http://hadoop.apache.org/hbase/|HBase]], [[http://incubator.apache.org/pig/|Pig]] (bad name if you ask me) and [[http://hypertable.org/|Hypertable]]. Google might have the largest, most efficient MapReduce and BigTable implementations, but their implementations are just that — theirs, and extremely closely coupled to their infrastructure. Opening up such a platform for others and building a healthy community around it is I think a Good Thing.
* Yahoo! Developer Network: This crew has churned out some remarkable products (such as [[http://developer.yahoo.com/yui/|YUI]] and [[http://developer.yahoo.com/yslow/|YSlow]]) as well as some really well organized guidelines (such as [[http://developer.yahoo.com/ypatterns/|the Design Pattern library]]).
* [[http://finance.yahoo.com/|Finance]]: the [[http://finance.google.com|competition]] is not even close.
* Flickr and Del.icio.us
* [[http://mobile.yahoo.com|Yahoo! Mobile]]: I have yet to get on the mobile Internet bandwagon, so I really have no first hand experience here. But I’ve heard that Yahoo products have much better support across a wide variety of devices compared to the competition. In fact, until the Java based GMail reader came out, the mobile version of GMail’s web interface was quite lacking.

That said, I feel there are two main areas Y! needs to work on if they want to get back in the game:
* Brand image: Y! needs to work on how they are perceived //externally// as well as //internally//. I feel that people who work at Y! themselves don’t believe in the company, or have the feeling it is somehow not as good as or not as cool as other companies. A lot of Google’s brand image comes from the attitude of its employees, and the work culture. Ditto for Microsoft.
* Streamlined products: Yahoo! Maps and Mail are good applications, but they are far too bloated. Even on my reasonably powerful dual-core desktop, these applications feel sluggish and drive the CPU to saturation which is just not acceptable. In comparison, offerings from Google feel much leaner, load quicker and are more responsive.

In the end, the company that remains competitive and offers the best value to its customers and shareholders will prevail. And I feel that a combined Microsoft-Yahoo entity will not make the space any more interesting. On the other hand (as many fear) I think it might kill and certainly slow down innovation that might otherwise have happened. If Yahoo! can somehow manage to stay afloat on its own, it will at least be a little more exciting. So come on Yahoo! dikha de (translates to “show us”)!!

How gTalk pushed jabber

Diwaker Gupta — Mon, 25 Feb 2008 07:10:57 +0000

I remember signing up for a Jabber account several years back. Since there were a lot of Jabber servers to choose from, and really no “canonical” choice, I ended up trying out a few different ones, until I finally settled on the jabber.org server. Of course, since hardly any one I knew was using Jabber at that time, that account was rarely used.

{{ http://floatingsun.net/wordpress/wp-content/uploads/2008/02/screenshot6.png|Jabber}}

I subsequently tried to convince my friends to start using Jabber, even issuing a [[http://floatingsun.net/2005/07/28/call-for-jabber/|call for Jabber]] on my blog. Suffice to say that in all I added perhaps three friends to my Jabber buddy list. So much for technological merit driving adoption!

Somewhat naively, in that post I said:

//I should point out that Jabber is meant for (and only for) instant messaging. This means that there is protocol bloat for supporting webcams or voice chats. Use video conferencing or VoIP if you want those. Lets keep IM simple.//

Oh, how wrong I was. There are now official [[http://xmpp.org|XMPP]] extensions for [[http://www.xmpp.org/extensions/xep-0167.html|audio]], [[http://www.xmpp.org/extensions/xep-0180.html|video]] and [[http://www.xmpp.org/extensions/xep-0096.html|file transfer]]! Nevertheless, the basic premise of Jabber remains the same: open, free standard, distributed implementation and rich functonality.

However, it wasn’t really until Google embraced XMPP for Google Talk that Jabber really took off. Even now most end users are not familiar with the technological underpinnings of Google Talk. When Google Talk launched, it was a closed network. That is, though it used Jabber as the communication protocol, non Google Talk Jabber users could not communicate with Google Talk users. After some initial resistance, Google finally gave in, making Google Talk an open Jabber network.

It is kind of unfortunate that one of the main “features” of Jabber — a distributed implementation much like that of email — has essentially been nullified by Google Talk, since the vast majority of Jabber users //are// Google Talk users. Of course, it has been a boon to Jabber as well, since it piqued interest in Jabber from all kinds of commercial interests, leading to the significant increase in interest in the XMPP protocol stack. The extensions I mentioned earlier are just a small sampling of the [[http://www.xmpp.org/extensions/|total extensions available]].

It is interesting, as well as a little disappointing, that good ideas often get ignored not due to lack of technical merit, and some how endorsement by a powerful and recognized brand suddenly lends credibility to them.

New MapReduce article in CACM

Diwaker Gupta — Sat, 12 Jan 2008 00:03:51 +0000

Several people have already [[http://googlesystem.blogspot.com/2008/01/google-reveals-more-mapreduce-stats.html|noted]] that Google has published updated statistics on [[http://labs.google.com/papers/mapreduce.html|MapReduce]] in a [[http://doi.acm.org/10.1145/1327452.1327492|recent article]] published in the Communications of the ACM.

While numbers from Google are certainly always interesting, what struck me was the **absolutely pathetic** quality of the graphs in the article. To see what I mean, check out the graphs on Page 5 (you need an ACM account to get the PDF I think). They are hardly readable, both in print and on screen (zooming in doesn’t help). Here is a screenshot (I have included some of the surrounding text to give you an idea of the resolution):

{{ http://floatingsun.net/wordpress/wp-content/uploads/2008/01/screenshot15.png|MapReduce graph}}

As a member of the academic community, I’m quite disappointed and surprised that neither the authors, nor the editors took note of such an obvious shortcoming. MapReduce is great work, and a publication like CACM reaches out to a much broader audience than the conference proceedings of OSDI (where MapReduce was originally published) so I would expect the presentation to be top-notch (and remember, this is Google we’re talking about). Besides, what irks me most is that these are the //exact// same graphs (or at least some of them are) from the original MapReduce paper ({{http://labs.google.com/papers/mapreduce-osdi04.pdf|pdf here}}). Was is so hard to just copy paste or import the figures without messing up the resolution so bad?