But the Make model is rife with problems. Here are a few of them:

• Make requires the use of its own domain specific language — this is, in general, not a good idea. Have you looked at any sizable project’s Makefile lately? Its hard to understand, and harder to modify.
• In the same vein, autoconf/automake are notoriously hard to use. Bear in mind that these tools are supposed to make your life easier.
• Makefile are so hard to write and extend that several popular build systems today are essentially Makefile generators. A good example is CMake.
• Make relies heavily on file timestamps to detect changes.
• Make is slow.
• Makefile are not modular. Recursive Make is especially evil.
• …

I recently began work on a new pet project. As is usually the case, I spent a lot more time figuring out what tools and libraries I would use for my project, than in actually writing any code for the project :) Part of the investigation was to survey the state of the art in build systems. At work, we started using SCons for most of our build, which was already a huge improvement over Make. But SCons has its own set of issues.

One of the nicest features in SCons is that build files are regular Python files. This provides enormous flexibility and immediate familiarity. Unfortunately, the SCons documentation leaves much to be desired. I still don’t quite understand the execution model of SCons very well. For instance, I know how to extend SCons to support cross-compilation for multiple platforms. However, I don’t really understand why those modifications work — there’s quite a bit of black magic that goes on behind the scenes. As a concrete example, there are several magic variables such _LIBDIRFLAGS that have strange powers.

Waf

After some more looking around, I discovered Waf. And now that I’ve played around with it a little bit, I’m happy to say that it is the most pleasant build system I’ve ever used. Things I really like about Waf:

• The execution model just makes sense to me. You typically build a project in phases: there’s a configure phase, to sort out dependencies, tools etc; there’s the actual build phase; and then there’s the install phase. It is not uncommon to have a ‘dist’ step as well, to prepare the source for distribution. Waf understands these operations as first class entities. There is a very strong notion of workflow built into Waf.
• Comprehensive documentation. Check out the Waf book and the wiki.
• Waf has a very strong task model. There is a much stronger notion of dependencies (powered by content hashes, not timestamps). Waf also enforces that all generated code lands up in a separate “build” directory, so your source tree always remains clean.
• Using waf is a breeze — there are no big dependencies, no packages to install, no bloated software to include with your code. Just a single 80kb script.
• Progress indication and colored output is built in, not an after thought. Like SCons, Waf build files are regular Python files.
• Waf is fast. Faster than SCons.

Of course, Waf is not perfect. Coming from a Make/SCons world, I sorely miss the ability to build specific targets. Yes there are ways to achieve this in Waf, but they are all clumsy. The API documentation (and the source itself) are a bit hard to parse.

Related posts:

1. gooLego: Google’s software building blocks
2. Google Earth – System Requirements

I thought I had attained screen nirvana… until I found tmux (hat tip xed). So what exactly is tmux?

tmux is intended to be a modern, BSD-licensed alternative to programs such as GNU screen. Major features include:

• A powerful, consistent, well-documented and easily scriptable command interface.
• A window may be split horizontally and vertically into panes.
• Panes can be freely moved and resized, or arranged into one of four preset layouts.
• Support for UTF-8 and 256-colour terminals.
• Copy and paste with multiple buffers.
• Interactive menus to select windows, sessions or clients.
• Change the current window by searching for text in the target.
• Terminal locking, manually or after a timeout.
• A clean, easily extended, BSD-licensed codebase, under active development.

And how is tmux better than screen? Thats question #1 in the FAQ:

tmux offers several advantages over screen:

- a clearly-defined client-server model: windows are independent entities which may be attached simultaneously to multiple sessions and viewed from multiple clients (terminals), as well as moved freely between sessions within the same tmux server;

- a consistent, well-documented command interface, with the same syntax whether used interactively, as a key binding, or from the shell;

- easily scriptable from the shell;

- multiple paste buffers;

- choice of vi or emacs key layouts;

- an option to limit the window size;

- a more usable status line syntax, with the ability to display the first line of output of a specific command;

- a cleaner, modern, easily extended, BSD-licensed codebase.

There are still a few features screen includes that tmux omits:

- builtin serial and telnet support; this is bloat and is unlikely to be added to tmux;

- wider platform support, for example IRIX and HP-UX, and for odd terminals.

Related posts:

1. Tools I use: screen
2. Tools I use: dirvish
3. Tools for the savvy grad student

First, let us recap some basic requirements:

• Security: this is a no-brainer. If I’m going to trust my passwords to a software, it better be secure. In particular, the developer/owners of the software should not be able to look at my passwords.
• Online and offline access: I want access to my password regardless of whether or not I have internet connectivity. I should also be able to get to my passwords from any of my devices from anywhere in the world. This usually translates to a web-based system where passwords are stored at some server(s) in the “cloud”.
• Export: My password data is mine and mine alone, and I want to be able to export it out of the system (for personal backups, for instance).
• Desktop, Tools, API: I would prefer an open system, one that provides rich access interfaces. I’d love to have a desktop app, plugins for Do or QuickSilver etc. You get the idea.
• Simple to use: The password manager should not get in my way. Adding new passwords should be a breeze. Using stored passwords should be equally simple. Ideally, I shouldn’t even notice that I’m using a web-based password manager and not the stored passwords from my browser.

Without further ado, here are the top three web-based password managers.

clipperz

If you are really paranoid about security, clipperz might be a good option. clipperz is open-source, so you can audit the code yourself should you so desire. It is also a measure of confidence from clipperz — by revealing their source code, they are basically saying, “Hey, we are clean, you can check us out yourself”. It also signals that clippers does not believe in security by obscurity. Apart from being open source, clipperz has all the other expected goodies: you can export your data, it supports one-click logins, you can download an offline copy etc.

I personally did not end up using clipperz because a variety of small problems: I did not like the interface; when I started using clipperz, the one-click login was barely functional; and overall I found the user experience of PassPack much better (read below).

PassPack

PassPack is the first web-based password manager that I used seriously, and so far it has worked out great! The team is very responsive and constantly rolling out new features. I think PassPack did a really good job of promoting and educating the public on “host-proof hosting“, meaning that even the service provider does not have access to your data. This is something that most web-based password managers now support, but at least in my mind, PassPack really led the way in terms of awareness.

Some features that really drew me to PassPack: password tagging; I can mark certain passwords as “favorites” so they are loaded first; the two-level security; the desktop app based on Adobe AIR; the ability to store arbitrary notes (such as routing numbers or PINs). PassPack is particularly well-suited for groups. You can share passwords in a secure manner with people in your group. Recently they even added a feature to allow sending passwords securely via email. Now you no longer need to copy/paste your passwords into chats and emails.

What I always missed in PassPack was browser integration and seamless one-click login. With the PassPack bookmarklet, one-click login is almost seamless, but it never worked very well for me. For some websites it just won’t work. For others I’d have to re-login into my PassPack account. Yet other times there the bookmarklet would work in one browser but not in another. At the end of the day, it was just becoming cumbersome to manage multiple copies of my passwords — one in each of the browsers I used on each of my devices, and one in PassPack.

LastPass

I recently discovered LastPass, and right now it is my favorite tool. I found it via its Chrome extension, which is when I realized that they have plugins for Firefox and work with pretty much all the good browsers on all the major platforms. I have to admit though, LastPass is nowhere close to PassPack in terms of the maturity of the UI and the overall user experience. But the killer feature for me was browser integration. With LastPass, adding new websites is exactly like Firefox asking you to store password information for a website. In fact, the FireFox plugin for LastPass allows you to disable and bypass the Firefox password manager altogether. When you come to a website that has already been stored in LastPass, it will fill out your username and password just like your browser would do. No need to click on a bookmarklet or any thing else. Transparent, seamless integration.

Unlike PassPack, LastPass has no group features at this point, which is perfectly fine by me. In the words of Tara Kelly, a co-founder of PassPack:

Passpack is pwd mngr with sharing & workgroups. Lastpass is login tool for individuals. Different strokes 4 different folks.

If there is a better web-based password manager out there that you know of, I’d love to hear about it.

Related posts:

1. Web based password manager
2. Web services I wouldn’t mind paying for
3. Secure passwords: the other side of the story

A good plotting library

Don’t even mention gnuplot. Not only is it old school (how many times have you looked at a graph in a paper and just known that it was produced using gnuplot?), but it is extremely limited in its feature set. My biggest gripe with gnuplot, however, is that it forced me to separate my data collection/analysis from the actual plotting of the data. I personally am a huge fan of matplotlib — it is an uber-plotting library written in Python. It can produce high-quality graphics in dozens of formats (including interactive plotting), it has an object-oriented API as well as a imperative API along the lines of Matlab (hence the name). You can create amazingly rich plots and best of all, you can combine your data collection and analysis (which I was doing in Python anyways) with your plotting.

If you are more a Ruby person, check out gruff.

Bibliography Management

There are two aspects of bibliography management. First is the context of a specific paper: you are working on a paper and you want to collect all the relevant bibliographic information for citing in the paper. BibTex is the tool that is most commonly used for this, in combination with LaTeX. However, BibTex is buggy, the syntax is inconsistent across implementations, it lacks simple features like variables and the ability to “import” other bibtex files etc. Enter CrossTeX — a drop-in replacement for BibTex. CrossTeX is written in Python. It has an object-oriented model for representing citations. So once you define an object for author “Foo Bar” aliased as foobar, you can simply use foobar wherever you would like to cite “Foo Bar”.  CrossTeX also makes it trivial to define new formatting styles for your citations. For instance, if you want to change the capitalization of the titles or abbreviate “Proceedings” to “Proc.” everywhere. Finally, CrossTeX was built by some nice folks at Cornell, so they know exactly what the pain points of BibTeX were.

The second aspect of bibliography management is simply keeping a track of all the papers you read and review. These will come in handy when you are writing a paper, a dissertation, preparing for a talk or an interview, or simply trying to recall prior work in a given field. I highly recommend using CiteULike — it is an online bibliography management portal. Some features I really like: CiteULike has a really nice bookmarklet that you adding new items to your bibliography using a single click from various sites such as ACM, USENIX, IEEE, PubMed, arXiv and so on; it has some really nice social features as well such as tagging, groups, watch lists etc.; you can download selected citations in multiple formats; you can search easily by keyword, tag, author, area, year etc.

A Text Editor

I don’t mean an IDE (like Eclipse) or a Word processor (like MS Word). I mean a text editor and only a text editor. AFAIC, that means Vim or Emacs (if that works for you). The bottom line is, learn a text editor and become really really good at it. You will be amazed at how much time will save you and how much can it impact your productivity. Some features that are essential: syntax highlighting, regular expression support, spell check, support for snippets etc.

On that note, learn to write in LaTeX. I’m horrified by the fact that so many people are still using Word like tools to write papers. I don’t have anything against Word, but it is the wrong tool for writing papers. Just reference management, formatting, including figures etc are so incredibly easier in LaTeX. And if you are struggling to find the code for the right symbol in LaTeX, you’ll love detexify (hat tip: Nate)!

Version Control

I can’t stress this more — you must get in the habit of versioning everything. Not just code, but your notes, write-ups and obviously papers. Having some version control has saved me from disasters many a times. And if you are collaborating on papers, I can’t imagine how people do it without some kind of version control system. Now there are a lot of choices out there. But if you are really savvy, you must use git :) Basically use any reasonable distributed VCS (Mercurial and Bazaar are also ok), but avoid Subversion and absolutely refuse to use CVS at all costs. CVS has lived a good life, but its time is now past and we must let it go.

Information Management

And by that, I mean staying on top of the news and research in your research area and/or academic community. I’ve found it very useful to add all the relevant blogs to a ‘research’ tag in my Google Reader (yes, the blogging bug has bit academia). Likewise, you can find a lot of current information on Twitter. I’m sure people have already started live-blogging and twittering from academic conferences as well!

Of course, for more conventional searches, DBLP and Google Scholar are invaluable. CiteSeer used to be the go-to website a few years ago, but I personally find Google Scholar much nicer to use and with just as much information, if not more.

Related posts:

1. Tools I use: beamer
2. Tools I use: tmux
3. Must read for grad students

Over the past few years, Google has open sourced several projects that provide some commonly used building blocks in any large software project. Some of them I was aware of since when they were launched (like protobufs), while others I discovered only recently. I couldn’t find any location where all the projects were listed together and combing through Google Code looking for them was painful, so I’m putting together a list myself. Hope some of you find it useful.

• protobufs: Platform agnostic messages. Critical for any distributed system. Note that protobufs only provide message serialization/deserialization (for various languages). An important missing piece is an RPC framework built on top of them. There are several projects attempting to build one using protobufs, but none of them are robust or mature enough for production use.
• style guide: The importance of a style guide is probably understated. It is not about what is the “right” style — it is about consistency. While people may have different opinions, if everyone follows the same style, the code becomes much more readable and maintainable. Google maintains style guides for C++ and Python.
• config flags: Another important building block for all command line programs.
• logging: Self-evident. Google’s logging library supports various log levels and other useful macros.
• core dumper: A very nifty library — it allows you to dump core from within a running application. Extremely useful for debugging production systems.
• perftools: An extremely useful library for measuring and monitoring performance of programs. By simply linking against perftools, your application gets a much better malloc, heap checking, visual CPU profile of various routines (via graphviz), visualization of memory usage etc.
• googlemock: A framework to quickly build mock objects — useful for testing.
• googletest: Google’s C++ unit testing framework, built on top of xUnit. Integrates well with googlemock.

Of course, this is not an exhaustive list. There are numerous other open source projects from Google, some of them probably much more bigger and visible than the ones listed above — such as Wave, Go, GWT etc. If there’s a project that is a software building block that I missed out, do chime in the comments below.

Related posts:

1. India blocks blogs?
2. Best Web 2.0 Software of 2005
3. The silent victories of open source

First, some background. Lets just say that I lost my editor virginity to Vim. It was a brief, but violent introduction — the modal editing was too unfamiliar, the learning curve too steep. After dabbling with a few other conventional editors (such as KWrite), I settled upon Emacs (XEmacs actually, but thats another story).

For the next three years, I tweaked my .emacs file, fiddled around with settings and plugins and modes, played games and browsed the web, checked my email and newsgroups, all within the comfortable confines of Emacs. But I was getting wary of the long startup times and (at that time) the inability to use the same interface and features in console mode (such as over SSH) as in GUI mode. It was time to move on.

I rediscovered Vim around 6 years ago. I started with a clean slate. As the saying goes, Emacs is an operating system that also happens to have an editor in it. The relatively more focused feature set of Vim was refreshing in comparison. I loved that I could work in GUI mode, save my session, go back home and resume my session in a terminal over SSH, which the exact same interface and keybindings. I quickly became very productive with Vim, and over the years have honed my plugins, settings and color themes to just how I like them.

But recently, I’ve been thinking about this again, and I might just reconsider Vim. I highly recommend reading these two blog posts to better understand where I’m coming from:

• From Vim to Emacs — Part I
• From Vim to Emacs — Part II

Don’t get me wrong — I think Vim still has a lot to offer. But, I can not deny that Vim is not what I would call a “forward looking editor.” Here’s why:

• Development community: the Emacs development community is a lot more open and vibrant right now than the Vim community. Part of this has to do with the BDFL model in Vim. Bram Moolenar has done a tremendous job in bringing Vim to the stage where it is. People can and have forked Vim in the past. But for one reason or another, Vim has stayed Vim, and its development trajectory has been slow and incremental.
• Source code: Vim’s source code is not clean. At all. I just briefly skimmed over the source tree for Emacs 23, and it looks a lot more understandable and well structured.
• Architecture: Vim 7 finally got spell check. But the spell check does not use any of the existing tools or formats. Vim has its own scripting language, with its own interpreter, grammer and data structures. Why not just use one of the many wonderful programming languages out there? Yes, there are interfaces to allow writing Vim code in Python, Ruby, Perl etc. But why reinvent the wheel all over again?

When Bram Moolenaar — the lead developer of Vim –  joined Google, I had hoped that Vim would generate a lot more interest and enthusiasm. But so far, it hasn’t changed much.

And so, in the next few weeks, I’m going to take another look at Vim as well as Emacs. I’ll try to do an objective evaluation of where the editors stand today, where I perceive they are headed. I hope to make my decision on whether to move away from Vim or not by the end of this year.

Related posts:

1. The Ancient Wars
2. Emacs vs. Vim

Traffic spike

I don’t know if it was “vim” or it was “sexy”, but somehow this post landed up on Reddit. This is the closest I’ve come to being slashdotted — for that one day, Reddit drove nearly 95% of the traffic to my site. Also, before you start jumping to conclusions from the graph above, let me put some numbers out there. On average, my site gets anywhere between 200 to 400 visitors daily. On April 18th, my site got 7000+ visitors, an order of magnitude more than I normally get. That is the spike you see, and now the traffic is back to normal, thank you very much.

Since a lot of people seem to be interested in Vim hopefully, I want to discuss the space of text editors (in particular, editors for programming) and where I think we are headed.

The first observation is that both of the giants on the editing world — Vim and Emacs — are ancient by any standards. Depending o how you look at it, I think it is fair to say that neither editor has evolved significantly in terms of the underlying code, architecture and usage model in the past two decades, if not more.

The second observation is that despite the large number of editors out there, IMHO few have any significant mind and market share other than Vim and Emacs. Obviously there is Eclipse, Visual Studio, IntelliJ etc.

And so I wonder, what would the text editor landscape look 5-10 years down the line? There are many who would say if it ain’t broke, don’t fix it. However, I’m a big believer in change, and I think over time, change is inevitable. However, at this point I don’t know what, if any, substantial change is happening in the text-editor arena. No new editors with fundamentally new ways of manipulating text or amazingly compelling features are emerging. I actually don’t mind reinventions of the wheel either, as long as the reinvention delivers a much better wheel. For instance, the Vim source code is not the most modular, extensible or maintainable. And it is in C — not that there is anything wrong with it, but I think an object oriented language is better for a complex piece of software like Vim.

The only recent buzz has been about Textmate and the many clones it has since inspired. I’m really looking forward to Yzis reach a usable milestone. What are other promising editors out there that you are excited about?

No related posts.

I use vim as my editor of choice. As I note above, to each his own editor.

However, Vim is not what one would call a “sexy editor”. After all, it has been around (in some shape or form) since before I was born. It does not generate as much buzz in the blogosphere and is not the darling of all the new kids on the block, as some of the other editors out there. Not many Ruby on Rails developers, for instance, seem to be using Vim for coding (actually, saying that a lot of RoR developers seem to be using Textmate is probably more accurate, but you get the point). It is written in C and does not use git for hosting. The Vim website leaves much to be desired. In the social networking world, Vim barely has a presence.

But, I contend that Vim still has a lot to offer. Here are few of the things you can check out to spruce up your Vim usage:

• xoria256: a modern, functional and eye-pleasing color scheme.
• snipMate: Textmate like snippets for Vim. Interestingly, snipmate lives on github. There are actually several textmate like plugins for Vim, I prefer snipMate. I have succesfully used snippetsEmu in the past.
• Omni completion: Probably not as awesome as Visual Studio’s, but useful nonetheless.
• Check out my list of some essential vim plugins.
• A byte of Vim: from the author of the well known “A byte of Python” book.

I’m also very happy to see the number of Vim related repositories on github. Bottomline: don’t give up on Vim. Vim is still sexy baby, you just need to look in the right places :)

Update: I’m including a screenshot of Xoria below.

No related posts.

Having all my financial data pulled in a single place is fantastic. Previously, I used to have to go and log into 5 different websites to check on all my bank accounts and credit cards. Add to that the ability to examine spending trends, watch your investment grow (or crash, as is the case with the current economy) over time, alerts for fee, low balance and over-budgets etc, and you’ve got a killer service in your hands. Of course, Mint is not without its problems, but the benefits far outweigh the snags.

Passpack has been a real time saver for me. I have written about web based password managers before, and so far Passpack has been just fantastic! I just wish they would bring back search-as-you-type…

I’m a big believer in using the right tools you help you work smarter. RTM is a great way to offload your todo list from your brain. It has all the key elements of a good web service good — a great UI, keyboard shortcuts, the ability to email tasks, integration with services like IMified, a nice API (so you can use desktop apps such as Gnome DO or Quicksilver to interact with RTM without ever leaving your desktop).

Related posts:

1. Better services
2. How LeT change of mind saved IIT-K : HindustanTimes.com

Being a grad student (for that matter, in almost any profession these days), I frequently need to give talks or present some material. I have finally settled on Latex Beamer as my preferred presentation tool, and this article describes why.

Introduction

Presentation is one of the most effective means of communication for a small audience with diverse backgrounds. Both in the industry and the acedemia, it is becoming increasingly important to create affective and compelling presentations. Not surprisingly then, the presentation tool you use becomes very important in the work place.

The de facto tool for presentation out there is Microsoft Powerpoint. For more reasons than one, I prefer not to use it. I have tried several alternatives, and finally decided to use Latex Beamer for my presentations. Here I try to describe why I made this choice. I must mention here that the beamer web page looks ostentiously simple and naive — don’t be fooled by it. Beamer is one of the most sophisticated and extensively documented (user manual has more than 300 pages of professionally written documentation) presentation tools I have come across. Take a look at one of the sample slides to get a feel of what beamer can do.

Things I dislike about other presentation tools

While I’m not talking about any one tool in particular, the general flavor is of tools belonging to the Powerpoint family (this includes OpenOffice.org’s Impress, KOffice‘s KPresenter etc)

• I have to worry about layout
• Font sizes are a function of amount of content
• Changing parts of a “theme” is hard
• Powerpoint slides won’t run nicely on Impress or KOffice. The latter two won’t run at all on Powerpoint. Why do I need something as bulky as powerpoint just to do the presentation? While making, I can understand that we might need significant software complexity, but can’t we have something more lightweight for presenting?

Things I like about beamer?

• Its LaTeX: latex and friends have survived the test of time and for more than 2 decades people have been using tex derived technologies for typesetting their writings. With latex, beamer makes it easier than ever to put mathematical formualae and all kinds of symbols in your presentations, embed images, make tables and do everything else that you can do with latex. Since many of us already use latex, it means there is less tool to learn — I can make my presentations in a language that I’m already familiar with! And I don’t need any bulky tool to manipulate my presentation, just a text editor is enough, thank you.
• Its PDF: We all know what PDF stands for — Portable Document Format. Thats it! Portable! Latex runs on all major operating systems and architectures out there. Once you get a PDF from Latex, you can display it using any regular PDF viewer. Imagine how easy it now becomes to move your presentation around. You don’t have to worry if your laptop breaks down and the other laptops in the room don’t have the right version of Powerpoint installed. Put your PDF in a USB key and stop worrying about it!
• Takes care of layout
• Themes are endlessly customizable: beamer comes with dozens of pre-packages themes, and its very easy to modify an existing theme. Same thing with fonts and colors (you can even do alpha transparency!)
• Notes and handouts made the way you want them
• Organize your presentation in a logical manner: beamer sort of follows the MVC philosophy. In each presentation, there is a content structure, which determines how your content flows through (just like a regular article with sections and subsections). Then there is a slide structure, which determines how this content fits onto your slides. The content structure controls the generation of navitation and table of contents. The slide structure controls the slides and the control flow between them.
• Amazing documentation: The beamer user manual is over 200 pages long, and its all good solid documentation. It is amazing well written considering the fact that its mostly done by a single person. It starts off with a nice tutorial, followed by detailed references and examples.
• Accompanying packages: Just check out the documentations for xcolor and pgf. The documentation is just as comprehensive as beamer itself, and these packages make it easy and fun to do fancy stuff with beamer. Like draw pretty pictures and do some basic animation. Again, all with the comfort of latex.

But nothing is perfect

• No knowledge of projectors or screens — the user has to deal with that (or the operating system)
• Animation is still hard.
• In general, multimedia is hard: embedding audio and video clips may not work reliably on all platforms.

I highly recommend beamer to anyone who wants to try an alternative to Powerpoint, and if you write a lot of technical papers in latex, you’ll immediately love beamer. Check out my NSDI talk for a sample of what beamer can do for you.

Related posts:

1. Why beamer?
2. Its been a long time…
3. More Vim tools