Arvind Narayanan's journal

Dreams and the self

2009-06-22T16:58:07Z

In last night's dream, a T-Rex was trying to eat me. It went on for the longest time, it was horrible. I wish the T-Rex had cornered me at some point and said "game over, motherfucker!" so that I could have gone on to the next dream or whatever. But that's the funny thing, apparently you can never die on your own dream. I certainly never have.

I frequently fly in my dreams (this has been by far the most common recurrent motif); I hear that a significant proportion of the population does this as well. Do you? I wonder what it means for humanity as a whole.

Another thing that fascinates me about dreams is that since your conscious processing is subverted, you can pretty much simulate multiple entities in a way that you can't when you're awake. For example, this dude I was once talking to in a dream sat with his hands firmly planted in his pockets, and I was wondering why, until he pulled out a gun on me much later. Of course, my brain knew all along why he had his hands in his pockets -- both my character and the adversary are being simulated inside my head -- but it chose to keep that fact from me, just for shits and giggles. Or sometimes my character would need to sum a sequence of numbers, and find to his surprise that the sum is exactly (say) 100. While I can of course sum numbers in my head, quickly coming up with a sequence of random-looking numbers that sum to 100 seems beyond my wakeful arithmetical ability. The only conclusion is that my brain was working on the sum for a prolonged period of time -- and choose not to tell me.

The amazing, amazing thing is that even though almost everyone goes through these experiences when asleep, almost no one realizes that their waking consciousness is also a similarly fragile illusion. Even seeing the breakdown of coherent consciousness in other people -- such as patients with a severed corpus callosum, who develop two distinct personalities -- doesn't seem to help. Nor do studies showing that our consciousness is merely "informed" of the decisions we make, said decisions actually having been made in the subsonscious several seconds before we are even aware of making them. People have a remarkable ability to ignore any evidence that contradicts their model of the self.

If it isn't enough that most of the problems in the world are caused by our outsized egos and our little worlds that revolve around ourselves, chew on the fact that the basis for ego rests fundamentally in a fallacy :-)

Bioinformatics needs to be democratized

2009-06-15T19:39:27Z

I've started working with genome data, so I've been giving myself a crash course in genetics, especially the human genome, over the past week or two. I'm almost up to speed — I can now understand paragraphs like this without needing to look anything up:

We identified a 47 Kb interval containing an Alu insertion polymorphism (DXS225) and four microsatellites in complete linkage disequilibrium in a low recombination rate region of the long arm of the human X chromosome. This haplotype block was studied in 667 males from the HGDP-CEPH Human Genome Diversity Panel.

Just two weeks ago I would have been completely lost. That isn't your basic Mendelian genetics either, it's a research paper from 2007.

Learning a new set of concepts over a short period of time by immersing yourself in it is an intense experience, and one that I thoroughly enjoy. That's why privacy research has been so rewarding for me — it has given me a chance to read tons of papers in law, economics, sociology and now genetics, not to mention many, many areas of computer science.

My main aim was to understand the math behind meiosis. While it isn't very hard, it is still an area of active research, and our knowledge of it is incomplete. Consequently, there doesn't seem to be a way to get to it without cutting through layers of biology. Even the really basic stuff, like chromosomal crossover, is generally explained in a tedious way using observations of traits in plants and fruitflies. This is because of an accident of scientific history — until DNA sequencing became a reality, the only way to learn what happens during meiosis was to observe the results when animals mate and make inferences about the genome based on that. It's like the story of the blind men and the elephant.

More generally, bioinformatics seems to be populated by people whose background is primarily in genetics rather than in computer science. When computer scientists do study the genome, they end up spending their time on something inane like proving the NP-hardness of some computation on genomic sequences, apparently oblivious to the fact that human DNA does not consist of worst-case strings constructed by malicious adversaries! Even on basic information-theoretic questions such as the amount of entropy in the human genome, the best I could find is a marginally related, speculative blog post.

Someone needs to write "The Facts of Life, for Computer Scientists." I'm confident that any competent python hacker with a solid knowledge of algorithms and statistics could read that document, learn the basics in a few hours, download some data from HapMap, install a library of useful tools with a single apt-get command, and start producing useful code and generating interesting hypotheses, all in the course of an evening!

Bioinformatics needs to be democratized. There is a gigantic amount of data available, but the people who are producing the data aren't necessarily the ones best equipped to play with it. On the other hand, there is a huge community of hackers who would like to do just that, but don't realize how easy it is. If you could get these two groups to talk to each other, amazing things can happen.

Why are there so many theory bloggers?

2009-06-09T16:35:13Z

Theory seems to have by far the most bloggers of any subfield of Computer Science; I'm sure I'm not the only one wondering if this is more than coincidence. Here are a few possible reasons:

Theory is cohesive. Most pairs of theorists find each other's work somewhat interesting. At the same time, the field is unlike most of Computer Science, with its emphasis on proof and disregard of experiment. I doubt that this level of cohesion can be found elsewhere in CS: "systems" is too vague and broad, while most other areas — AI, data mining, information retrieval, semantic web, databases, logic, formal methods, programming languages, compilers, architecture and graphics — fall on a massive spectrum.

Missing from that list are crypto, security and privacy. Cryptographers, in my experience, are generally horrified by the prospect of saying anything publicly that isn't heavily peer-reviewed, so that's out. Security and privacy are highly interdisciplinary, so that's not a cohesive subfield either.

In a small way, my blog aggregator may have contributed to the cohesiveness of theory bloggers by fostering a sense of community.

Theory doesn't get press. Except for the occasional journalist making an amusingly feeble attempt to explain P =? NP to the lay public in the context of the Clay Math Institute prize, theory stays out of the press because it doesn't generate pageviews.

In most other fields, important papers have a non-zero probability of getting written about (in the case of graphics papers announcing new techniques, it is virtually guaranteed since they are accompanied by jaw-dropping animations.) Consequently, theorists need a way of spreading the word about papers that are important/interesting. Word of mouth and best paper awards only go so far in the 21st century.

Theory is hard. Don't get me wrong — I'm sure research in other fields is just as hard to perform, but in my opinion, theory papers are particularly hard to read, especially for newcomers, simply because of the high degree of abstraction. This gives theory authors a strong incentive to explain and motivate their papers in more readily understood terms, and blogging is a great way to do that.

All that said, a big part of it is probably pure chance. Specifically, Lance Fortnow's pioneering blog may have convinced many theorists, by setting an example, to shed the belief that blogging is a frivolous, vulgar activity indulged in solely by the unwashed masses, far too undignified for solemn researchers such as themselves :-)

Rant for the day

2009-05-26T22:51:56Z

Today has been all about people wasting my time.

First it was something to do with my damn car, but I'm not going to talk about that.

In the afternoon someone came over to look at my bike (I'm selling all my stuff since I'm moving to California). The price in the ad was $100. The buyer sounded like a really nice person. After taking my bike out for a spin, she brought it back, said she liked it, and took out the cash.

Specifically, $80.

She claimed she only brought along $80 because that's what her budget was, and wanted to see if I'd take that. THE AD SAID $100, BITCH! The whole time she knew perfectly well she was just wasting my time. And hers. On the off chance that I'd turn out to be a sucker. Fuckity fuck. Of course I didn't take it. I'd rather abandon the bike than sell to a shamelessly dishonest person.

This is yet another way in which Craigslist sucks big time. Honesty is largely a product of the circumstances; it is inherent in a person only to a small degree. In the absence of a reputation system, Craigslist buyers/sellers have no problem behaving in ways they wouldn't in any other context.

Next, I had to sell my all my books. I filled up suitcase with about 30 of them, and some DVDs, and took it to Half Price Books. I could have sold them on eBay, but I knew I'd go crazy listing and shipping them individually. Half Price Books had me wait for about 45 minutes. The dude evaluated each book minutely. On some of them he even brought in a colleague for consultation. They'd talk in hushed tones, then he'd write something down, and move on to the next one. It was a bizarre process. At the end of it, he had a number ready for me.

Specifically, $25.50.

If you know me, you know I don't show surprise easily. But when I heard that number, I literally stared at him with my mouth agape. And then I had him repeat it several times. The guy tried to remind me that they sell it for only 50% of the retail value. So I asked him why he was giving me 7%. He had no answer but a shrug.

It gets worse — they'd take either the whole lot or none of it. I took the offer, of course — the time I'd already wasted was a sunk cost, and they weren't being dishonest, just thoroughly inefficient.

My time was worth more than the money I made, especially if you include the time I spent driving to the store and back. Essentially, I was spending money to get rid of my books. Next time I'll just toss them in the dumpster.

Oh wait, there isn't going to be a next time. It's been a long time since I bought a dead-tree book. With the Kindle, not only are books cheaper, you don't have to delude yourself into thinking there's a meaningful resale value for them. You're buying 1's and 0's and you're paying for the convenience.

Use a foil, for crying out loud!

2009-05-17T17:17:03Z

Whenever a story has a significant scientific component — whether in a novel or on television — there's a tricky problem: how do you communicate the science to the viewer? The characters are already up to speed, so you can't have them just explain it to one another.

Obviously, science fiction is the genre that suffers most acutely. The writers of early sci-fi novels, in the 40s and 50s, for the most part just weren't very good, so they often didn't even make an attempt to solve the problem. For a brilliant illustration of how jarring it is to the reader when characters discuss with each other what must be obvious to them, see If all stories were written like science fiction stories.

But in the written medium, there is at least the crutch of the narrator. While long passages with no dialog are boring, they are not horribly unrealistic. Television doesn't have this escape route.

Take House, which is probably my favorite currently running show. When House and his team do their thinking in front of the whiteboard, they often converse in medical jargon, and then immediately translate it into English. While this makes the show intelligible to the lay person, it must make it unwatchable for real doctors. That's certainly the effect that shows or movies have on me when characters explain computer jargon in situations where they shouldn't.

The funny part is that the House writers seem to be aware of and rather embarrassed about this shortcoming. I say this because the characters sometimes make a remark about the fact that it is weird for them not to be using jargon :-)

The good news is that audiences seem to be tolerating unrealistic dialog less and less. The other three shows that I watch — all newer than House — have built-in premises to solve the problem. In Bones, all the principals have different specialities, so it is natural that they would explain their technology and reasoning to each other all the time. Burn Notice pretty much says "fuck it," and has the protagonist do voice-overs to explain what's going on.

Lie to Me has a foil in the form of a newly recruited crime-solver who is naturally brilliant at lie-detection but is unaware of the scientific principles behind it. Naturally, the other lie-detectors explain things to her a lot. Eventually this character will no longer be able to act as a foil, because she will have learnt the ropes, but I believe the show creators expect that by then, the audience will also have been educated to a large degree. An educated audience leaves them with a different problem — building suspense. So far I've been able to guess the conclusion in most of the episodes (it is yet Season 1). It remains to be seen if they can pull it off.

I think one of the best solutions is to have a true foil, a character who is completely uninitiated. The textbook example, and one that will perhaps never be surpassed for the completeness of the resulting dramatic effect, is Conan Doyle's Dr. Watson. The brilliance of Doyle's creation is that the relationship between the characters makes it possible for Holmes to explain what he wants, when he wants, and no more.

Of course, foils have been used effectively in modern television, such as in the show Psych which I've occasionally watched (although far from my favorite). At any rate, I think it is wonderful that audiences are rejecting unrealistic dialog, and shows are experimenting with different solutions.

All done

2009-05-08T07:04:19Z

So I'm officially done at UT. Now the only thing I need to do before packing up and flying to California is to sell my car :-)

The paperwork and general procedural nonsense they make you do to get a Ph.D is as arcane and retarded as any Hindu ritual I had to sit through as a teenager. This is in spite of the fact that the CS department do their best to ease the pain. People are generally trying to do the right thing, but the system is badly broken and change-resistant.

Smartphone versus Kindle

2009-04-23T05:49:17Z

On my last trip I packed my kindle in my checked luggage instead of my backpack. When I took it out, it looked like this:

It wasn't physically cracked, but half the display was gone, and it won't turn on any more.

I'm going to call and check if my warranty covers it, but if it doesn't, I don't even really care. In the last couple of months, 80% of my book reading has been on my iPod Touch. As it turns out, Amazon has a free and very slick Kindle for iPhone app that lets you read books you've purchased for the Kindle on your iPhone (or iPod Touch). The devices even automatically sync with each other, so that you always pick up where you left off. Of course, the letters aren't as crisp on the Touch as they are on the Kindle (the latter is almost equivalent to paper), but to me this disadvantage is monumentally outweighed by the fact that the Touch fits in my pocket.

There are still many people (albeit a dwindling number) who won't read books on anything but dead trees, but the average person just doesn't care. Consequently, the market for e-books on the iPhone is huge. On the other hand, Amazon's margins on the Kindle device are slim, probably even negative. This is because it uses Sprint's EVDO network, but Amazon eats the cost and doesn't pass it on to the user. Further, there are 10-15 times more iPhones than Kindles in circulation. Amazon has therefore done the obvious thing by allowing even non-Kindle owners to buy books for the iPhone using the app.

I have a hunch that smartphones are going to swallow the book reader market, just as they did PDAs and Internet Tablets.

Edit. Amazon is replacing my Kindle. 1-day shipping on the replacement! Woohoo!

Thoughts on the Y Combinator interview process

2009-04-19T18:11:09Z

We (watchuwant.tv) interviewed yesterday. We didn't make it. Here are some thoughts.

First of all, a lot of people have talked about an adversarial interviewing style. Our experience was the opposite. Everyone was super-friendly the whole time. David's theory is that if you only have an idea or a demo, they push you during the interview to see how you can perform under pressure. On the other hand, we had a working website that had been out there for a month, so we'd already proven we could build something. I find it plausible; I think in addition, maybe the more they like you, the more they push you. It would be good to know what affects the interview style, so that you can know ahead of time what you can expect.

In any case, I just don't get why a lot of interviewees were nervous. The waiting room makes you feel very comfortable, and whatever little nervousness you have should evaporate once Jessica makes her appearance and greets you with her radiant smile. I guess for me there were a couple of more reasons why I wasn't nervous: we weren't dependent on YC; we're pushing ahead with the site regardless, and also, my day job involves often speaking in front of 300 people. Sitting in front of 4 people is nothing by comparison :-)

I flew in from Austin. Near the end of the interview, Jessica said, "so do I make the check out to you?" I said yes. "Did you actually spend 700?" I said I'd spent most of it. I actually spent more than 700, but I only had receipts for slightly less. So she wrote a check for 700 right there on the spot. I was kinda speechless.. I was expecting a bunch of paperwork. We kept laughing about it later. That little incident tells you a lot about why YC is awesome. No bullshit.

We thought the interview went really well. But looking at our notes later, we realized we might not have conveyed fundamental things about the site. For instance, they kept asking us if our featured channels were manually curated. We thought we'd said pretty prominently in our application that we do automated, instantaneous channel creation. So guys, remember that they interview a bazillion companies all at once, so don't assume they remember what you said in your application.

Do mock interviews with as many people as possible. We did a bunch, but only one was a YC alumnus. That interview predicted their reactions far better than the others. Remember this. Each person/group has their own world-view.

One last thing: people who got in got the call as early as 6pm or as late as 11:30 pm. We got emailed around 9pm. So stay by that phone.

So that's pretty much it.. our experience was great, even though we didn't get it, and the preparation for the interview taught us a lot, and spawned a bunch of new directions and monetization ideas. Good luck to today's and future interviewees. You'll hear more about WatchUWant.tv soon :-)

The future of jobs

2009-04-16T23:52:25Z

I think it is fair to say, to a first approximation, that rewarding professions are those that involve some amount of creativity. Of course, you could be unhappy in any job, say because your boss is a jerk, but under the right circumstances you should be able to find job satisfaction in a creative field. On the other hand, it is hard to imagine this being the case with thoroughly repetitive, monotonous tasks.

Mike Rowe would appear to take a contrarian stance in his TED talk celebrating dirty jobs, but I think a distinction must be made between taking up a job in order to pay the bills (and still finding a way to be happy with it), versus actually choosing something as a rewarding career. There is empirical evidence for the legitimacy of this distinction: there are people who assemble computers, cars or motorcycles as a hobby, even though these tasks have been completely automated, whereas you'd be hard pressed to find someone who spends time separating toxic sludge into its constituent components just for fun. I think Paul Graham makes the same point when he makes fun of a job posting calling for applicants with a "passion for service."

Given this (admittedly crude) dichotomy, it becomes important to ask if there will be a day when everyone has a rewarding job. I think this would be a great thing to look forward to, and entirely within the realm of possibility, since technology has been making relentless progress in automating anything that is repetitive. In the rest of this essay I will analyze the unrewarding jobs that have not yet been automated, and discuss what is holding back automation and what it would take to get there.

A couple of quick points of clarification: first, I'm looking a few decades into the future; the current recession has no bearing on the analysis here. Second, even though the average person has an emotional (and adverse) reaction to the loss of jobs to automatons, I will assume I am writing to a more mature audience, and won't waste time arguing that automation is almost always a net benefit to humanity.

The first category consists of tasks that we can already automate, but for some reason continue to be performed by humans, like taking orders at a restaurant. The main reason that these jobs still exist, in my opinion, is inertia: people react with surprise or revulsion at seeing a machine in a role formerly filled by humans, and are confused by the user interface. But they will eventually get used to it, just as they have with ATMs and voice menus. Technologies such as Microsoft Surface, which make it possible to market automated restaurant service as a positive experience, will speed the process along. It is likely that a small percentage of people will still pay for the privelege of human interaction, but that doesn't affect my overall argument.

Incidentally, businesses that involve fully or partially automating tasks previously performed by humans have always been excellent avenues for entrepreneurship. The early history of fast food is a great example.

The second category consists of jobs that we can't yet automate, but are easily with reach, such as fast-food preparation. Human labor is still available too cheaply, preventing the investment needed to build, perfect and deploy the technology to fully automate these jobs. Increasing automation and increasing labor costs form a virtuous cycle, so in a sense it is just a matter of time. But can the cycle be sped up? I can think of three factors that might help:

A slightly over-educated workforce. This puts pressure on workers to try to migrate up the chain toward more creative jobs.^[1]
Less government interference in business. I think this is somewhat nuanced: labor unions have a negative effect on automation, whereas a high minimum wage probably encourages it.
Stimulating developing economies. This will result in their labor prices rising, thus diminishing cheap pools of immigrant laborers and cheap offshore outsourcing.

Finally, it is important to note that there will be "collateral damage," if you will, in terms of jobs or skills that are rewarding to various degress but that we will nevertheless learn to automate. This is not a huge issue today, but this happy state won't last long. Already it has raised some very interesting questions: do singers need to work on their pitch at all, given that software can correct for human imperfection? Technology that falls into this category will be met with much resistance. Some will embrace it, but others will decry the downfall of civilization and probably try to get it banned. Eventually, however, I think we will have to learn to treat jobs that have been automated as hobbies rather than professions, and continue to strive to find even more challenging avenues for human creativity.

^[1] Although it is hard to envision and the process is slow, society can indefinitenly accomodate migration toward jobs at the creative end of the spectrum. Jobs such as entrepreneurship or documentary film-making have elastic demand, unlike say sanitary work. If entrepreneurship becomes highly feasible and attractive for whatever reason, the economy will gradually adjust so that there are more, smaller companies.

Tragedy of the commons: WiFi version

2009-04-08T07:12:19Z

For years now I've run an open wireless network at home. I'm in agreement with Schneier here. Sadly, my network is no longer open.

It always annoyed me that everyone else's network was password protected. On the rare occasion when you need it, such as when you're traveling and the person you're staying with doesn't have a wireless router, a neighbor's open wireless can be a life-saver. Since we don't have a culture of open networks, I'm forced to pack my wireless router when I travel.

Unfortunately, some of my neighbors seem to have decided to steal my wireless permanently. They must be running bit-torrent or something, because my connection quality is seriously degraded. In classic tragedy-of-the-commons fashion, they've forced me to protect my network.

Actually, the admin interface to my router doesn't work any more, so I can't even do that. I'm just plugging my laptop directly into the modem for now; I'm going to have to get a new router. I've had seriously bad luck with wireless routers: this is the third one to die on me in as many years. The only category of electronics I've had worse luck with are hard drives — six failures in two years. Fortunately that will never happen again, because I've switched to SSDs. Seriously, those of you still on hard drives have no idea what you're missing.

Quick note on the social networks paper

2009-03-27T00:09:25Z

Our social networks paper has been getting some press, including on slashdot and BBC news. The de-anonymization attack in our paper is very broad in scope, and unsurprisingly, it has been whittled down into phrases like "we can find you on twitter."

Please read our FAQ. Our attack applies to a variety of scenarios, some involving very sensitive data, including phone call networks which are often shared in anonymous form. The paper is online (html) and most of it is readable with a minimal technical background. I hope our "State of the Union" section will convince you that social network anonymity is a rather important issue.

On a personal level, working on this paper was very gratifying for me--the de-anonymization algorithm is far deeper than the one in the Netflix paper, and took far longer to develop.

Finally, a plug for 33bits.org, my blog about the de-anonymization work I've been part of.

Edit. And I also have a shiny new web page! After testing about 15 different widgets for embedding an RSS feed into a page, and finding none of them satisfactory, I gave up and wrote the damn thing myself :-)

Ubuntu is a joke

2009-03-03T01:38:46Z

I installed Ubuntu 8.10 (whatever it's called.. Intrepid?) on my new laptop, and here are some of the things that are seriously wrong just in the first week of using it:

I have to type in my password for every little thing. Vista's UAC annoyance doesn't even come close. There is no simple way to stop it from forgetting my password, but fortunately I figured out once and for all how to hand-edit some stupid deliberately undocumented config file in order to do that.
I had to spend a few hours to get basic shit like flash working, because the Intel graphics chipset I have is apparently not properly supported. Open source nerds like to scream that hardware vendors don't release specs. Intel is the opposite, they even go so far as to write open source drivers themselves. And it's still not supported? What the fuck?
Using a password instead of a key for WPA2 is evidently unsupported. You'd think security is the one thing they'd get right.. no? Read the spec, write the code. What's the problem?
There are many, many UI fubars like the fact that the window manager allows you to resize a window so that its titlebar disappears underneath the top-panel, and if you didn't know Alt-drag you wouldn't be able to get it back. Oh, and I can't change the freaking time! Every time I change it it goes right back.

I have absolutely no religious reason to use Ubuntu, Linux or open source. But the fact remains that it's a programmer's dream, apt-get is manna from heaven, and using anything else is like poking yourself in the eye. Yes, including OS X. And I just don't partition my time between programming and other things; that's not how I roll. So I'm forced to use this piece of crap as my main system.

Computers suck.

Edit. Dear anonymous Linux zealots who've been commenting on this post. I have no idea where you're coming from. I'm not a n00b (this is like my three hundredth linux install in ten years of using it as my primary OS). More importantly, I do NOT want your help. And I'm not interested in installing another distro. You are free to keep writing stupid comments but I haven't been approving them and I won't. kthxbai.

Eat. Sleep. Lift.

2009-02-13T07:55:47Z

Lifting. 2008 was a wipeout—almost zero net muscle gain. This was because I spent a full 3 months traveling (spread over 7 trips) between January and October, which tends to be harsh on my body. I stayed put after that, but then I got sick a couple of times and the gym was closed during the winter break.

2009 has been much better so far. I'm back to pushing my max in almost all the exercises, and I also manged to get a brief fat-loss/cardio stint in. I have a ton of travel coming up again soon, and I'm hoping to convert those into productive fat-loss cycles by the simple heuristic of limiting myself to a salad for dinner (see below).

Overall, I'm happy with the strength gains I've made, especially core strength. I now deadlift 300 lb sets, which should go up a bit more pretty soon as I improve my grip strength, which seems to be giving out before my back muscles do.

I'm not happy with the size gains though, which do not seem to be commensurate. Perhaps it has something to do with fast vs. slow twitch muscle fibers. I dismissed that as esoteric crap when I first looked at it, but perhaps it's time for another look.

Diet. Occasional hiccups involving tiny sandwiches aside, I've pretty much figured out my diet these days. My constraint is that I need to get enough calories and proteins to weight-train effectively. My problem was that I was brought up in a culture where cold food is just not food, period. No, not even our desserts, at least not the ones that weren't recent imports from elsewhere. I still remember my consternation when presented with a salad when I first visited Europe in '99. Neither I nor any of my friends who were with me at the time had ever before eaten vegetables that weren't at least steamed :-)

Of course, I soon understood that culinary preferences are almost entirely the result of cultural conditioning. I pity the Italians for clinging to their idiotic notions of authenticity and missing out on the wonderful American mutations of the Pizza. (Although the univsersally addictive nature of hot foods seems to be an exception to cultural relativism. I believe it is due to the endorphin release triggered by capsaicin.)

Anyway, the understanding that my taste barriers were merely the result of my upbringing didn't in anyway help overcome them. Time, however, did. It's been five long years, but I'm proud to say that I can finally tolerate green salads, and occasionally even enjoy them.

What this means is that I can spend the whole day eating whatever I want, as long as has roughly the right carb-protein-fat ratio, and at the end of the day, consume a giant bowl of salad (ready-made at the grocery, purchased in weekly installments) to get my portions of vegetables. So simple, I wish I'd realized it before.

Sleep. I submitted a paper last week, and by now I've gotten good enough with time-management that I don't need to cram, miss sleep, or curtail gym or social activities before the deadline. I can't seem to avoid drifting West when I have a deadline, though—I was off by 8 hours on the day we submitted.

Getting out of it is basically the same as beating jet-lag. Somehow I never realized that before, and used to try to drift East, which never ever worked. The right way, of course, is to miss a night of sleep, and force yourself to stay awake the next day until your regular bed-time. That seems to have worked this time. I'm hoping not to go right back to drifting again.

Remembering Dilip

2009-02-09T10:35:26Z

Dilip Veeraraghavan passed away earlier this week. He was one of the most remarkable people I knew.

I first met him when I took his class on Indian history back in college. It might sound absurd if you didn't know him, but that history class had a profound effect on my life. I had given up my faith only two or three years ago, and I still had a ton of unresolved questions about the culture I was raised in. Somehow, every time Dilip talked about an obscure historical event that happened 1,200 years ago, it would turn out to be intricately connected to some contemporary aspect of Indian society. At the end of the class, magically, I understood India much better, and for the first time, I was completely comfortable with my rejection of religion. His class gave me closure and a sense of identity as a free-thinking adult.

The wonderful thing was that he did it without ever telling us anything. He merely asked questions—questions so compelling we had no choice but to seek and find answers. A different person might have found different answers, and that was the way Dilip wanted it. Apart from the whole "finding myself" aspect, I still maintain an avid interest in Indian (and World) history as a result of that course.

My best memories of him, though, are not from his class. We later became friends, and I was one of the dozens (hundreds?) of students at IIT who would stop by his office whenever we had a free hour. He was always completely down-to-earth, and had a peer relationship with every one of us, rather than a student-teacher one. Remember that this was India, where hierarchy and social order are very important.

He was a communist, I was a libertarian. I think there were only two things we agreed on: George W. Bush and open-source software, the former being bad and the latter good :-) Nevertheless, we got along really great. That's when I learned that you don't have to agree with someone to respect them or to have a good time with them, something that I'm reminded of when I see homophily all around me.

He was great at keeping in touch with students even after they graduated. When I was in his office, he would often be interrupted by a call from some kid who had moved on to graduate school in America. I told myself I'd do the same, but while I did visit him whenever I went back to Madras, I never made the time to call. Even when I heard a few months ago that he wasn't doing well, I assumed he would get better, and that he would be there in his office the next time I visited, just as he'd been for the last 20 years, cheerful and ever-ready to launch into an argument about the evils of capitalism.

Guilt, then, was right up there along with sadness and shock on the list of emotions I felt when I heard he'd died. I was surprised to find that I wasn't alone—others, too, had intended to spend more time with him, but kept putting it off until it was too late.

There are so many of us whose lives he touched. The least we can do is to remember him properly. I've set up a wiki page to collect people's memories of him. I urge you to go there and share whatever you feel. [Note: let me know if you know a wiki site that has fewer/less obtrusive ads than this one.]

Why Craigslist needs to die

2009-01-27T18:44:46Z

Edit. There is a bit of name-calling and emotionally charged responses both here and on news.yc, where someone seems to have posted it. I guess I brought that on myself with my inflammatory tone, for which I apologize. But I stand by the content.

Craig Newmark, the Craig in Craigslist, is a do-gooder thoroughly uninterested in profits. When I saw him on Colbert, I got the feeling he didn't particularly aspire to anything in life. The site charges for job ads -- its sole source of revenue -- because companies that post them begged Craig to do so. This was because legitimate job ads were getting drowned out in a sea of pranks and frauds. Late last year, when the government essentially forced Craigslist to charge for erotic services ads, they decided to donate the revenues to charity.

Normally, that would all be just fine. The problem is that Craigslist is a monopoly. Let's get that completely straight: Craigslist is what it is because the site got lucky (having been founded back in '95) and quickly reached a position of dominance in a few different classifieds categories. Just like Microsoft did with Internet explorer until a couple of years ago, Craigslist has been able to stay on top without having to improve a single thing.

I have personally talked to multiple people who were planning or trying to build Craiglist competitors, and it became clear to me how much better the site could be, and how big a disservice Craigslist is doing the general public by not improving their technology and their interface. It was also clear that all of its potential competitors were intimidated by the task of trying to unseat a behemoth from its monopoly position in the face of the network effect.

When I talk about Craigslist's lack of technological progress, people assume I'm talking about the front page design, and get all defensive. Actually, that's one of the more effective aspects of the site's UI. Rather, I mean the lack of any sophistication in the search, the absence of any form of user reputation tracking, the unavailability of map views, the lack of an API that might enable third parties to provide any of these functionalities, and many, many other things.

And that's why Craigslist needs to die. I believe they are holding back a huge market segment in a significant way, and aren't going to change. Y Combinator is looking to fund a Craigslist killer. Whoever they end up funding, I wish them the best of luck.

Let me end by saying that the Craigslist story is a great example of why even if your goal is public service, a non-profit is often not the way to go because you lose your motivation to improve once the founders' personal hopes/aspirations/fantasies are realized.

Edit. Some clarifications. Many people have a problem with my calling Craigslist a monopoly. First of all, monopoly doesn't automatically mean "evil." I didn't call them the latter. As a matter of fact, though, they've shut down services which screen scraped them to provide value-added services. Perfectly within their legal rights? Of course. Good behavior? I don't know.

Second, in case it isn't perfectly obvious, all I'm saying is that it's time for a competitor to rise up by building something better. I'm not calling for a boycott or a government intervention of any sort. There is nothing hateful about my post, the title notwithstanding.

Third, many people have trouble imagining how you can make Craigslist far better. Let me paint you a picture. (This is a comment I made below, but worth promoting here.)

Let's say you're searching for apartments. Hit the street, forget Craigslist. Just start driving. As you drive, the apartment listings in the neighborhood will appear on your iPhone (or dashboard). It will integrate with GPS to tell you where is the next place to stop. When you come to the location of a listing, if you don't like the location/looks, etc., keep driving on. If you like it, pull up pictures of it right there.

I can similarly envisage making each activity on Craigslist far superior in terms of user-experience. As it is, Craigslist is more like a backend for data on top of which you'd imagine apps would have to be built, except that Craigslist won't build those apps nor let other people build them.

Further reading: Is Craigslist rotting on the vine?

Will someone please invent the germometer?

2009-01-23T06:12:48Z

I'm sick—the third time in two months—and it's not the flu. Each time it's some combination of cold, sore throat, headache, and occasionally a light fever. It wouldn't even be worth mentioning, except that not knowing where I'm getting the germs from leaves me with a feeling of helplessness that's much worse than the malady.

The last time I was sick, I tried spraying clorox on my keyboard (which has 400 times as many germs as my toilet seat.) All that that accomplished was to cost me my advisor a new keyboard.

I do realize I'm grasping at straws; it could be anything. I have regular germ-swapping activity going on, to begin with. I've also been working out almost every day this month, and gym equipment is so ridden with filth that it's a wonder how anyone who touches it doesn't fall sick instantly. I sometimes eat in my car and never wipe the interior. I could go on, but I'm sure you have enough unpleasant mental images already.

What I want is a germometer. I want a little hand-held device that I can point at things and get an instant germ-count reading. I know I'm at the very bottom of the germ-paranoia spectrum, so if I want this you can be sure that most of the population will. Especially overprotective parents (and is there any other kind these days?)

I actually don't think it's hard to build one. Take a microsocope-camera and stick an Eye-Fi card into it. This lets the user automatically upload a microscopic image of a surface to a server. Now all you need on the server is an off-the-shelf image classification library trained on a bunch of germ pictures. Any webmonkey can build a snazzy user-interface to display the blown-up images along with approximate germ statistics in a satisfyingly scary manner.

If you sell germometers for $200 a pop, can you imagine any responsible parent not buying one?

Note. A microscope camera with 400x magnification is $80 on Amazon. An Eye-Fi card is $50. You can easily see bacteria at 400x. Viruses, of course, are way smaller.. maybe some day.

Edit. I'm quite aware of the problems: many infections are viral, water or air-borne. Further, detection of viable micro-organisms is hard. Nobody is even trying to do it with microphotographs; this recent paper shows it can be done by suspending the substance in a liquid crystal.

The primary benefit that I see is a feeling of control, instead of one of helplessness, which, as I alluded to in the first paragraph, I'd pay good money for. Such placebos can have an actual and significant effect on health. Have you noticed how every single ad for a diswashing liquid shows you before and after pictures of germs? Now you can see that for yourself.

The second benefit is that even if the results are unreliable, it's a good starting point to figure out what needs cleaning around the house. I mean, in spite of the studies, people don't seem to believe that their mobile phones are dirtier than anything else they come into contact with. Having visual confirmation would help greatly.

Just so we're clear

2009-01-20T04:34:56Z

A friend of mine says* he believes that people should have a right to a comfortable life even if they choose not to work. You already know I don't agree with that; in fact, I can't think of any other viewpoint that I disagree with more strongly. (I mean, sure, I might disagree with racism more, but then I don't think of it as a legitimate viewpoint and I don't respect people who advocate it.)

To me, choosing not to work is morally reprehensible. I was raised that way, and it's one of the things about my parents' belief system that I've chosen to keep. I don't advocate anything as extreme as denying emergency medical treatment to the wilfully unemployed, but I don't think they deserve any of the comforts of life at the expense of others. (If you're idle rich, I might find that unpleasant, but at least you're not a drain on society.)

If there were a technology that allowed us to put people in hibernation, wake them up once a year and ask them if they were ready to start working, and shove them right back in there for another year if they said no, then I might be okay with using it on people who can't support themselves and choose not to work ;-) (Obviously, being unable to work or unable to find work is a very different situation.)

I was on the phone with my grandma the other day. We don't normally connect very well conversationally, since our lives are so different. On this particular occasion, however, we re-discovered that we both believed in the importance of working until you dropped dead, and that nothing else mattered in life. I'm exaggerating a little bit, but only a little. It was great.

*I can't link to it because it was a friends-only post. Own up to your opinions, people!

More of my crazy views on brain drugs

2009-01-14T20:07:00Z

Brain pills for healthy people are for the most part currently illegal. Some scientists—by far the minority—are calling for change.

I find it disturbing that this is even controversial.

First, the notion that there is an optimal "healthy" state is make-believe. We all get old and die, so in a sense we are all terminally ill. We should have a right to improve our minds and bodies as long as there is room for improvement.

More to the point, brain types fall on a spectrum. ADHD is just one extreme. Equating "median" with "healthy" is a fallacy. Consider this: is synesthesia a disease? It can make life very hard, but it can also make the patient extremely creative. Imagine a world where synesthesia is the normal condition—non-synesthetes would be considered retards!

If there were a drug that could turn you into a synesthete, should the drug be illegal?

Second, most of us already take a brain-boosting drug: caffeine, which is both undeniably effective, and has a wide spectrum of side effects. Other drugs like modafinil have almost zero side-effects in comparison, but they are prescription-only because they don't have a history of social acceptability. "Used historically" = "safe" is an even bigger fallacy.

Third, you can't prevent people from taking brain pills by not funding research (if the research doesn't happen in the U.S, it will happen in other countries.). All that our current policy will achieve is encourage a black market and unsafe usage.

The whole thing reminds me strongly of this Kurt Vonnegut piece.

There's a good bit of discussion where I originally posted this. Happy to hear your comments as well.

Previous writing: The Calculus of Caffeine Consumption, The Brain User's Guide to Pills.

Heuristic Intelligence FAIL

2009-01-14T06:16:00Z

Facebook trying to be extra helpful and auto-detect names in my post. Cute :-)

Python puzzle solution

2009-01-09T05:37:34Z

In my last post I asked why the expression >>>''r''=="" evalutes to True in python. As I expected, the responses were a mix of "duh," bewilderment, and partial answers :-)

Ok, so here's what's going on. The first thing to notice is that I have two pairs of single quotes on the left and a pair of double quotes on the right. With some browser fonts, the two look almost identical, whereas with others the difference is obvious. One way to see the difference clearly is to copy-paste the text into something that uses a fixed-width font, such as the box where you type in comments :-)

The next thing to note is that the expression ''r'' is tokenized as '' followed by r'', and the two tokens are concatenated—python has implicit string concatenation, somewhat like bash (except that in python you can have spaces between the strings, and bash doesn't distinguish between strings and commands, which Tcl takes to an extreme.) Try this in bash sometime: $ "l"''"s"

But what's r''? That's a python "raw string," which means that backslashes don't get interpreted, although that doesn't change anything here.

The other things to know about python strings are that triple quotes are also valid (they are multi-line quotes), and that you can multiply strings. You can probably guess what this evaluates to: >>>"ha "*3

You can do insane things combining by combining single, double, and triple quotes with raw strings. Here's something I came up with in 5 minutes, I'm sure you can do significantly more horrifying things if you put your mind to it. Do any two of these evaluate to the same thing? (Try it out!)

>>>""'''''''''''r''''''''''', ''""''''''''r'''''''''''', ''""'''''''''r''''''''''', ''''''""'''''r''''''''''', ''''''''""'''r''''''''''', '''''''''''""r'''''''''''

Together with the signficant whitespace that python is notorious for, no serious language has made you work so hard to understand it's lexer!

Puzzle

2009-01-07T05:19:22Z

$ python
>>> ''r''==""
True
>>>

Points only if you figure it out without trying it in a shell :-)

Comments screened. Edit. Now unscreened.

Mathematicians don't understand people?

2008-12-31T19:48:08Z

This is again a comment that I posted elsewhere, in response to the following question:

Someone tells you they have two children, and one of them is a girl. What are the odds that person has a boy and a girl?

My answer, reproduced below, got voted up like crazy—I guess it struck a chord with nearly everyone. I know that many of you are math people.. I look forward to your vehement objections :-)

This is a famous puzzle. The answer is supposed to be 2/3, because what the question is asking you to do is consider all the parents in the world where at least one of the two children is a girl. Then you're left with 3 possibilities, BG, GB and GG.

If you phrase the question like that, everyone will get the right answer. The reason people get it wrong is that people don't normally talk like that. Imagine you're at a party, and someone tells you they have two kids, and "one of them is a girl." Clearly, they mean that the other is a boy, which means the answer is 100%.

But the most intuitive way of interpreting the question is that you know that a specific child is a girl, say because the person brought one kid to the party, who turns out to be a girl. With this interpretation, the obvious answer of 50% is in fact correct.

You often hear the complaint that people don't understand math. In this instance, however, an equally valid way of explaining what's going on is that mathematicians don't understand people.

This criticism applies partially to the normal game-show version of the Monty Hall problem, but I think there the wording is genuinely ambiguous regarding the host's behavior, and my answer would be "not enough information."

Seriously, when is the last time anyone told you that at least one of their kids was a girl? You could imply it, for instance by saying "I need to pick up my daughter from school," but I cannot think of a phrasing in English that lets you state it directly. Not without sounding like a huge dork, anyway.

Edit. For the record, the missing information in the Monty Hall problem that I'm talking about is, "does the host only open doors which he knows not to have the prize, or any door other than the one the contestant picked?"

Sex differences in math/science

2008-12-30T17:09:17Z

I've been posting a lot on other forums lately and I'm going to repost things here occasionally, because I enjoy your comments.

This is what I feel on the issue of intrinsic sex differences in math/science ability:

What I fail to understand is why this is such a huge issue. Whatever sex or gender differences there may or may not be, it is clear that they are not sufficient to explain the vastly skewed sex ratio we find today, and that bias plays a big role in explaining it. So why not work together to eliminate the bias instead of getting bogged down in divisive debates?

As I see it, if and when participation levels in math and science reach, 55-45, or even 60-40, then it will be time to talk about whether that last 5-10% is due to bias or intrinsic differences.

Oh, by the way, I guess I've toned down my views on the sex ratio in the VC industry, mainly as a result of the comments there :-)

So what do you think?

Writing papers is only half of your job

2008-12-22T04:44:01Z

I was reading Yes!—an excellent book, and contrary to initial appearances, not about slimeball marketing tactics at all—when I had the curious feeling of reading my own words. I realized the paragraph I was looking at had been lifted from the Fischer-Spassky World Championship article on Wikipedia that I wrote several years ago. Which is totally awesome.

I wouldn't be surprised if a million pairs of eyes have pored over the words I've written on Wikipedia—my heaviest editing was in 2003, back when most of you hadn't heard of it, and therefore had a much greater footprint than current contributions would. It gives me a warm fuzzy feeling because I often think about whether or not what I'm doing has an impact on the world.

I'm sure that my contributions to Wikipedia, in terms of the time people have spent reading them, dwarf my research work. You could argue that research is a specialized activity, whereas if I hadn't edited Wikipedia, someone else would have put in the same information. But my point is that research papers aren't disseminated nearly widely enough.

Academics are happy to win points by circulating papers among their colleagues via conferences and journals, but there is a huge bottleneck when it comes to propagating that knowledge further. The few researchers who blog about their work perform a valuable service, but the vast majority of us aren't bloggers. On the other hand, journalists usually do a horrible job of describing research because papers aren't written with a layperson in mind.

But things are changing. Videolectures.net has been making talks+slideshows available online in a very appealing format. There are a few blogs dedicated to accessible explanations of breaking research; This Week in Evolution is a great example. There is a concerted effort to improve Wikipedia coverage of theoretical computer science. Finally, the RNA Biology journal is requiring authors to create Wikipedia articles about their submissions. Woohoo!

Anyone submitting to a section of the journal RNA Biology will, in the future, be required to also submit a Wikipedia page that summarizes the work. The journal will then peer review the page before publishing it in Wikipedia.

Academia is moving, albeit slowly, into the 21st century. My latest hobby project is along these lines—stay tuned.

Losing my traffic ticket virginity -- not fun at all, but at least I'll remember it

2008-12-18T07:38:13Z

I got a speeding ticket a few months ago and the Defensive Driving paperwork was due this Monday. I registered online for it a couple of weeks ago, got started, found out how mind-numbing it was, and decided I'd get to it on Sunday so that I could print out the certificate and take it in on Monday.

So I got to it on Sunday and realized that you need a physical certificate, and that even with the overnight delivery option, I wouldn't be able to get it before Tuesday. Dang.

I figured I was going to have to pay the $200 fine and accept a slightly higher insurance premium. I drove down to the court at 2pm on Monday, and as I was pulling out my credit card, the clerk asked, "you know this means you're going to have a criminal record, right?"

"Wait, what?"

It was a Class C misdemeanor, apparently, making me a Grade A moron. I resigned myself to the fact that I was truly and royally fucked, and asked her if there's anything at all I could do. "Well, that all depends," she said. "Do you have access to a computer?" I'm surprised how often I get that question. If she hadn't been in a position of absolute and complete power over me, I would have been tempted to let loose some snark.

"Hold on, let me pull up the calendar on my phone and check the century.. it's the 21st. Yes I do!"

or

"Well, I have four personal ones, and access to a couple of hundred at my University. Will those do?"

Anyway, she said she'd give me two extra days to turn it in, but the completion date on the certificate couldn't be late. At this point I realized the true enormity of my stupidity: I had 3 hours until the day ran out at 5pm, and the course would take 6 hours. If I'd at least gotten started the previous day, I would have been able to make it.

With nothing to lose at this point, I went back and spent the next 6 hours listening to the course, hoping that somehow it was all going to work out. Mind-numbing doesn't quite being to describe it—I think I have PTSD from it.

At this point things started to go right. At 9am the next morning, the defensive driving people woke me up with a phone call. Apparently I'd forgotten to put in some of my personal information. Wow. It was like everyone was trying to save me from myself.

What's more, even though the certificate wouldn't be processed until later that day, they'd record the completion date as the previous day. They'd overnight the certificate, which would reach me by 3pm the following day, and that would give me 2 hours to take it to the court. Piece of cake. The other bit of paperwork I needed was my driving record, which you can get from the Department of Public Safety for 10 bucks. There are websites that seem to scam you into paying $25 for it.

The next day, I kept glancing at the clock on my computer every twelve seconds on average, listening for that fateful knock on the door with my ears pricked up like a hunted deer, although hope started to wane as the afternoon wore on. I didn't even think of showering in case I missed the Fedex guy. 3pm— Nothing. I barely had time to start imagining the consequences, when, in a scene reminiscent of the Deliverator, I heard footsteps on the corridor followed by a pounding on the door. My computer said 3:00:31.

So that ended well. I've been in this country five years and I still haven't gotten used to how helpful government employees generally are. Mad props to Comedy Defensive Driving, the clerks at Travis Country Justice Court Precinct 5, Fedex, Austin DPS, and Vitaly (for talking sense into me and getting me to do the course even though I didn't know if they'd take it.)