Arvind Narayanan's journal

The socialite game

2009-11-27T07:09:09Z

Socialites are often famous for being famous. Although this description is used pejoratively, a variety of human activities center around social games where one gains power by associating with those who are already powerful. Power is defined purely in terms of the network of relationships, with no inherent measure of skill, quality or worth of an individual.

I will try to formulate this game mathematically. The hope is that the analysis will have some ability to predict and/or explain human social structures and behavior.

Consider a graph with N nodes or players. Players compete by creating or destroying edges in the graph. Mutual agreement is required for an edge to exist. Players have limited social capital, i.e., there is a bound on the degree of each node. All players have the same degree bound, which captures the fact that there is no intrinsic measure of strength of a node.

The goal of each player is to maximize their (relative) centrality in the network. This can be captured in various ways, but a natural measure is Eigenvector centrality, which is basically PageRank.

Unfortunately, as presented, the game is trivial because of the algebraic fact that the only Eigenvector with no negative values of the adjacency matrix of a regular connected graph is (1, 1, ... 1). Therefore socialites cannot exist—everyone is equally popular.

I see two main ways to make the model more realistic and make socialites possible: one is to make edges directional (and weighted.) This captures asymmetric power relationships between individuals, and enables hierarchical tree structures, among other things. The other is to remove the restriction that every player has the same degree bound. This would let us ask, "if some people have more social capital than others, can they leverage it to become vastly more influential?"

I played around with these alternatives for a few hours, but couldn't find what I wanted—a model that is halfway between the studies of game theorists such as a network creation game (which is too abstract to imply anything about human networks) and the techniques of social anthropologists, such as the well known 1977 Karate Club study by Zachary (which is not abstract enough to prove anything of general importance).

So I'm just going to leave this here on the off chance that it might inspire someone to take it forward.

[P.S. Two concepts that I think are important are stability and connectedness. A graph is k-stable if no k players can collude to improve each of their scores. By connectedness I refer to the standard graph-theoretic concept; I think it is important to study which types of of initial conditions result in connected vs. disconnected social graphs, i.e., co-operative vs. competing power structures.]

The death of the printed book is closer than you think

2009-11-27T00:21:51Z

I've been saying for a while that the e-books are going to take over soon. Let me elaborate on that, now that I have some data to back up my claims.

First, let's get the obvious stuff out of the way. The Kindle seems to be following roughly the same adoption curve as the iPod. Barely two years after it was first released, everyone my age has at least played with one or knows someone who has one. Amazon has been pushing it massively and adoption is only going to accelerate with the recent price cut, international availability, and the emergence of serious competition.

Bezos announced back in May that 35% of book sales are on the Kindle when a Kindle version is available. How can that be, when penetration is still small in absolute terms? It's because Kindle owners are disproportionately voracious readers.

But my real point is about digital-only books. Let's ponder the consequence of the above 35% figure. An author reveals her numbers from being on the NYTimes bestseller list. The most striking number to me is the fact that her royalties are only 6-8%. I assume that the number is roughly the same for sales of the Kindle version. (If anyone knows otherwise, please let me know.)

On the other hand, Amazon shares 35% of revenue with the author for self-published books. In one sense that's unfairly low: Apple for instance shares 70% of revenue with app publishers. Still, it is five times higher than royalties from a traditional book publisher.

Let's do some math. Consider a typical book that sells for $14.99 in print and $9.99 on the Kindle (the only thing that matters here is the ratio of 3:2. If the ratio is closer to 1, then my argument is even stronger.) Let's assume that half of all sales are via Amazon, and the rest are through physical bookstores. That means that Kindle sales are 35% × ½ = 17.5% of the total. Let's say an author were to self-publish digitally with Amazon, and thus forgo all non-Kindle sales, but maintain the same volume of Kindle sales as they would get with a publisher. Their revenue with the self-publishing route would be 0.175 N * 9.99 * 0.35 = $ 0.61 N. With the traditional route, the revenue would be (0.175 N * 9.99 + 0.825N * 14.99) * 0.07 = $ 0.99 N.

Conclusion: Kindle penetration is already three-fifths of the way to the crucial tipping point, where kicking out your publisher generates more royalties. This is no doubt a simplified model, and ignores several factors:

The publisher gives you an advance, which might make it attractive to authors without a financial cushion.
The publisher has an advertising budget to spend on your book.
On the other hand, without a publisher, web-savvy authors would be better able to leverage social media to get the word out about their books.
Amazon will probably start sharing more revenue with authors, making the self-publishing route even more attractive.
On the other hand, publishers will probably increase royalties for Kindle sales, due to the same pressures.
I've ignored the fact that you can self-publish on multiple platforms, although none are as yet competitive with the Kindle.

In spite of the shortcomings and shortcuts, I think my model provides a good ballpark estimate, and I trust the prediction that we are close to a tipping point. Just as Radiohead generated a lot of free publicity (and hence extra sales) by breaking ground with their "pay what you want" model, the first major author to self-publish will generate a lot of publicity. I predict that we're no more than a couple of years away from this happening.

Far more interesting than what digital books will do to the head of publishing is what they will do to the tail. The idea of the struggling author trying to land a deal is a cultural staple, but one that exists purely because publishers have had a monopoly on distribution channels. With ebooks, someone who thinks they are a great writer doesn't have to wait and beg to be discovered—they can find out for themselves by self-publishing, promoting their work on Facebook and Twitter, and seeing what kind of response they get.

People will continue to read printed books for a long time, just as some people still watch movies on VHS. But the printed book will be "dead" in a few short years in the sense that the bulk of the adoption curve, the pragmatic majority, will have moved on. For the first time in history, the discovery of writing talent will depend more on skill and persistence than on luck. And the notion of the book itself will morph to occupy an entire spectrum—traditional linear, textual narrative on the one end and videogame-like interactive, graphical narrative on the other.

I can't wait.

Update. Here is another author who reveals his numbers, including Kindle royalties and self-publishing revenues. His calculations are similar to mine, as are his conclusions:

I don't think I'll ever take a print contract for less than $30,000 per book, because I'm confident I could make more money [with ebooks] over the course of six years than I could with a publisher over six years.

Isn't that bizarre?

For the bestselling author, this is all still very trivial. These numbers are chump change compared to the advances they get.

But for the midlist author, I'm beginning to think it's possible to make a living without print contracts.

I've struggled mightily to break into print. And I've made a nice chunk of change on my print novels.

Now I'm hoping those novels go out of print, so I can get my rights back.

I never would have guessed my mindset would change so dramatically in so short a time.

Putting your money where your mouth is

2009-11-15T21:22:07Z

I often marvel at the human capacity for self-deception, one aspect of which is the fact that people often haven't a clue what they really believe. There's no better way to demonstrate this than to get them to bet on something they claim to be a sure thing.

Dick Lipton tells the depressing story of trying to get another prominent computer scientist (Ken Steiglitz) to bet on P != NP. Steiglitz started out claiming the odds were a million to one, but when forced to bet, he wouldn't take odds longer than two to one.

Two to one. So a respected scientist was 500,000 times less certain of his opinion than he claimed to be — on a question that was within his sphere of competence. What does that tell us about the certitude that we each feel on issues relating to the economy or the environment? We're simply lying to ourselves.

A few years ago, when intrade was popular, I went around asking people why they weren't cashing in by betting on their 'sure' beliefs. If they were right, they could double their money in mere months. It was amusing to listen to people's rationalizations and excuses.

I think it would be fun exercise to bet on one's views, regardless of whether one feels certain of them or not.

Detecting a tech bubble

2009-11-09T20:28:51Z

The graph below compares the NASDAQ with the Dow over the last decade and a half. The tech bubble sticks out like a sore thumb, which makes sense since the NASDAQ is technology-dominated. On the other hand, the current recession affected both indices in exactly the same way, since it wasn't tech-related.

Back in 2007, the Richter Scales made a famous video which made the point that we were in another tech bubble that was about to pop. It was scary, and at the time I didn't know whether to believe it or not. But now I wonder if the NASDAQ — DJIA spread might be a simple and reasonably reliable indicator.

I'm not very knowledgeable about finance and the economy; any thoughts?

Funny bug

2009-11-09T17:26:27Z

It's winter, and naturally, my bathroom floor is cold in the mornings. When I step into the shower, and warm water hits me, it apparently confuses the hell out of the temperature sensors in my feet, and I feel like I'm standing on ice and fire at the same time.

Bookmarklet to bypass NYT registration

2009-11-09T05:12:55Z

Bugmenot stopped working for me yesterday for no discernible reason. The New York Times is the only website I use it for, so I whipped up a little piece of Javascript to bypass the registration on NYT.

javascript: ( function() { window.location.href = "http://www.google.com/url?sa=t&url=" + window.location.href.split('URI=')[1] })()

The way to use it is to create a bookmarklet and paste the code above as the URL. When you get to a NYT "registration required" page, click the bookmarklet and it will let you in.

Even though the NYT seems by far the most web-savvy of the major newspapers, they fundamentally fail to understand the web and won't survive for long in their current form. The same goes for the rest of the newspaper industry. Numbers don't lie. In the mean time, enjoy the bookmarklet.

We are still a society of nature-worshippers

2009-10-10T03:43:45Z

Evolution has come up with many, many clever designs over the eons, and engineers have a lot to learn from studying nature. On the other hand, it is equally true that on average the designs in nature are riddled with inefficiencies, bugs and tortuous mechanisms at every level of complexity. Most people don't realize this, and are in fact repelled by the idea.

Since I've been learning about the human genome as part of my current research project, I often find myself explaining to other computer scientists how some aspect of genetics works. At some point they interrrupt me to ask, "but wouldn't it be way more efficient to instead do... ". When I tell them it certainly would, but that evolution has never managed to figure it out, they are usually surprised. But it's true — evolution has explored only a tiny, tiny part of the design space. I find it ironic that if only scientists did a better job of pointing out all the ways in which nature has failed to find good designs, the man on the street would have an easier time believing that there is no intelligent designer.

In particular, co-operative strategies never occur in nature unless it is a game-theoretic "stable equilibrium" — that is, even from a purely selfish perspective, it must be advantageous to follow the strategy (this is a slight oversimplification.) Worse, the strategy needs to be beneficial not to the individual, but rather to the genes. This is an highly unintuitive idea to wrap one's head around, and it is very easy to fall back into fallacious ways of reasoning even after learning it. (The Selfish Gene is still the best and most enjoyable text on this, 33 years after publication.)

I could go on talking in the abstract, but I'm not going to change anyone's mind. Let me instead leave you with a fun excerpt to chew on from Lions: A Photo Essay that provides insight into the reasons why we idolize nature:

Anybody who has seen a documentary "knows" that lions hunt cooperatively to bring down prey. Unfortunately, nobody seems to have told the lions this. Indeed, for many years field biologists who study lions have realized that cooperative hunting is an illusion. ... So, how come the Discovery Channel says they are cooperative? Partly it is because these figures are buried in Appendix B of Shaller's book, or in dense academic papers. Mostly it is because the story of cooperative strategy in hunting is so endearing to people. Especially to film editors, which means we are destined to seeing cooperation in every nature documentary. The cases where the hunting fails due to lack of cooperation end up on the cutting room floor.

Watching lions hunt, the trends are quite obvious. The primary reason that groups of lions are no more effective than two by themselves is that typically only two lions do the actual hunting. They all make a show of hunting, but in the cases I watched, in several different prides, there were always a couple females that were the most aggressive and took the lead. The others hang back for the hard part then rush up at the end after the worst danger is over. Their primary goal is to be at the kill early so they can eat, not to actually help. Field studies have confirmed that lions do not seem to keep track of this and punish slackers.

Lions can seem quite inept at hunting, because they have no way to communicate complicated information. The Discovery Channel case happens when one lion flushes prey in past another for a perfect catch. More often, what happens is than one lion blows it and scares the game too early, or flushes it in the opposite direction. After watching hunt after hunt fail, you soon decide that lions are not very coordinated. Indeed their only saving grace is that the buffalo can't communicate very well either.

And this is just delicious:

Lions don't wait to kill the animal before starting the process of eating it—as soon as the buffalo stops thrashing, lions start to eat. This is much harder than it sounds however, because the hide is very thick. The prime spot to start is always claimed by the dominant female, or if the male is there, he takes the prime spot. The prime spot is not what you might think—it is the rectum. Believe it or not, the king of beasts starts his dinner by carefully licking the rectum clean. Since the buffalo defecates while dying this is a bit messy. The lion then works very hard to gnaw through the skin and get an incision open.

Sorry, couldn't help it :-P

Haskell

2009-09-16T10:07:45Z

I started learning Haskell today. After a couple of hours of learning the syntax, I decided to dive in and write a function to shuffle a list, because linear-time permutation is non-trivial in pure functional languages. (The standard idiom translates to a quadratic algorithm because it is fundamentally destructive.)

I think I can get linear-time by assuming that the associative array operations in Data.Map are O(1), but in reality they have a logarithmic cost, since they are implemented as binary trees unlike Python's dictionaries which use hash tables. Besides, this was literally day 1 so I wasn't comfortable jumping into Data.Map.

So I ended up with something not particularly satisfactory, but I did learn some Haskell in the process. Any comments/criticism greatly appreciated.

Edit. Does anyone know if linear-time permutation is even possible in a functional setting? I thought I found a paper that did that but it turned out to be a simpler problem.

Bigger delicious.com dataset -- 1.25 million entries

2009-09-14T22:55:17Z

As promised, here's a much bigger dataset of delicious.com bookmarks with around 1.25 million entries, weighing about 170 megs gzipped. I'm no longer collecting data, so this will probably be the final release. It includes every bookmark in a more-or-less contiguous period spanning the last 10 days or so.

Folksonomy dataset for NLP (delicious.com bookmarks)

2009-09-07T11:03:53Z

Several of my current projects have to do with Natural Language Processing. Over the weekend, I built a topic categorization engine; one of the datasets that I used was bookmarks from delicious.com. I was mainly interested in the tags and the relationships between them.

Delicious doesn't make this data available for bulk download, but they do have an RSS feed of site-wide bookmarking activity. The average rate is slightly over 1 new (public) bookmark per second. I've been slurping up the feed for the last couple of days, so I have a dataset of around 200k bookmarks. You can grab it here. The format is JSON, one entry per line — trivial to parse.

I plan to leave my bot running until I have at least a million entries, at which point I will do another release. If anyone wants the source, I'd be happy to share, although it shouldn't take more than half an hour to write it yourself.

To learn more about the nifty data-mining possibilities of social-bookmarking data, see folksonomy. If you find the data useful, drop me a note and let me know. Enjoy.

How to sample in constant time

2009-07-30T08:24:40Z

I can't remember the last time I felt like a bigger idiot than I do right now.

I spent like forever writing up my ugly solution to the constant time sampling puzzle. With exercises for the reader, no less. Dear god, the pomposity. Just as I was about to post it, I found this extremely elegant solution that can be explained in 30 seconds (slides 41-48).

Of course, I never thought I'd discovered something new; that would have been downright delusional. On the contrary, I thought this must be folklore knowledge, and therefore perhaps never actually published. I did some Googling before I started writing and even asked a couple of statisticians, but got no answer. I guess I just didn't look hard enough.

On another note, this is also a sad reminder of how much my puzzle solving skills have degenerated since my IMO days.

At any rate, I encourage you to have a hearty laugh at my expense.

Edit. Fixed broken link.

Programming puzzle

2009-07-24T17:40:06Z

If you're a programmer, I suggest giving this puzzle a shot because you're likely to encounter this problem a few times in your programming career :-)

You're given N objects and a probability distribution function over these objects specified as a an array [x_1, x_2 ... x_N] of arbitrary nonnegative real numbers. Write a function that generates independent samples from this distribution. An O(N) pre-processing step is allowed, but after that, each new sample should be generated in expected constant time.

Any takers? Too easy?

Edit. Assume that you can generate a uniform real number in the interval [0, 1] at unit cost.

Theory blog aggregator: stats and directions

2009-07-20T07:08:36Z

Traffic to the Theory of Computing Blog Aggregator has more than doubled over the past year, and is now at 16,000 visits/month.

Visits / month

Incidentally, the readership is tech-savvy, as you might expect — more than half the visits come from Firefox, Safari is the second most popular, MSIE barely breaks 10% and Chrome is almost tied with MSIE. Opera traffic is a rounding error, and bots aren't shown on the chart.

Visits by browser: Jun 2009

Given that Theory of Computing is a fairly niche community, the amount of traffic is surprising for a site that was slapped together with a weekend's worth of effort. Combined with the fact that the RSS reader is dying, it seems clear to me that topic aggregators have a great future.

So I'm wondering if it's time to update the aggregator with some new features. The problem with this is that I already worked on a snazzy Web 2.0 UI (hit '?' for keyboard shortcuts), which turned out to be much worse for usability than the current avatar.

Without going that route, here are some of the things I'm thinking of doing:

Write special-purpose code to import comments from Wordpress blogs. Currently the aggregator imports Blogger comments but not Wordpress, because the latter works in a different way. Blogger and wordpress cover the vast majority of the blogs, so that's all I really care about.
Polish up the recent comments feature (see the Web 2.0 UI link above for a demo) and implement it on the main site.
Import TOC-related tweets. I will use a special hash-tag, (#cstheory?) to import. This will be combined with either a blacklist or a whitelist of twitter users, in case anyone tries to abuse it.
Set up a bunch of other aggregators.

Thoughts?

Amazon is not the enemy

2009-07-18T00:40:27Z

Everyone who's flipping out about the Orwellian Kindle fiasco needs to relax.

Yes, Amazon did something stupid. Yes, DRM is bad for consumers and bad for society. But Amazon is not the enemy here, the publishers are. Right now, the balance of power is still with the publishing houses, so Amazon needs to play with them. When it comes to DRM, people at Amazon "get it." They're techies, after all. If they do evil things, it's because their hands are tied. It is the publishers who are pushing for DRM. Market forces are going to push them out of existence, and they will do all they can to prolong their misery.

Dramatic as the current incident may be, the same story has already played out with Apple and the iTunes store. Apple bent over backwards for the labels in order to sign them up to sell their wares online, but once they came on board, the power shifted to the technology companies, and Jobs turned around and attacked DRM (the "Thoughts on Music" letter.) By now, just two years later, everyone agrees that music DRM is on its way out and the future of recorded music is Free.

So please do express your outrage, but make sure you have the right target :-) Do complain to Amazon about how much DRM sucks, go write reviews of the product on their site and others, but don't "boycott" the Kindle. That's just plain counterproductive — books aren't magically going to become DRM-free without first becoming digital, and there's no way that's going to happen except under whatever terms the publishers choose to impose.

There's still quite a ways to go before all the major publishers succumb to the pressure of growing Kindle sales and start offering their content digitally. For that to happen, the Kindle userbase needs to grow a lot. Paradoxically, now, more than any other time, the Kindle needs your support.

Oh, and while I'm at it, can we stop making silly comparisons between the Kindle and Sony's or whoever's two-bit book reader that no one's ever heard of? Without the EVDO network and Amazon's catalog behind it, it's not even the same category of device as fas as the average person is concerned. Thanks.

Dreams and the self

2009-06-22T16:58:07Z

In last night's dream, a T-Rex was trying to eat me. It went on for the longest time, it was horrible. I wish the T-Rex had cornered me at some point and said "game over, motherfucker!" so that I could have gone on to the next dream or whatever. But that's the funny thing, apparently you can never die on your own dream. I certainly never have.

I frequently fly in my dreams (this has been by far the most common recurrent motif); I hear that a significant proportion of the population does this as well. Do you? I wonder what it means for humanity as a whole.

Another thing that fascinates me about dreams is that since your conscious processing is subverted, you can pretty much simulate multiple entities in a way that you can't when you're awake. For example, this dude I was once talking to in a dream sat with his hands firmly planted in his pockets, and I was wondering why, until he pulled out a gun on me much later. Of course, my brain knew all along why he had his hands in his pockets -- both my character and the adversary are being simulated inside my head -- but it chose to keep that fact from me, just for shits and giggles. Or sometimes my character would need to sum a sequence of numbers, and find to his surprise that the sum is exactly (say) 100. While I can of course sum numbers in my head, quickly coming up with a sequence of random-looking numbers that sum to 100 seems beyond my wakeful arithmetical ability. The only conclusion is that my brain was working on the sum for a prolonged period of time -- and choose not to tell me.

The amazing, amazing thing is that even though almost everyone goes through these experiences when asleep, almost no one realizes that their waking consciousness is also a similarly fragile illusion. Even seeing the breakdown of coherent consciousness in other people -- such as patients with a severed corpus callosum, who develop two distinct personalities -- doesn't seem to help. Nor do studies showing that our consciousness is merely "informed" of the decisions we make, said decisions actually having been made in the subsonscious several seconds before we are even aware of making them. People have a remarkable ability to ignore any evidence that contradicts their model of the self.

If it isn't enough that most of the problems in the world are caused by our outsized egos and our little worlds that revolve around ourselves, chew on the fact that the basis for ego rests fundamentally in a fallacy :-)

Bioinformatics needs to be democratized

2009-06-15T19:39:27Z

I've started working with genome data, so I've been giving myself a crash course in genetics, especially the human genome, over the past week or two. I'm almost up to speed — I can now understand paragraphs like this without needing to look anything up:

We identified a 47 Kb interval containing an Alu insertion polymorphism (DXS225) and four microsatellites in complete linkage disequilibrium in a low recombination rate region of the long arm of the human X chromosome. This haplotype block was studied in 667 males from the HGDP-CEPH Human Genome Diversity Panel.

Just two weeks ago I would have been completely lost. That isn't your basic Mendelian genetics either, it's a research paper from 2007.

Learning a new set of concepts over a short period of time by immersing yourself in it is an intense experience, and one that I thoroughly enjoy. That's why privacy research has been so rewarding for me — it has given me a chance to read tons of papers in law, economics, sociology and now genetics, not to mention many, many areas of computer science.

My main aim was to understand the math behind meiosis. While it isn't very hard, it is still an area of active research, and our knowledge of it is incomplete. Consequently, there doesn't seem to be a way to get to it without cutting through layers of biology. Even the really basic stuff, like chromosomal crossover, is generally explained in a tedious way using observations of traits in plants and fruitflies. This is because of an accident of scientific history — until DNA sequencing became a reality, the only way to learn what happens during meiosis was to observe the results when animals mate and make inferences about the genome based on that. It's like the story of the blind men and the elephant.

More generally, bioinformatics seems to be populated by people whose background is primarily in genetics rather than in computer science. When computer scientists do study the genome, they end up spending their time on something inane like proving the NP-hardness of some computation on genomic sequences, apparently oblivious to the fact that human DNA does not consist of worst-case strings constructed by malicious adversaries! Even on basic information-theoretic questions such as the amount of entropy in the human genome, the best I could find is a marginally related, speculative blog post.

Someone needs to write "The Facts of Life, for Computer Scientists." I'm confident that any competent python hacker with a solid knowledge of algorithms and statistics could read that document, learn the basics in a few hours, download some data from HapMap, install a library of useful tools with a single apt-get command, and start producing useful code and generating interesting hypotheses, all in the course of an evening!

Bioinformatics needs to be democratized. There is a gigantic amount of data available, but the people who are producing the data aren't necessarily the ones best equipped to play with it. On the other hand, there is a huge community of hackers who would like to do just that, but don't realize how easy it is. If you could get these two groups to talk to each other, amazing things can happen.

Why are there so many theory bloggers?

2009-06-09T16:35:13Z

Theory seems to have by far the most bloggers of any subfield of Computer Science; I'm sure I'm not the only one wondering if this is more than coincidence. Here are a few possible reasons:

Theory is cohesive. Most pairs of theorists find each other's work somewhat interesting. At the same time, the field is unlike most of Computer Science, with its emphasis on proof and disregard of experiment. I doubt that this level of cohesion can be found elsewhere in CS: "systems" is too vague and broad, while most other areas — AI, data mining, information retrieval, semantic web, databases, logic, formal methods, programming languages, compilers, architecture and graphics — fall on a massive spectrum.

Missing from that list are crypto, security and privacy. Cryptographers, in my experience, are generally horrified by the prospect of saying anything publicly that isn't heavily peer-reviewed, so that's out. Security and privacy are highly interdisciplinary, so that's not a cohesive subfield either.

In a small way, my blog aggregator may have contributed to the cohesiveness of theory bloggers by fostering a sense of community.

Theory doesn't get press. Except for the occasional journalist making an amusingly feeble attempt to explain P =? NP to the lay public in the context of the Clay Math Institute prize, theory stays out of the press because it doesn't generate pageviews.

In most other fields, important papers have a non-zero probability of getting written about (in the case of graphics papers announcing new techniques, it is virtually guaranteed since they are accompanied by jaw-dropping animations.) Consequently, theorists need a way of spreading the word about papers that are important/interesting. Word of mouth and best paper awards only go so far in the 21st century.

Theory is hard. Don't get me wrong — I'm sure research in other fields is just as hard to perform, but in my opinion, theory papers are particularly hard to read, especially for newcomers, simply because of the high degree of abstraction. This gives theory authors a strong incentive to explain and motivate their papers in more readily understood terms, and blogging is a great way to do that.

All that said, a big part of it is probably pure chance. Specifically, Lance Fortnow's pioneering blog may have convinced many theorists, by setting an example, to shed the belief that blogging is a frivolous, vulgar activity indulged in solely by the unwashed masses, far too undignified for solemn researchers such as themselves :-)

Rant for the day

2009-05-26T22:51:56Z

Today has been all about people wasting my time.

First it was something to do with my damn car, but I'm not going to talk about that.

In the afternoon someone came over to look at my bike (I'm selling all my stuff since I'm moving to California). The price in the ad was $100. The buyer sounded like a really nice person. After taking my bike out for a spin, she brought it back, said she liked it, and took out the cash.

Specifically, $80.

She claimed she only brought along $80 because that's what her budget was, and wanted to see if I'd take that. THE AD SAID $100, BITCH! The whole time she knew perfectly well she was just wasting my time. And hers. On the off chance that I'd turn out to be a sucker. Fuckity fuck. Of course I didn't take it. I'd rather abandon the bike than sell to a shamelessly dishonest person.

This is yet another way in which Craigslist sucks big time. Honesty is largely a product of the circumstances; it is inherent in a person only to a small degree. In the absence of a reputation system, Craigslist buyers/sellers have no problem behaving in ways they wouldn't in any other context.

Next, I had to sell my all my books. I filled up suitcase with about 30 of them, and some DVDs, and took it to Half Price Books. I could have sold them on eBay, but I knew I'd go crazy listing and shipping them individually. Half Price Books had me wait for about 45 minutes. The dude evaluated each book minutely. On some of them he even brought in a colleague for consultation. They'd talk in hushed tones, then he'd write something down, and move on to the next one. It was a bizarre process. At the end of it, he had a number ready for me.

Specifically, $25.50.

If you know me, you know I don't show surprise easily. But when I heard that number, I literally stared at him with my mouth agape. And then I had him repeat it several times. The guy tried to remind me that they sell it for only 50% of the retail value. So I asked him why he was giving me 7%. He had no answer but a shrug.

It gets worse — they'd take either the whole lot or none of it. I took the offer, of course — the time I'd already wasted was a sunk cost, and they weren't being dishonest, just thoroughly inefficient.

My time was worth more than the money I made, especially if you include the time I spent driving to the store and back. Essentially, I was spending money to get rid of my books. Next time I'll just toss them in the dumpster.

Oh wait, there isn't going to be a next time. It's been a long time since I bought a dead-tree book. With the Kindle, not only are books cheaper, you don't have to delude yourself into thinking there's a meaningful resale value for them. You're buying 1's and 0's and you're paying for the convenience.

Use a foil, for crying out loud!

2009-05-17T17:17:03Z

Whenever a story has a significant scientific component — whether in a novel or on television — there's a tricky problem: how do you communicate the science to the viewer? The characters are already up to speed, so you can't have them just explain it to one another.

Obviously, science fiction is the genre that suffers most acutely. The writers of early sci-fi novels, in the 40s and 50s, for the most part just weren't very good, so they often didn't even make an attempt to solve the problem. For a brilliant illustration of how jarring it is to the reader when characters discuss with each other what must be obvious to them, see If all stories were written like science fiction stories.

But in the written medium, there is at least the crutch of the narrator. While long passages with no dialog are boring, they are not horribly unrealistic. Television doesn't have this escape route.

Take House, which is probably my favorite currently running show. When House and his team do their thinking in front of the whiteboard, they often converse in medical jargon, and then immediately translate it into English. While this makes the show intelligible to the lay person, it must make it unwatchable for real doctors. That's certainly the effect that shows or movies have on me when characters explain computer jargon in situations where they shouldn't.

The funny part is that the House writers seem to be aware of and rather embarrassed about this shortcoming. I say this because the characters sometimes make a remark about the fact that it is weird for them not to be using jargon :-)

The good news is that audiences seem to be tolerating unrealistic dialog less and less. The other three shows that I watch — all newer than House — have built-in premises to solve the problem. In Bones, all the principals have different specialities, so it is natural that they would explain their technology and reasoning to each other all the time. Burn Notice pretty much says "fuck it," and has the protagonist do voice-overs to explain what's going on.

Lie to Me has a foil in the form of a newly recruited crime-solver who is naturally brilliant at lie-detection but is unaware of the scientific principles behind it. Naturally, the other lie-detectors explain things to her a lot. Eventually this character will no longer be able to act as a foil, because she will have learnt the ropes, but I believe the show creators expect that by then, the audience will also have been educated to a large degree. An educated audience leaves them with a different problem — building suspense. So far I've been able to guess the conclusion in most of the episodes (it is yet Season 1). It remains to be seen if they can pull it off.

I think one of the best solutions is to have a true foil, a character who is completely uninitiated. The textbook example, and one that will perhaps never be surpassed for the completeness of the resulting dramatic effect, is Conan Doyle's Dr. Watson. The brilliance of Doyle's creation is that the relationship between the characters makes it possible for Holmes to explain what he wants, when he wants, and no more.

Of course, foils have been used effectively in modern television, such as in the show Psych which I've occasionally watched (although far from my favorite). At any rate, I think it is wonderful that audiences are rejecting unrealistic dialog, and shows are experimenting with different solutions.

All done

2009-05-08T07:04:19Z

So I'm officially done at UT. Now the only thing I need to do before packing up and flying to California is to sell my car :-)

The paperwork and general procedural nonsense they make you do to get a Ph.D is as arcane and retarded as any Hindu ritual I had to sit through as a teenager. This is in spite of the fact that the CS department do their best to ease the pain. People are generally trying to do the right thing, but the system is badly broken and change-resistant.

Smartphone versus Kindle

2009-04-23T05:49:17Z

On my last trip I packed my kindle in my checked luggage instead of my backpack. When I took it out, it looked like this:

It wasn't physically cracked, but half the display was gone, and it won't turn on any more.

I'm going to call and check if my warranty covers it, but if it doesn't, I don't even really care. In the last couple of months, 80% of my book reading has been on my iPod Touch. As it turns out, Amazon has a free and very slick Kindle for iPhone app that lets you read books you've purchased for the Kindle on your iPhone (or iPod Touch). The devices even automatically sync with each other, so that you always pick up where you left off. Of course, the letters aren't as crisp on the Touch as they are on the Kindle (the latter is almost equivalent to paper), but to me this disadvantage is monumentally outweighed by the fact that the Touch fits in my pocket.

There are still many people (albeit a dwindling number) who won't read books on anything but dead trees, but the average person just doesn't care. Consequently, the market for e-books on the iPhone is huge. On the other hand, Amazon's margins on the Kindle device are slim, probably even negative. This is because it uses Sprint's EVDO network, but Amazon eats the cost and doesn't pass it on to the user. Further, there are 10-15 times more iPhones than Kindles in circulation. Amazon has therefore done the obvious thing by allowing even non-Kindle owners to buy books for the iPhone using the app.

I have a hunch that smartphones are going to swallow the book reader market, just as they did PDAs and Internet Tablets.

Edit. Amazon is replacing my Kindle. 1-day shipping on the replacement! Woohoo!

Thoughts on the Y Combinator interview process

2009-04-19T18:11:09Z

We (watchuwant.tv) interviewed yesterday. We didn't make it. Here are some thoughts.

First of all, a lot of people have talked about an adversarial interviewing style. Our experience was the opposite. Everyone was super-friendly the whole time. David's theory is that if you only have an idea or a demo, they push you during the interview to see how you can perform under pressure. On the other hand, we had a working website that had been out there for a month, so we'd already proven we could build something. I find it plausible; I think in addition, maybe the more they like you, the more they push you. It would be good to know what affects the interview style, so that you can know ahead of time what you can expect.

In any case, I just don't get why a lot of interviewees were nervous. The waiting room makes you feel very comfortable, and whatever little nervousness you have should evaporate once Jessica makes her appearance and greets you with her radiant smile. I guess for me there were a couple of more reasons why I wasn't nervous: we weren't dependent on YC; we're pushing ahead with the site regardless, and also, my day job involves often speaking in front of 300 people. Sitting in front of 4 people is nothing by comparison :-)

I flew in from Austin. Near the end of the interview, Jessica said, "so do I make the check out to you?" I said yes. "Did you actually spend 700?" I said I'd spent most of it. I actually spent more than 700, but I only had receipts for slightly less. So she wrote a check for 700 right there on the spot. I was kinda speechless.. I was expecting a bunch of paperwork. We kept laughing about it later. That little incident tells you a lot about why YC is awesome. No bullshit.

We thought the interview went really well. But looking at our notes later, we realized we might not have conveyed fundamental things about the site. For instance, they kept asking us if our featured channels were manually curated. We thought we'd said pretty prominently in our application that we do automated, instantaneous channel creation. So guys, remember that they interview a bazillion companies all at once, so don't assume they remember what you said in your application.

Do mock interviews with as many people as possible. We did a bunch, but only one was a YC alumnus. That interview predicted their reactions far better than the others. Remember this. Each person/group has their own world-view.

One last thing: people who got in got the call as early as 6pm or as late as 11:30 pm. We got emailed around 9pm. So stay by that phone.

So that's pretty much it.. our experience was great, even though we didn't get it, and the preparation for the interview taught us a lot, and spawned a bunch of new directions and monetization ideas. Good luck to today's and future interviewees. You'll hear more about WatchUWant.tv soon :-)

The future of jobs

2009-04-16T23:52:25Z

I think it is fair to say, to a first approximation, that rewarding professions are those that involve some amount of creativity. Of course, you could be unhappy in any job, say because your boss is a jerk, but under the right circumstances you should be able to find job satisfaction in a creative field. On the other hand, it is hard to imagine this being the case with thoroughly repetitive, monotonous tasks.

Mike Rowe would appear to take a contrarian stance in his TED talk celebrating dirty jobs, but I think a distinction must be made between taking up a job in order to pay the bills (and still finding a way to be happy with it), versus actually choosing something as a rewarding career. There is empirical evidence for the legitimacy of this distinction: there are people who assemble computers, cars or motorcycles as a hobby, even though these tasks have been completely automated, whereas you'd be hard pressed to find someone who spends time separating toxic sludge into its constituent components just for fun. I think Paul Graham makes the same point when he makes fun of a job posting calling for applicants with a "passion for service."

Given this (admittedly crude) dichotomy, it becomes important to ask if there will be a day when everyone has a rewarding job. I think this would be a great thing to look forward to, and entirely within the realm of possibility, since technology has been making relentless progress in automating anything that is repetitive. In the rest of this essay I will analyze the unrewarding jobs that have not yet been automated, and discuss what is holding back automation and what it would take to get there.

A couple of quick points of clarification: first, I'm looking a few decades into the future; the current recession has no bearing on the analysis here. Second, even though the average person has an emotional (and adverse) reaction to the loss of jobs to automatons, I will assume I am writing to a more mature audience, and won't waste time arguing that automation is almost always a net benefit to humanity.

The first category consists of tasks that we can already automate, but for some reason continue to be performed by humans, like taking orders at a restaurant. The main reason that these jobs still exist, in my opinion, is inertia: people react with surprise or revulsion at seeing a machine in a role formerly filled by humans, and are confused by the user interface. But they will eventually get used to it, just as they have with ATMs and voice menus. Technologies such as Microsoft Surface, which make it possible to market automated restaurant service as a positive experience, will speed the process along. It is likely that a small percentage of people will still pay for the privelege of human interaction, but that doesn't affect my overall argument.

Incidentally, businesses that involve fully or partially automating tasks previously performed by humans have always been excellent avenues for entrepreneurship. The early history of fast food is a great example.

The second category consists of jobs that we can't yet automate, but are easily with reach, such as fast-food preparation. Human labor is still available too cheaply, preventing the investment needed to build, perfect and deploy the technology to fully automate these jobs. Increasing automation and increasing labor costs form a virtuous cycle, so in a sense it is just a matter of time. But can the cycle be sped up? I can think of three factors that might help:

A slightly over-educated workforce. This puts pressure on workers to try to migrate up the chain toward more creative jobs.^[1]
Less government interference in business. I think this is somewhat nuanced: labor unions have a negative effect on automation, whereas a high minimum wage probably encourages it.
Stimulating developing economies. This will result in their labor prices rising, thus diminishing cheap pools of immigrant laborers and cheap offshore outsourcing.

Finally, it is important to note that there will be "collateral damage," if you will, in terms of jobs or skills that are rewarding to various degress but that we will nevertheless learn to automate. This is not a huge issue today, but this happy state won't last long. Already it has raised some very interesting questions: do singers need to work on their pitch at all, given that software can correct for human imperfection? Technology that falls into this category will be met with much resistance. Some will embrace it, but others will decry the downfall of civilization and probably try to get it banned. Eventually, however, I think we will have to learn to treat jobs that have been automated as hobbies rather than professions, and continue to strive to find even more challenging avenues for human creativity.

^[1] Although it is hard to envision and the process is slow, society can indefinitenly accomodate migration toward jobs at the creative end of the spectrum. Jobs such as entrepreneurship or documentary film-making have elastic demand, unlike say sanitary work. If entrepreneurship becomes highly feasible and attractive for whatever reason, the economy will gradually adjust so that there are more, smaller companies.

Tragedy of the commons: WiFi version

2009-04-08T07:12:19Z

For years now I've run an open wireless network at home. I'm in agreement with Schneier here. Sadly, my network is no longer open.

It always annoyed me that everyone else's network was password protected. On the rare occasion when you need it, such as when you're traveling and the person you're staying with doesn't have a wireless router, a neighbor's open wireless can be a life-saver. Since we don't have a culture of open networks, I'm forced to pack my wireless router when I travel.

Unfortunately, some of my neighbors seem to have decided to steal my wireless permanently. They must be running bit-torrent or something, because my connection quality is seriously degraded. In classic tragedy-of-the-commons fashion, they've forced me to protect my network.

Actually, the admin interface to my router doesn't work any more, so I can't even do that. I'm just plugging my laptop directly into the modem for now; I'm going to have to get a new router. I've had seriously bad luck with wireless routers: this is the third one to die on me in as many years. The only category of electronics I've had worse luck with are hard drives — six failures in two years. Fortunately that will never happen again, because I've switched to SSDs. Seriously, those of you still on hard drives have no idea what you're missing.

Quick note on the social networks paper

2009-03-27T00:09:25Z

Our social networks paper has been getting some press, including on slashdot and BBC news. The de-anonymization attack in our paper is very broad in scope, and unsurprisingly, it has been whittled down into phrases like "we can find you on twitter."

Please read our FAQ. Our attack applies to a variety of scenarios, some involving very sensitive data, including phone call networks which are often shared in anonymous form. The paper is online (html) and most of it is readable with a minimal technical background. I hope our "State of the Union" section will convince you that social network anonymity is a rather important issue.

On a personal level, working on this paper was very gratifying for me--the de-anonymization algorithm is far deeper than the one in the Netflix paper, and took far longer to develop.

Finally, a plug for 33bits.org, my blog about the de-anonymization work I've been part of.

Edit. And I also have a shiny new web page! After testing about 15 different widgets for embedding an RSS feed into a page, and finding none of them satisfactory, I gave up and wrote the damn thing myself :-)