Log in

No account? Create an account
How did academics get left behind? - Arvind Narayanan's journal [entries|archive|friends|userinfo]

How did academics get left behind? [Apr. 18th, 2008|05:36 pm]
Arvind Narayanan
[Tags|, , ]

Correct me if I'm wrong, but as I understand it, we academics were at the forefront of information technology up until the early 90s. The Internet, and later the Web were both invented by and for academics. But that changed abruptly with the commercialization of the Web: we mostly ignored the new developments; whether it was the hatred of anything commercial or the fact that the new web was too mainstream I don't know. By the time the dot com boom came around, we were definitely laggards.

Today the situation is much worse: the majority of my colleagues say they don't get facebook, and some even claim that all of online social networking is a passing fad. Forget facebook, academics by and large haven't even adopted IM, which has only been ubiquitous for what, 15 years?

Perhaps the most egregious example is LaTeX. While it was a huge deal when it was first developed, it hasn't moved forward in more than two decades and is incredibly clumsy and limited by today's standards. The worst part is that LaTeX users will ask you what's wrong with it—with a straight face.

While unfortunate, this is an excellent opportunity if you're an entrepreneur: the academic community is a niche market that has been mostly ignored. The revenue model is also pretty clear-cut: once your product/service is sufficiently popular, you can make licensing deals with universities.

Edit: Since there is much confusion in the comments, I'm talking more about the failure to adopt technology than the failure to invent it.

[User Picture]From: theswede
2008-04-18 11:14 pm (UTC)
LaTeX is incredible. I have absolutely no idea why people complain about it. Sure, some of the libraries are a bit out of date, but generally it's awesome. As long as you use it for what it's intended for, which is *not* layout.
(Reply) (Thread)
[User Picture]From: mmk
2008-04-19 04:06 am (UTC)



Enough said. Makes me look like a genius with no gritty work involved. Atleast twice, folks looked at stuff I printed out and said "OMG, that's LaTeX".

Needless to say, I didn't feel any need to explain how.
(Reply) (Thread)
[User Picture]From: arvindn
2008-04-19 04:32 am (UTC)

Re: LyX

yeah i've used lyx. i agree it's marginally better than plain latex. like being just crippled vs. blind and crippled. but everyone is missing the point. you can produce amazingly beautiful typesetting that makes people swoon as soon as they see it and it's still pointless. think of all the things we can do with technology if you're only willing to escape the confines of the static medium: say if you could highlight (or just mouse over) any mathematical expression and have a graph of that function pop up immediately. or if you could see at a glance all the lemmas used in a particular theorem. i can name ten things like that just off the top of my head. user controlled stylesheets. so that i can turn off heavy images if i want and not jam the bloody printer every time. it's a huge, huge mess.
(Reply) (Parent) (Thread)
[User Picture]From: theswede
2008-04-19 09:02 am (UTC)

Re: LyX

You can do that in PDF from LaTeX. The only reason it isn't in LaTeX is because noone has wanted it enough to put it in there. We generate industrial system documentation in LaTeX, and it beats everything else we've tried for power and lets us do pretty much everything we want to do and then some. Not sure what your problem with it actually is, and yes, I say that with a straight face - have you tried the few competing typesetting solutions?
(Reply) (Parent) (Thread)
[User Picture]From: arvindn
2008-04-19 05:22 pm (UTC)

Re: LyX

the problem i'm talking about is more cultural than technological. if you can do all this now *then why aren't people doing it*? you're not an academic; maybe latex does do everything you might want from a documentation system. i don't know.

as for me, my paper reading/writing processes are severely impeded because the medium is static. and there's no reason for it to be static. that's all.
(Reply) (Parent) (Thread)
[User Picture]From: theswede
2008-04-19 05:26 pm (UTC)

Re: LyX

So why do you phrase it as a technological problem? There's absolutely nothing wrong with LaTeX, and I honestly don't see why you're targetting it instead of targetting the lack of desire to actually use its potential.
(Reply) (Parent) (Thread)
[User Picture]From: arvindn
2008-04-19 05:44 pm (UTC)

Re: LyX

because it is a technological problem. we don't have the tools to run on top of latex and do all the things that need to be done. if we do, no one has heard of them, and that's the same thing. maybe that's not a "problem with latex", in a limited sense. it's like those old linux-is-unusable no-it-isn't-drivers-are-proprietary arguments. does it matter? the bottom line is that someone needs to do something about it.
(Reply) (Parent) (Thread)
[User Picture]From: theswede
2008-04-19 05:48 pm (UTC)

Re: LyX

I don't see that bottom line. What, exactly, is it these academics need to do which they cannot? What is it LaTeX can't do that must be done? I use it to create manuals and documentation which knocks the socks off of the competition, both in terms of speed of creation and quality, and the few things which were missing to do this were easy to code in TeX. What, more precisely, is it that someone needs to do something about?
(Reply) (Parent) (Thread)
[User Picture]From: theswede
2008-04-20 10:49 am (UTC)

Re: LyX

To address your two examples: you can link to graphs or lemmas if you like, and if you wish, you can then use a PDF reader which can pop up previews just like my Firefox does on mouseover of a link. I don't know if such a reader exists, but if it does, LaTeX will *as it is today* automatically make use of this functionality.

If you simply wish to link to the lemmas used, that's a matter of you putting in the links. That's not a problem of technology - raw LaTeX lets you do that, and LyX will allow you to use your biography to make the process immensely painless. But it has to be done by *you*, unless you build a tool which can on its own derive this data - which then can pop out the LaTeX code in your document.

This is how I use LaTeX. Automatic generation of internal and external crosslinks. Automatic duplication of texts in the document so that I don't have to write things more than once, and yet they appear wherever needed. And I end up with a document which is far from static, yet will print out to something useful. This is what LaTeX was intended to do, and does very well. I honestly don't see your problem with it. Your problem appears to be with people not using the available power in ways you would like to see it used. So lead the way! ;)
(Reply) (Parent) (Thread)
[User Picture]From: skthewimp
2008-04-19 04:15 am (UTC)
regardless of how popular the idea becomes among academics, there is a good chance that many will refuse to use them as soon as they figure out that you are trying to sell it to the univs, and that it's a "dirty commercialized thing".

and i'm surprised that most academics don't use IM!
(Reply) (Thread)
[User Picture]From: arvindn
2008-04-19 04:36 am (UTC)
that's a very valid objection. i've considered it, and i think it's the major hurdle to overcome. but i think it can be done. first, anti-commercialization is only part of it. i think the more important reason is elitism. so as long as it's not something that's used by the general population, i expect there'd be less resistance. second, i believe it's possible to change the process so dramatically that someone who doesn't use new technologies simply can't keep up in productivity with someone who does, no matter how smart they are.
(Reply) (Parent) (Thread)
[User Picture]From: virgi
2008-04-19 05:46 am (UTC)
actually, a surprising number of academics use gtalk -- it could be because it's embedded in gmail (unless you turn it off).
(Reply) (Thread)
From: (Anonymous)
2008-04-19 01:47 pm (UTC)
In my experience, IM is not very helpful when doing research. I use it only when discussing the writing of the paper (especially when it is done on the last few days before the deadline). Email is much better for discussing research since it makes you think over your own ideas before writing any crap. Also, I like to talk on the phone rather than on IM.

As for latex, I prefer it over all the other typesetting softwares. Give me something better and I'll take it. It might not be as good as something you have in mind but is certainly the best out there.


(Reply) (Thread)
[User Picture]From: arvindn
2008-04-19 05:25 pm (UTC)
yeah, i agree that using ms word or whatever is even worse than latex. that's why i said it's an opportunity for an entrepreneur.

i disagree re. IM. for me it's very very productive. the bias i'm pointing out is more than that. it's not just that they don't use it *for* research, people don't even leave their IM open (if they use it at all) while doing working. (and since they're working most of the time when they're at the computer they basically end up not using IM.) the excuse is that it interrupts your workflow or whatever. well guess what, you're not required to look at or respond to im's immediately. yes, it takes a while to get used to that. that's what one means by adapting to technology.

"it makes you think over your own ideas before writing any crap"

ah. that's something i have opinions about as well :) i feel academics would actually a lot more productive if we communicated our thoughts without worrying so much if they are correct.
(Reply) (Parent) (Thread)
[User Picture]From: rfc9000
2008-04-20 12:44 am (UTC)
"well guess what, you're not required to look at or respond to im's immediately."
Thats what emails are for. A flashing IM window is hard to be oblivious to and continue working. And if you disable flashing windows, then it becomes functionally equivalent to emails.

The problem with IMs is different people's usage models are different. Some people leave it on all the time (even when they're sleeping), and you might as well send them an email. Some people prefer to stay invisible, and without presence notification, IM becomes a lot similar to emails.

Emails (as against IM/phone) are nice because the mode of communication is asynchronous. IMs can never replace emails IMO.

As to the latex rant, I have the same qn as many other commentors before me.. what exactly is your complaint about it?

One complaint I have is when making presentations using beamer is its very annoying to not be able to dragdrop/copypaste images from anywhere but to copy them to the directory and generate appropriate tags. But then thats a problem with the editor and not latex, and lyx for eg. solves it. So in effect, yes, *straight face* I have no complaints as such with latex.
(Reply) (Parent) (Thread)
[User Picture]From: arvindn
2008-04-20 02:48 am (UTC)
ah, beamer user. let me assure you, there is no way we can communicate meaningfully on this subject :)

as far as i'm concerned i've already explained in the comments what my problem with latex is. i guess instead of harping on the point i'm going to put it on my to-do-someday list to actually do something about it.

and again, re. IM's, i disagree. but i'm not going to argue.
(Reply) (Parent) (Thread)
From: (Anonymous)
2008-04-19 02:01 pm (UTC)
Isnt it more an issue with the output format? I do not think pdf viewers support any fancy features, and hence I dont see how latex should support it.

Apriori, latex's main output format is the ps format, for printing purpose. It serves this purpose exceptionally well.

The only other reasonable formats are html, and pdf. Pdf does not support anything too fancy, except hyperlinks which latex can produce. Ofcourse, latex could do better on the html output, but anyway why would one want to write html using latex.

(Reply) (Thread)
[User Picture]From: arvindn
2008-04-19 05:12 pm (UTC)
you're playing the name game. it doesn't matter if it's the fault of latex or something else. i'm talking about the whole academic typesetting/publishing/distrubition process. it's broken and hugely inefficient and needs to change. and if there isn't a suitable output format, invent one.
(Reply) (Parent) (Thread)
[User Picture]From: ephermata
2008-04-20 08:54 pm (UTC)
It's funny to read this post, since Butler Lampson points out the *failure* to invent the Web as a shortcoming of academic computer science research:

Yes, if you look at "academics" more broadly, then of course Tim Berners-Lee qualifies and we get credit. Still, even then you could argue that the place for coming up with big new ideas had started to broaden or shift away from the traditional sources.

From what I can see, the phenomenon comes down to broader access to information technology and different incentives for academics vs. others. I tend to think less of it is due to outright predjudice against commercial applications of these technologies...but maybe that's because I've seen too many professors and friends go off to start-ups.

First, access. Back before the 90s, you needed to be at a university or major corporation to have an Internet connection. Starting around 1993 or 1994, suddenly many more people could play. Even if the probability of any individual in the new wave of Internet users having a good idea and then putting it into practie is small, the expectation is a lot larger than that taken over a relatively small number of academics.

Second, academic incentives focus on publishing papers and impressing your peers. This is excellent for maintaining high standards of (some kinds of) evaluation and rigor. It's bad for trying out lots of ideas to see which one sticks. It's especially bad when each idea requires nontrivial investment and may not pay off. Furthermore, impressing your peers tends to require generating an "a-ha" moment or showing that your work is deep. What's deep about ICQ or AIM? What's deep about yet another Friendster clone? This isn't even getting into the question of ideas that are so different as to not be comprehensible to the target audience; Scott Aaronson's STOC '36 rejection is a parody of that response.

So it's not surprising that academia would fail to produce IM or facebook -- and other people filled in the gap. Why is that bad? It seems more interesting to ask if there are big ideas that only academia can pursue which haven't been being pursued.

If you've read Asimov's Robot novels, you'll remember that human colonization of space had two major waves : first, the Spacers, who were few in number and emphasized high ability of individuals (along with robot companions), controlled growth, and propagating a fairly static civilization. The Settlers came later and focused on large numbers, widespread propagation, and being just good enough to get to where they were going. Eventually the Settlers "win" - the Spacers die out, their cultures stagnate, their worlds are forgotten...and the knowledge of how to make robots is lost. That's one melodramatic way to characterize things. The analogy is useful for other things too, like Microsoft vs. open source.

For facebook and IM, part of that may be generational gap in working styles between advisors and students. Part of it may also be that we tend to be in proximity with the people with whom we work, despite the rise of e-mail collaborations. Academic predjudice against mainstream stuff seems like only a part of the issue. That being said, I have corresponded with co-authors on AIM...it was helpful but not a giant leap. I haven't tried it so much for the hashing out ideas stage rather than the writing up stage.

About licensing deals with universities: what products are there now which have such licensing deals? The only one that comes to mind is the Blackboard style of class web site software. When I think of research tools that I or my friends use, they're all free software. Probably I'm just not thinking hard enough.

One of the issues that comes to mind is that for research tools, like a super-typesetter, you'd have to sell to each professor or group individually, since that's the only way you'd get adoption. Just having some central IT department in the university make it available probably won't cut it.
(Reply) (Thread)
[User Picture]From: arvindn
2008-04-20 09:50 pm (UTC)
thanks for the detailed comments, david.

i was indeed thinking of tim berners-lee as an academic, but more importantly, i thought the web was used mainly by academics in the early days, for sharing research papers and stuff. i could be wrong.

and i found lampson's slides amusing myself. such as his 'what didn't work' column and the fact that he puts simulation before communcation on his what-are-computers-for slide.

i'm going to disagree with your assessment that there's nothing innovative about aim and facebook. what's innovative about IM is jabber. simple question: why was jabber not built by academics?

and have you noticed the explosion of theoretical and semi-theoretical papers on social networks in the last 2-3 years? why are we trailing and not anticipating commercial developments?

it's not even the failure to produce IM or facebook that i'm complaining about, it's the failure to adopt them, thus moving the culture in academia farther and farther away from what's innovative in the consumer space, ensuring that future innovations in that space won't happen in academia either.

openid is the perfect example. it should have been invented by academics 10 years ago. have you looked at all the useless papers that have been written on the topic of identity? and yet we didn't come up with the one simple thing that actually works, whereas bradfitz invented it all by himself.

re. licensing, there are quite a few products that sell to universities. many of my friends use scientific computation, statistical and parallel computing tools under such licenses. there's also microsoft, adobe etc. who license more consumer oriented products. heck, sun even managed to convince universities to waste huge amounts of money on those chimerical sunray machines and the software that ran on it.

there aren't many web services on that list now, but of course that's going to change as the landscape of computing shifts. i'm not too worried on that account. and as for selling to individual groups, my answer is the same as before: if you make a truly valuable product, people will either have to get with the program or get left behind. it worked for powerpoint.

i do agree that academics are still building truly innovative, hard stuff that only academics can build, and always will. what i'm complaining about is the rejection of anything that's useful but not very hard. when luis von ahn was interviewing, there was a lot of contempt from a good fraction of academics that i think is ill deserved.
(Reply) (Parent) (Thread)
[User Picture]From: ephermata
2008-04-26 02:37 am (UTC)
Sorry, I should have been clearer: when I asked what is deep about ICQ, that was tongue-in-cheek. To get in to a conference, the focus is on technical novelty and "depth," not at all on number of users. Even if you did write a paper on, say, an IM system, there isn't much value to you in continuing to work on it unless you see a new paper coming out at the end. That's what I was trying to get at by saying academics have incentives different from those of people who produce widely adopted technology.

That being said, with respect to adoption of instant messaging, academics created zepyhr in 1988 at MIT as part of Project Athena:
(and yes, OK, there is actually a Usenix Technical paper on Zephyr)

So in that case, to answer "why was Jabber not built by academics," there were in fact academic projects which pre-dated commercial IM by about a decade. As to why Zephyr didn't take over the world, my guess is that the people who developed them hit a local maximum: all the people they knew or cared about were using the tools, and they didn't see the need to make user-friendly versions or open the system to the general public.

Later, the people who built ICQ, AIM, and Jabber realized that they could make a mass market IM, they saw a clear path to reward for doing so, and they put in the necessary work. The kind of work needed to get wide adoption is hard to justify in an academic setting. This is why you see professors found startups with some of their star grad students to commercialize projects (e.g. Monica Lam/Moka5, Dawson Engler/Coverity, Mendel Rosenblum/VMware out of Stanford, George Varghese/NetSift out of UCSD, Robert T. Morris/Meraki out of MIT).

With respect to identity, it's an overstatement to say we didn't come up with anything that works. Some people do manage to run Kerberos, with a lot of work; that was another MIT project. There's also Shibboleth, which some Internet2 universities use:

Neither of these are as simple as OpenID, of course.

As to the explosion of social network papers, that's what is hot right now. Academia has herd behavior, since there's pressure to publish and working on a "hot" topic is perceived to help your chances of getting in. So most of the work will tend to be following a trend instead of setting it - the remaining non-following stuff is harder to find and will tend to be in the minority. Another issue is that collecting the kinds of data sets which Facebook or Friendster have under the constraints of an academic Human Subjects committee would be "interesting." (danah boyd sometimes touches on this in her blog, since she does a lot of work trying to understand social structures in friendster and other social networking services.)

Again, that being said, there were in fact people in academia looking at online communities long before Facebook existed. From a qualitative perspective, there was a small boom in MUD and MOO studies in the 1990s, where social structures came up for discussion but people didn't go off just yet and compute social network graphs. Sherry Turkle's _Life on the Screen_ is probably one of the better known books in this vein.

A friend of mine also did a paper in 2004 on structure in Usenet discussions, which has a few quantitative pieces. the related work section also points to some previous work by academics about online social structures:

From a quantitative perspective, I first heard about the Milgram six-degrees-of-separation experiment sometime around 2000-2001 in the context of Watts and Strogatz' book on small world models. That followed their 1998 Nature paper on "Collective dynamics of 'small-world' networks." So this paper, at least, seems to anticipate the social networking craze.

I'm sorry to hear that Luis von Ahn had a poor reception when he was interviewing. That is indeed shortsighted and ill-deserved. Thankfully, Luis is changing the world regardless of what they think. The key issue is whether you and I can do the same.
(Reply) (Parent) (Thread)
From: (Anonymous)
2008-04-20 09:54 pm (UTC)

We are at the forefront.

Academics *are* and *have been* at the forefront of information technology. Many famous results have come about in the past few years, not the least of which is the "PRIMES is in P" paper or the Gupta-Kumar "Wireless networks" paper.

Perhaps we very strongly disagree on what "forefront" of information technology really is.

(Reply) (Thread)
[User Picture]From: arvindn
2008-04-20 10:00 pm (UTC)

Re: We are at the forefront.

dude, primes is in P is mathematics, it has nothing whatsoever to do with information technology. we don't disagree about the meaning of "forefront", we disagree about the meaning of "information technology" :)
(Reply) (Parent) (Thread)
[User Picture]From: arvindn
2008-04-20 10:01 pm (UTC)

Re: We are at the forefront.

and once again, i'm talking about adopting technology, not inventing it.
(Reply) (Parent) (Thread)