Every weekday, a truck pulls up to the Cecil H. Green Library, on the campus of Stanford University, and collects at least a thousand books, which are taken to an undisclosed location and scanned, page by page, into an enormous database being created by Google. The company is also retrieving books from libraries at several other leading universities, including Harvard and Oxford, as well as the New York Public Library. At the University of Michigan, Google’s original partner in Google Book Search, tens of thousands of books are processed each week on the company’s custom-made scanning equipment.
Google intends to scan every book ever published, and to make the full texts searchable, in the same way that Web sites can be searched on the company’s engine at google.com. At the books site, which is up and running in a beta (or testing) version, at books.google.com, you can enter a word or phrase—say, Ahab and whale—and the search returns a list of works in which the terms appear, in this case nearly eight hundred titles, including numerous editions of Herman Melville’s novel. Clicking on “Moby-Dick, or The Whale” calls up Chapter 28, in which Ahab is introduced. You can scroll through the chapter, search for other terms that appear in the book, and compare it with other editions. Google won’t say how many books are in its database, but the site’s value as a research tool is apparent; on it you can find a history of Urdu newspapers, an 1892 edition of Jane Austen’s letters, several guides to writing haiku, and a Harvard alumni directory from 1919.
No one really knows how many books there are. The most volumes listed in any catalogue is thirty-two million, the number in WorldCat, a database of titles from more than twenty-five thousand libraries around the world. Google aims to scan at least that many. “We think that we can do it all inside of ten years,” Marissa Mayer, a vice-president at Google who is in charge of the books project, said recently, at the company’s headquarters, in Mountain View, California. “It’s mind-boggling to me, how close it is. I think of Google Books as our moon shot.”
Google’s is not the only book-scanning venture. Amazon has digitized hundreds of thousands of the books it sells, and allows users to search the texts; Carnegie Mellon is hosting a project called the Universal Library, which so far has scanned nearly a million and a half books; the Open Content Alliance, a consortium that includes Microsoft, Yahoo, and several major libraries, is also scanning thousands of books; and there are many smaller projects in various stages of development. Still, only Google has embarked on a project of a scale commensurate with its corporate philosophy: “to organize the world’s information and make it universally accessible and useful.”
In part because of that ambition, Google’s endeavor is encountering opposition. A federal court in New York is considering two challenges to the project, one brought by several writers and the Authors Guild, the other by a group of publishers, who are also, curiously, partners in Google Book Search. Both sets of plaintiffs claim that the library component of the project violates copyright law. Like most federal lawsuits, these cases appear likely to be settled before they go to trial, and the terms of any such deal will shape the future of digital books. Google, in an effort to put the lawsuits behind it, may agree to pay the plaintiffs more than a court would require; but, by doing so, the company would discourage potential competitors. To put it another way, being taken to court and charged with copyright infringement on a large scale might be the best thing that ever happens to Google’s foray into the printed word.
Though Google has more than ten thousand employees—about fifty new ones are hired each week—and a market capitalization of more than a hundred and fifty billion dollars, the company cultivates the air of a college campus at its headquarters, in Silicon Valley. Now and then, there are self-consciously wacky stunts, like Pajama Day, which happened to take place when I visited. (The event was to be madcap within reason; supervisors were told to convey the message that “pajamas means ‘pajamas,’ not ‘what you sleep in.’ ”) When I met with Sergey Brin, a co-founder of Google, he was wearing bright-blue p.j.s, with the company’s logo stitched on the breast pocket.
The story of how Brin and Google’s other co-founder, Larry Page, met as graduate students in computer science at Stanford in the mid-nineties, and devised a series of elegant software algorithms that allowed Web searchers to find relevant information quickly and efficiently, has become part of Silicon Valley lore. Less well known is that, at the time, Brin and Page were also working on Stanford’s Digital Library Technologies Project, an attempt, funded by the federal government, to organize different kinds of stored information, including books, articles, and journals, in digital form. “There was an attitude in computer science that putting things on dead trees was obsolete and getting it all into a searchable, digital format was a quest that had to be accomplished someday,” Terry Winograd, a Stanford professor who was a mentor to Page and Brin, said.
After founding Google, in 1998, Page and Brin—who are now in their mid-thirties and worth around fourteen billion dollars each—began to talk about how to include books in the company’s database. Page, in particular, embraced the idea of putting books online; at one point, he set up a primitive lab in his office, with a scanner and a page-turning machine. “I think it was motivating to have those kinds of aspirations, but nobody really took it seriously,” Brin told me. The men were less interested in making it easy for people to obtain the full texts of books online than in making accessible the information those books contained. “We really care about the comprehensiveness of a search,” Brin said. “And comprehensiveness isn’t just about, you know, total number of words or bytes, or whatnot. But it’s about having the really high-quality information. You have thousands of years of human knowledge, and probably the highest-quality knowledge is captured in books. So not having that—it’s just too big an omission.” As Marissa Mayer put it, “Google has become known for providing access to all of the world’s knowledge, and if we provide access to books we are going to get much higher-quality and much more reliable information. We are moving up the food chain.”
In 2002, Google quietly made overtures to several libraries at major universities. The company proposed to digitize the entire collection free of charge, and give the library an electronic copy of each of its books. “Larry is an undergrad alum here at Michigan, and he knew we were already interested in digitizing the library as part of our preservation efforts,” John Wilkin, an associate university librarian at Michigan, told me. “There was a lot of back-and-forth between Google and us in the process. We wanted to insure that the materials wouldn’t be damaged and that what came out could be used as a preservation surrogate. They started experimenting with different ways of copying the images, and we started a pilot project in July, 2004. We’ve been getting better, going faster. We’re doubling our output all the time.” The Michigan library holds seven million volumes, and Wilkin believes that Google will have copied the entire collection in about six years.
Last month, at the New York Public Library, Google hosted a conference on the future of the publishing industry. About four hundred people—mainly publishing executives and agents—attended, most of them grimly aware of the simultaneous lethargy and panic that have characterized their industry’s response to the digital age. Nearly all attempts to sell books in an electronic format have been disappointing, and now Google appeared to be encroaching on the publishers’ domain. The implicit message of the conference was summed up by a quotation from Charles Darwin that was projected on a screen: “It is not the strongest of the species that survive, nor the most intelligent, but the ones most responsive to change.” As Laurence Kirschbaum, a longtime publishing executive who recently became a literary agent, told me at the conference, “Google is now the gatekeeper. They are reaching an audience that we as publishers and authors are not reaching. It makes perfect sense to use the specificity of a search engine as a tool for selling books.”
Google thought so, too, and designed the books project accordingly. In addition to forming partnerships with libraries, the company has signed contracts with nearly every major American publisher. When one of these publishers’ books is called up in response to search queries, Google displays a portion of the total work and shows links to the publisher’s Web site and online shops like Amazon, where users can buy the book. “We are helping the publishers reach consumers that otherwise might not have known about their books and helping them market their books by giving limited but relevant previews of the books,” Jim Gerber, Google’s director of content partnerships, told me. “The Internet and search are custom made for marketing books. When there are a hundred and seventy-five thousand new books each year, you can’t market each one of those books in mass market. When someone goes into a search engine to learn more about a topic, that is a perfect time to make them aware that a given book exists. Publishers know that ‘browse leads to buy.’ ” (Google says that it does not take a cut of sales made through its books site.)
Still, on October 19, 2005, several leading publishers, including Simon & Schuster, the Penguin Group, and McGraw Hill—all of which are partners in Google Book Search—filed a lawsuit against the company, seeking to stop the project. The publishers don’t object to Google’s plan for helping them sell new books, but they assert that the library component of the project is illegal. They claim that Google’s “massive, wholesale and systematic copying of entire books still protected by copyright” infringes on the publishers’ rights. They demand that Google stop further copying and “destroy all unauthorized copies made by Google through the Google Library Project of any copyrighted works.” (The Authors Guild filed its lawsuit around the same time.) The publishers, who have the support of the Association of American Publishers, are suffering from a version of the problem that John Kerry had in the last Presidential campaign: they are for Google Book Search at the same time that they are against it.
Copyright law dates to the birth of the Republic. Article I of the Constitution assigns Congress the right to pass laws “securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries.” The first copyright law was passed in 1790, and it has been frequently and confusingly amended over the years, most recently in the Sonny Bono Copyright Term Extension Act of 1998, which extended copyright terms by twenty years. (The law is also known as the Mickey Mouse Protection Act, because the Walt Disney Company, seeking to protect its copyright on early animated classics like “Steamboat Willie,” lobbied heavily for it.) The twisted history of copyright law has insured an awkward passage into the digital age.
The legal assertion at the core of Google’s business plan is its purported right to scan millions of copyrighted books without payment to or permission from the copyright owners. Approximately twenty per cent of all books are in the public domain; these include books that were never copyrighted, like government publications, and works whose copyrights have expired, like “Moby-Dick.” Google has simply copied such books and made them available on the Web. Roughly ten per cent of books are copyrighted and in print—that is, actively being sold by publishers. Many of these books are covered by Google’s arrangement with its publisher partners, which allows the company to scan and display parts of the works.
The vast majority of books belong to a third category: still protected by copyright, or of uncertain status, and out of print. These books are at the center of the conflict between Google and the publishers. Google is scanning these books in full but making only “snippets” (the company’s term) available on the Web. (Google searches turn up only the search term and about twenty words on either side of it.) Copyright law has never forbidden all “copying” of a protected work; scholars and journalists have long been allowed to quote portions of copyrighted material under the doctrine of fair use. Google maintains that the chunks of copyrighted material that it makes available on its books site are legal under fair use. “We really analogized book search to Web search, and we rely on fair use every day on Web search,” David C. Drummond, a senior vice-president at Google who is overseeing the response to the lawsuits, told me. “Web sites that we crawl are copyrighted. People expect their Web sites to be found, and Google searches find them. So, by scanning books, we give books the chance to be found, too.” (Google also has an “opt out” policy, which allows copyright holders to request that specific titles be omitted from the company’s database.)
However, according to the plaintiffs in the cases against Google, the act of copying the complete text amounts to an infringement, even if only portions are made available to users. “What they are doing, of course, is scanning literally millions of copyrighted books without permission,” Paul Aiken, the executive director of the Authors Guild, said. “Google is doing something that is likely to be very profitable for them, and they should pay for it. It’s not enough to say that it will help the sales of some books. If you make a movie of a book, that may spur sales, but that doesn’t mean you don’t license the books. Google should pay. We should be finding ways to increase the value of the stuff on the Internet, but Google is saying the value of the right to put books up there is zero.”
Google asserts that its use of the copyrighted books is “transformative,” that its database turns a book into essentially a new product. “A key part of the line between what’s fair use and what’s not is transformation,” Drummond said. “Yes, we’re making a copy when we digitize. But surely the ability to find something because a term appears in a book is not the same thing as reading the book. That’s why Google Books is a different product from the book itself.” In other words, Google says that being able to search books on its site—which it describes as the equivalent of a giant library card catalogue—is not the same as making the books themselves available. But the publishers cite another factor in fair-use analysis: the amount of the copyrighted work that is used in the creation of the new one. Google is copying entire books, which doesn’t sound “fair” to the plaintiff publishers and authors. “Traditional copyright analysis says that a transformation leads to the creation of a new and independent work, like a parody or a work of criticism,” Jane Ginsburg, a professor at Columbia Law School, said. “Copying the entire work, which is what Google is doing, does not preclude a finding of fair use, but it does fall outside the traditional paradigm.”
Harvard, Stanford, and Oxford have prohibited Google from scanning copyrighted works in their collections, limiting the company to books that are in the public domain. Because of the opacity of copyright law, and the extension of protections mandated by the 1998 act, it’s not always clear which works are still protected. (Copyright status can become murky when authors die or publishing houses go out of business.) Stanford has drawn a line at 1964 and prohibited Google from copying most works published since that date. “When Google got sued, we got nervous,” Michael A. Keller, the university librarian at Stanford, told me. “We’re not a public institution. We don’t have any state immunity from being sued ourselves, so we started sorting out the stuff that we know is public domain.” (Several of the public institutions that are Google’s partners, including the Universities of Michigan, California, Virginia, and Texas at Austin, are allowing the scanning of copyrighted material.)
The chief engineer of Google’s system for scanning books in the library collections is Dan Clancy, who joined the company after eight years at NASA, where he supervised teams of Ph.D.s. working on problems related to artificial intelligence. Google provides its employees with free food twenty-four hours a day, and Clancy, a tall, shambling man with a shock of white-blond hair, conducted most of our conversations with bits of granola bar clinging to his shirt.
“Previously, when people have done scanning, they always were constrained by their budget and their scale,” Clancy told me. “They had to spend all this time figuring out which were the perfect ten thousand books, so they spent as much time in selection as in scanning. All the technology out there developed solutions for what I’ll call low-rate scanning. There was no need for a company to build a machine that could scan thirty million books. Doing this project just using commercial, off-the-shelf technology was not feasible. So we had to build it ourselves.”
Google will not discuss its proprietary scanning technology, but, rather than investing in page-turning equipment, the company employs people to operate the machines, I was told by someone familiar with the process. “Automatic page-turners are optimized for a normal book, but there is no such thing as a normal book,” Clancy said. “There is a great deal of variability over books in a library, in terms of size or dust or brittle pages.” (To needle Google, several blogs have posted images from the books site that include the scanners’ fingers.) Google will not reveal how much it is spending on the books project. In 2005, Microsoft announced that it would spend two and a half million dollars to scan a hundred thousand out-of-copyright books in the collection of the British Library. At this rate, scanning thirty-two million books—the number in WorldCat’s database—would cost Google eight hundred million dollars, a major but hardly extravagant expenditure for a multibillion-dollar corporation.
Copying all those pages presents many difficulties, but writing software to make the books useful to searchers is even harder. “The scanning technology is boring,” Clancy said. “The real challenge is to get somebody something that they are actually interested in, inside a book. Web sites are part of a network, and that’s a significant part of how we rank sites in our search—how much other sites refer to the others.” But, he added, “Books are not part of a network. There is a huge research challenge, to understand the relationship between books.”
Still, the basic search protocols function well. A search for “Heart of Darkness” leads immediately to Joseph Conrad’s novel, which is not as obvious as it sounds, considering how common the words in the title are. As Clancy said, “If you put in ‘Heart of Darkness,’ we have to know that you’re looking for the novel, not a book about lighting conditions in cardiac surgery. So how do we do that? We rank some words more important than others. The title may matter more than the content, so we may weight that more. You could also look at what other people have searched for, so if everyone who searched for ‘Heart of Darkness’ clicked on the novel, we might figure that you probably will, too.”
The most important data for ranking searches, Clancy explained, may come from Web pages that link to books in Google’s database. (For instance, if links on the phrase “Clinton’s autobiography” direct users to a copy of “My Life” on the books site, there is a high probability that people who use the same search terms will also want this result.) “We just started, and we need to make these books networked, and we need people to help us do that,” Clancy said.
Google’s database contains many books in languages other than English, but for now they must be searched in the original tongue. On the company’s Web site, there is already a primitive translation feature, and it may someday be enhanced to allow books to be rendered in another language at the touch of a button. “In terms of democratization, you want to be able to access information,” Clancy told me. In places like the Arab world, where few titles are translated into the local languages each year, he said, access to the world’s books could have a substantial impact. “We are talking about a universal digital library,” Clancy went on. “I hope this world evolves so that there exists a time where somebody sitting at a terminal can access all the world’s information.”
Such messianism cannot obscure the central truth about Google Book Search: it is a business. Google has pledged not to show advertising next to the pages of library books, but the company does sell advertising alongside search results that lead to books obtained from publishers. Google’s prospects for producing revenue from the books project appear rather modest, but the company has often made a profit on ventures that initially seemed unlikely to be lucrative. “We’ve had this fortunate streak that when we’ve done things that have impacted our users and society as a whole—positively, in a significant way—we’ve been rewarded by that downstream in some way, even though we may not have envisioned exactly what it was right offhand,” Sergey Brin told me. “We didn’t have ads when we first put up Web search. It wasn’t clear it was great business when we started search. In fact, the companies that were doing search were moving away from it. But we just thought it was important, and we thought that where there was a will there would be a way. And in fact it turned out to be a great way to make money—doing search with targeted advertising. And I think you’ll find the same sort of thing here.”
The key legal question is whether the courts will allow Google to continue to scan copyrighted material without permission. But the schedule of the lawsuits may turn out to be as significant as the merits of the cases, which are before Judge John E. Sprizzo. In keeping with the stately pace of federal litigation, the depositions of witnesses are to begin sometime this year, and the parties will be allowed to file motions for summary judgment—in Google’s case, to dismiss the suits—in early 2008. Then there could be a trial. If the cases are appealed, they could linger well into the next decade.
However, most people involved in the dispute believe that a settlement is likely. “The suits that have been filed are a business negotiation that happens to be going on in the courts,” Marissa Mayer told me. “We think of it as a business negotiation that has a large legal-system component to it.” According to Pat Schroeder, the former congresswoman, who is the president of the Association of American Publishers, “This is basically a business deal. Let’s find a way to work this out. It can be done. Google can license these rights, go to the rights holder of these books, and make a deal.”
The terms of such a deal aren’t hard to imagine. The Authors Guild is concerned that pirated copies of the books on Google’s site could leak to the public, and so the organization would insist on security measures. (Sadly, for writers and publishers, demand for their products has never been robust enough to generate a major piracy problem.) As for distribution of the proceeds from the site, Google might agree to share revenue with publishers, in the way that radio stations pay for the music they play; publishers could receive a fee based on a statistical analysis of how often their books are viewed. Google could pay in cash or in kind, with advertising.
But a settlement that serves the parties’ interests does not necessarily benefit the public. “It’s clearly in both sides’ interest to settle,” Lawrence Lessig, a professor at Stanford Law School, said. “Businesses in Internet time can’t wait around for years for lawsuits to be resolved. Google wants to be able to get this done, and get permission to resume scanning copyrighted material at all the libraries. For the publishers, if Google gives them anything at all, it creates a practical precedent, if not a legal precedent, that no one has the right to scan this material without their consent. That’s a win for them. The problem is that even though a settlement would be good for Google and good for the publishers, it would be bad for everyone else.”
Libraries have recognized for some time that they must adapt to the digital age, and many have taken steps in that direction. In 1995, Stanford founded the HighWire Press, which now provides electronic access to more than a thousand scholarly journals. A few years later, Stanford digitized most of its card catalogue, and circulation of its books increased by fifty per cent. “Once our students could sit in their dorm rooms and find out what we had in the library, they sought out more books,” Michael Keller, the university librarian, says. Individual libraries sometimes received grants to scan specific collections—in 2001, the New York Public Library used federal money to digitize a substantial portion of the collection at its Schomburg Center for Research in Black Culture—but a comprehensive effort seemed inconceivable. According to Paul LeClerc, who has been the president of the New York Public Library for the past thirteen years, “For the first decade of my tenure, I was always asked, ‘Weren’t libraries going to go online?’ And I’d say of course we want to do it, but it’s not going to happen, because no one is going to give us the money to do it. Nowhere on the horizon was that amount of money predictable or identifiable. Then came Google. This struck us as being the quickest, the fastest, and the most efficient way of getting large-scale additions to our collections online for free use.”
Among Google’s potential competitors in the field of library digitization are members of the Open Content Alliance, which facilitates various scanning projects around the country and overseas. Funded largely by Microsoft and the Alfred P. Sloan Foundation, the O.C.A. has formed alliances with many companies and institutions, including the Boston Public Library, the American Museum of Natural History, and Johns Hopkins University. For the moment, though, the O.C.A.’s members are copying only material in the public domain (and works from copyright owners who have given explicit permission), which limits the scope of the projects substantially.
Google’s advantage may well be cemented if the company settles its lawsuits with the publishers and authors. “If Google says to the publishers, ‘We’ll pay,’ that means that everyone else who wants to get into this business will have to say, ‘We’ll pay,’ ” Lessig said. “The publishers will get more than the law entitles them to, because Google needs to get this case behind it. And the settlement will create a huge barrier for any new entrants in this field.”
In other words, a settlement could insulate Google from competitors, which would be especially troubling, because the company has already proved that when it comes to searches it is not infallible. “Google didn’t get video search right—YouTube did,” Tim Wu, a professor at Columbia Law School, said. (Google solved that problem by buying YouTube last year for $1.6 billion.) “Google didn’t get blog search right—technorati.com did,” Wu went on. “So maybe Google won’t get book search right. But if they settle the case with the publishers and create huge barriers to newcomers in the market there won’t be any competition. That’s the greatest danger here.”
The most striking thing about Pajama Day at Google was how few people participated. Most of the rank and file saw the stunt for the manufactured fun that it was. They came to work in their usual slacker uniforms of jeans and T-shirts—which are, in their way, as conformist as white shirts and ties were at I.B.M. in the nineteen-sixties. Google, as its employees seem to recognize, cannot pretend to be anything other than a large and powerful corporation.
It’s easy to mock Google’s unofficial motto—“Don’t be evil”—but there is nothing evil about Google Book Search. At the same time, there is nothing inherently virtuous about it. Google has succeeded because, on the whole, it has developed excellent products; it’s folly to judge the company’s behavior on moral grounds. Its shareholders certainly don’t.
Nor can publishers and authors, who are struggling for a way to survive in a new age, portray their conflict with the company as one between good and evil. The dual status of several leading publishers as both partner and adversary to Google underscores their desperate need to hedge their bets in a digital world that they have yet to master. The publishers’ complaint against Google states that “the Publishers support making books available in digital form so that those books can be, among other things, researched through electronic means.” That may be true in theory, but trade publishers, in particular, have been slow to embrace new technology, especially for out-of-print books; Google will almost certainly bring more attention to these works than their own publishers have.
The law is supposed to resolve issues like these—between self-interested parties with reasonable claims and legitimate arguments. But the rules of copyright are so ambiguous, and the courts so slow, that the judicial system serves largely to implement the law of the jungle. “There is a real opportunity to move books into the digital arena,” Marissa Mayer told publishers during the conference at the New York Public Library. “And we are going to do it together.”