The
goal of all search engines is to find and organize distributed data
found on the Internet. Before search engines were developed, the
Internet was a collection of File Transfer Protocol (FTP) sites in which
users would navigate to find specific shared files. As the central list
of web servers joining the Internet grew, and the World Wide Web became
the interface of choice for accessing the Internet, the need for
finding and organizing the distributed data files on FTP web servers
grew. Search engines began due to this need to more easily navigate the
web servers and files on the Internet.
The
first search engine was developed as a school project by Alan Emtage, a
student at McGill University in Montreal. Back in 1990, Alan created
Archie, an index (or archives) of computer files stored on anonymous FTP
web sites in a given network of computers (“Archie” rather than
“Archives” fit name length parameters – thus it became the name of the
first search engine). In 1991, Mark McCahill, a student at the
University of Minnesota, effectively used a hypertext paradigm to create
Gopher, which also searched for plain text references in files.
Archie
and Gopher’s searchable database of websites did not have natural
language keyword capabilities used in modern search engines. Rather, in
1993 the graphical Mosaic web browser improved upon Gopher’s primarily
text-based interface. About the same time, Matthew Gray developed
Wandex, the first search engine in the form that we know search engines
today. Wandex’s technology was the first to crawl the web indexing and
searching the catalog of indexed pages on the web. Another significant
development in search engines came in 1994 when WebCrawler’s search
engine began indexing the full text of web sites instead of just web
page titles.
While
both web directories and search engines gained popularity in the
1990s, search engines developed a life of their own becoming the
preferred method of Internet search. For example, the major search
engines found in use today originated in development between 1993 and
1998.
Excite – Born in 1993
Excite
was born in February 1993 as a university project called Architext
involving six undergraduate students at Stanford seeking to use
statistical analysis of word relationships to improve relevancy of
searches on the Internet. This school project eventually led to Excite’s
commercial release as a crawling search engine at the end of 1995. With
solid growth in 1996, Excite purchased WebCrawler and Magellan. Toward
the end of the 1990s, Excite partnered with MSN and Netscape providing
search services. In 1999, Excite was sold to broadband provider
@Home.com (later becoming Excite@Home) as part of a $6.7 billion merger
after its traffic started to decline with the release of Google in 1998.
With significant debts, Excite@Home filed for bankruptcy in October 2001 and sold its high-speed network to AT&T for $307 million. A month later, InfoSpace made a $10 million bid to buy Excite@Home’s assets including domain names and trademarks from bankruptcy court.
Infospace’s
offer was accepted and they subsequently powered the Excite web site
and sold portal components to iWon. InfoSpace’s Dogpile crawler replaced
Excite’s making Dogpile and Excite’s search results the same. Both
Excite and Dogpile are also powered by LookSmart’s directory, except
that Dogpile includes a number of other InfoSpace directories. Also as
part of the deal, InfoSpace acquired rights to WebCrawler. Ask Jeeves
(now Ask.com) purchased the Excite.com portal in 2004. Now, Excite
offers search results through a metasearch tool combining results from
pay-per-click and natural search tools.
Yahoo! - Born in 1994
David Filo and Jerry Yang started Yahoo! in
1994. Originally it was a highly regarded directory of sites that were
cataloged by human editors. This directory provided an extensive listing
of websites supported by a network of regional directories. In 2001,
Yahoo! started charging a fee for inclusion in its directory listing.
Yahoo!’s action helped control the number of sites listed and helped
cover costs with additional revenue.
Initially,
Yahoo! used secondary search engine services to support its directory.
Partnerships have included agreements with Inktomi and Google. In
October 2002, Yahoo shifted to crawler-based listing for its search
results. In 2004, Yahoo! purchased Overture’s pay-per-click service,
which had only months earlier purchased AltaVista and AlltheWeb, and
Inktomi’s search database. With these acquisitions, Yahoo! combined
these tools to create its own search index. Today, Overture is renamed
Yahoo! Search Marketing and provides paid search advertising revenue.
The Yahoo! Directory remains one of the top indexes powering search
listings.
WebCrawler – Born in 1994
WebCrawler
was the first search engine to provide full text search. In 1994, Brian
Pinkerton, a Computer Science and Engineering student at the University
of Washington, used his spare time to create WebCrawler. With
WebCrawler, Brian generated a list of the Top 25 websites on March 15, 1994. Only a month later on April 20, 1994, Brian announced the release of WebCrawler live on the web with a database of 4000 websites. On June 11, 1994, Brian posted to
the Usenet group comp.infosystems.announce that the WebCrawler Index
was available for searching. By November 14, 1994, WebCrawler served its
one millionth query. By the end of the year, WebCrawler signed two
sponsors, DealerNet and
Starwave providing needed capital to finance WebCrawler. A little less
than a year later, WebCrawler was fully operating on advertising
revenue.
A
young America Online, without access to the web, acquired WebCrawler on
June 1, 1995. On September 4, 1995, Spidey was created as WebCrawler’s
mascot. On April 1, 1997 (no fooling), WebCrawler was sold by AOL to
Excite. Initially, WebCrawler was going to run by its own dedicated team
within Excite, but eventually the two were merged together on the same
back end. In 2001, Excite went bankrupt and was purchased by InfoSpace.
As part of the agreement, InfoSpace acquired WebCrawler. Today,
InfoSpace runs WebCrawler as a meta-search tool blending results from
Google, Yahoo!, Live Search (formerly MSN Search), Ask, About.com, MIVA,
LookSmart, and others. As for Spidey, he is now purple.
Lycos – Born in 1994
Lycos
was one of the earliest search engines developed in 1994 at Carnegie
Mellon University by Dr. Michael Mauldin and a team of researchers. The
Lycos name came from the Latin term “lycosidea” referencing wolf spiders
that hunt and actively stalk prey. The company was founded on $2
million in venture capital funding from CMGI. The company was headed by
Bob Davis, who concentrated on building Lycos into an
advertising-supported web portal. The company went public in April of
the next year with little money. With phenomenal growth in its catalog,
Lycos had the largest index at the end of 1996 with 60 million
documents. In 1997, Lycos Pro was launched with a new search algorithm
and continued to grow. By 1999, Lycos would emerge from a crowded pack
to become the most-visited web portal. Over the next few years, Lycos
would become one of the most profitable Internet businesses and acquired
nearly two dozen high profile Internet brands.
For
example, in February 1998, Lycos acquired Tripod Inc., a website where
people built their own web pages. That summer, language search was
introduced and the search results pages were redesigned. At this same
time, WhoWhere Inc.’s directory services and Mail City’s email services
were acquired for $133 million in stock. Toward the end of 1998, Lycos
acquired Wired Digital (owner of HotBot) for $83 million.
By
1999, Lycos was one of the most visited search tools on the web and
would continue to be involved in new projects and acquisitions. In
February that same year, Lycos became USA/Lycos Interactive Networks
when USA Networks bought a 61.5% ownership in the company for $18
billion. As a result, Lycos later announced a project to create a search
tool to query information on USATODAY.com’s news site. In April 1999,
Lycos joined the Open Directory Project run by Netscape. In June 1999,
Lycos joined with Intelliseek to provide a directory of over 7,400
databases previously not on the web. In September 1999, Lycos acquired
Quote.com, an investment information site and launched the Lycos Zone,
an educational website for kids with content filtering. Finally, in
December 1999, Lycos invested in FAST search technology which began
powering Lycos advanced search technology.
In
May 2000, Terra Networks, an Internet arm of the Spanish
telecommunications giant Telef?nica, purchased Lycos for $5.4 billion,
forming a new company Terra Lycos. This takeover marked a 3,000 times
return on the initial venture capital investment and 20 times the
initial public offering value. Lycos remained the name of the US
franchise brand. Overseas, the company was known as Terra Networks, and
founder Bob Davis left the company.
Lycos
suffered from the dotcom crash in 2001. In late 2001, Lycos abandoned
its own crawler and began serving results exclusively from FAST. In
August 2004, Terra sold Lycos to Daum Communications Corporation for
$95.4 million. This low number was less than 2% of Terra’s initial
investment. By October 2004, the deal was finalized and the company name
returned to Lycos.
With
new ownership, Lycos refocused on becoming a community destination for
broadband entertainment rather than a search portal. In July 2006, Wired
News, which had been a part of Lycos since the acquisition of Wired
Digital in 1998, was sold. The Lycos Finance division, known for
Quote.com and Raging Bull.com, and it’s online dating site,
Matchmaker.com, were also sold. Lycos also regained ownership of the
Lycos trademark from Carnegie Mellon University allowing it to become
Lycos, Inc.
Since
2006, Lycos has announced the introduction of Lycos Phone, Lycos Mail,
and Lycos MIX. These services and tools combine IM video chat, mp3
player, and unlimited file size sending and receiving via email, video
watch and chat, and social media applications. Lycos remains a top 25
Internet destination, is the 13th largest online property worldwide and
remains a top 5 Internet portal behind Yahoo!, MSN, AOL, and MySpace.
Infoseek – Born in 1994
Infoseek,
also known as the “big yellow,” was founded by Steve Kirsch in 1994. At
the start in January 1994, InfoSeek was a pay-for-use service. The fee
service was dropped in August 1994 and Infoseek was re-presented as
Infoseek Search in February 1995.
Infoseek’s
position in the search engine world was accelerated in 1995 by a deal
with Netscape in which it became the default Netscape search engine. In
June 1996, Infoseek went public and by September the next year served
7.3 million visitors per month.
Infoseek
uniquely featured a very complex system of search modifiers including
Boolean modifiers. In November 1996, InfoSeek introduced
Ultrasmart/Ultraseek and redesigned its website the next year to include
channel or directory information. By March of 1998, Infoseek included a
search page with advanced search techniques. Infoseek also uniquely
offered a free web hosting service without advertising and had no limit
on the amount of storage space for users.
In
the summer of 1998, 43% of Infoseek was bought by Disney. From this
point forward Infoseek was part of the Disney Corporation’s vast media
business. The deal included Infoseek acquisition of Starwave
Corporation, including ESPN.com and ABCNews.com. Infoseek’s technology
was then merged with Starwave’s to form the Go.com network. In July
1999, Disney had a 72% interest in Go.com and Disney’s media interest
included Disney.com, Family.com, ABC.com, ABCNews.com, and ESPN.com.
Aside
from its relationship with Disney, Infoseek continued to change. In
September 1998, Infoseek began offering a combination of
manually-reviewed and traditional web-search results. The next month,
Infoseek took over Excite as the default search engine on Microsoft’s
WebTV and Infoseek Express, a free software website, was launched. In
1999, Li Yanhong, an Infoseek engineer, moved to Beijing, China and
co-founded the search engine Baidu.
In
the summer of 2000, Infoseek’s Ultraseek Server software technology was
sold to Inktomi and renamed Inktomi Enterprise Search. In December 2002
(prior to Yahoo!’s purchase of Inktomi) Ultraseek was sold to a
competitor Verity Inc. Verity re-established the Ultraseek brand name
and development of the product until it was acquired by Autonomy PLC in
December 2005. Autonomy continues developing and marketing Ultraseek’s
site search.
In
February 2001, many Infoseek employees tried to collectively buy out
Infoseek from Disney when Disney laid off the entire Infoseek staff.
Today, the Infoseek.com domain forwards to Go.com and the brand name is
unused in North America. Only in Japan (as Infoseek Japan) and Australia
(Infoseek Australia) is the Infoseek name used.
AltaVista – Born in 1995
AltaVista,
meaning “a view from above,” developed out of research by scientists at
Digital Equipment Corporation’s (DEC) Western Research Laboratory in
Palo Alto, California during the spring of 1995. They were trying to
showcase their computer database system called the Alpha 8400 TurboLaser
that was faster than its competitors. Scientists developed a search
tool to crawl, store and quickly index every word of all HTML web pages
on the Internet. This new search tool was powerful. For example, in
August 1995, it conducted its first full-scale crawl of the web bringing
back about 10 million pages.
The
two key scientists involved with AltaVista’s development were Louis
Monier and Michael Burrows. Louis wrote the crawler (called Scooter) and
Michael wrote the indexer. After testing the new search engine with
10,000 DEC employees, AltaVista opened to the public on December 15,
1995 at altavista.digital.com. Initially, the back-end processing
machines received 13 million queries per day. With its release,
AltaVista became the first searchable full-text database on the World
Wide Web with a simple interface. Over 300,000 visitors used the search
tool on the first day and had 19 million hits by the end of 1996, and 80
million per day at the end of 1997.
Ironically
in 1996, AltaVista started exclusively providing search results for
Yahoo! who would one day become AltaVista’s owner. Before this time, DEC
was acquired by Compaq for $9.6 billion at the start of 1998. Despite
success as a modest search interface, in 1999 Compaq released AltaVista
as a web portal once again ironically hoping to compete with Yahoo! It
may be reasoned that AltaVista’s portal strategy was a cause of its
eventual decline as well as the rise of Google. By 2002, efforts
returned to sharpening the quality of the simple search interface.
In
June 1999, CMGI, an Internet investment company with 20% ownership in
Lycos, agreed to acquire 83 percent of AltaVista. CMGI planned a public
offering of AltaVista in April 2000, but cancelled the IPO as the
Internet bubble collapsed.
Near
the end of 2002, AltaVista became the first Internet search engine to
offer image, audio, and video search as part of a new range of
multimedia functionalities. Additionally, AltaVista gained innovative
recognition for its release of Babel Fish, the web’s first-ever
multi-lingual Internet search. Babel Fish could translate words,
phrases, or entire website to and from English, Spanish, French, German,
Portuguese, Italian, and Russian. With its advanced multimedia search
capabilities and language translation and recognition services,
AltaVista was likely the Internet’s most technologically advanced search
tool at the time.
In
February 2003, Overture purchased AltaVista for $140 million. This
price was a fraction of AltaVista’s valuation of $2.3 billion three
years earlier. Unfortunately, AltaVista’s likelihood to continue its
innovative heritage with Overture Services, Inc. was quickly killed when
Yahoo! purchased Overture at the end of 2003. With this purchase,
AltaVista became Yahoo! using the same search index and user interface.
In May 2008, AltaVista’s Babel Fish translation service was also branded
under its parent company as Yahoo! Babel Fish.
Inktomi – Born in 1996
Inktomi
was founded in February 1996 by Eric Brewer, an assistant professor of
computer science at the University of California Berkeley, and Paul
Gauthier, a graduate student. They were involved in a government
research project funded by the US government’s Advanced Research
Projects Agency on parallel computing between personal computers and
workstations making them function like a supercomputer. In the process
they founded a company based upon their success developing a search
tool.
The
company was named after a mythical Lakota Indian spider known for
cunning rather than brute force. After formation with Dave Perterschmidt
as CEO, Brewer as the chief scientist, and Gauthier as the chief
technology officer, the company established its first customer with
HotWired who introduced the search engine HotBot in 1996. HotBot
included Inktomi’s Audience 1 software that customized web pages and
advertisements according to the user’s browser. HotBot eventually
evolved into newer versions including 5.0 released in 1998 using a
Windows NT rather than Unix platform. The interface and server
infrastructure was changed to increase usability and offer new features.
Inktomi
was not just a search provider; its mission was to build scalable
software applications at the core of the Internet. As such, Inktomi’s
efforts drove many of the search tool developments and functionalities
of known search tools and also management tools of Internet traffic and
capabilities in general. For example in 1997, Inktomi beta tested web
traffic servers that managed network data flow eliminating bottlenecks
and redundant Internet and corporate Intranet traffic and bandwidth. The
servers used caching to create localized repositories of information.
This
moved information closer to the user rather than relying each time on
the Internet’s backbone. At the time, the amount of redundant traffic on
the Internet was between 40 and 80 percent. Accordingly, a large market
existed for such technology with Internet service providers, network
providers, and business enterprises. A large customer who took interest
in Inktomi’s traffic server technologies was Microsoft in 1997.
Microsoft agreed to use Inktomi’s traffic server and search engine
technology in the Microsoft Network starting in 1998.
Starting
in 1998, Inktomi signed many customer deals. America Online and Digex
Inc. became licensees of Inktomi’s Traffic Server and began using it in
their own networks later that year. Inktomi also worked out deals with
Digital Equipment Corporation and Intel to port Traffic Server with
their Unix and Windows NT platforms. Also in May of 1998, Yahoo! decided
to use Inktomi’s search engine technology as their preferred choice.
All of these deals influenced Inktomi’s initial public offering in June
1998. The price of the share the first day doubled from $18 to $36 and a
month later was trading at $90. In November 1998 it had climbed to $130
a share.
Some
other significant events in 1998 included the release of Traffic Server
2.0 with a streaming media cache, more protocols, and other performance
and support functions. That same year, Inktomi acquired C2B
Technologies for about $90 million, helping development of shopping
search abilities and services for customers like Yahoo! and New Media.
The shopping engine debuted in the spring of 1999. About the same time,
Inktomi acquired ImpulseBuy.net for $110 million, providing a database
and more capabilities for merchants. As its ecommerce shopping services
expanded in 1999, Inktomi upgraded and released Traffic Server 3.0.
Traffic Server 3.0 included more support for operating systems including
Windows NT and new application programming interfaces allowing third
party providers to provide value-added services. As a result, with
Traffic Server 3.0’s release, Inktomi announced partnerships with six
service partners.
In
August 1999, a secondary stock offering was made raising $300 million.
About the same time, Inktomi purchased WebSpective Software for $106
million. Towards the end of the year, Inktomi released Traffic Server
E5000 and Traffic Server E200. These traffic servers helped corporate
networks manage data for users without having to use the entire network
and server infrastructure. At the end of 1999, America Online dropped
Excite for Inktomi’s database to power its search engine. Likewise, MSN
dropped AltaVista, whom they had moved to using earlier in the year, for
Inktomi.
In
2000, Inktomi created alliances with several partners to enter the
wireless Internet infrastructure market. In June 2000, Inktomi acquired
Ultraseek from Infoseek for $344.7 million. Inktomi announced creation
of a 500 million record search engine database called GEN3. In August
2000, Inktomi created an alliance with America Online and Adero Inc. to
form the Content Bridge distribution network. The Content Bridge would
allow web producers and hosts of information and ecommerce, to pay to
have their content pushed to caching servers of a large network of
Internet hosting delivery providers. Right before the Content Bridge was
to begin operation in January 2001, Adero backed out and sold its
interest to Inktomi for $23.5 million.
In
another big business deal in 2000, Inktomi acquired FastForward
Networks for $1.3 billion. FastForward Networks dropped its name and
became a part of Inktomi’s media division. FastForward Network was a
software developer for Internet broadcasts providing support for
thousands of simultaneous Internet broadcasts. With this acquisition,
Inktomi was able to release a product suite called Media Distribution
Network that handled distribution of steaming media in a network. The
Media Distribution Network suite complimented well their Content
Delivery Suite for managing and distributing static content.
In
2001, Inktomi introduced its Search Everywhere solution integrating
Inktomi’s various search products. Inktomi also commenced enhancing its
search engine software with enterprise-level XML (Extensible Markup
Language) and more comprehensive search results including relevance,
classification and rankings. Inktomi added a distributed crawling
architecture scanning the web more frequently with content blending from
separate databases. In February 2001, Inktomi released Traffic Server
4.0 extending the platform to Linux operating systems with increased
processing power and performance. In May 2001, Inktomi introduced a
pay-for-placement program called Index Connect in which participants
could submit meta information about multimedia and other files enabling
them to appear in search results.
About
the same time, in order to lessen the strain of its ecommerce business,
Inktomi sold off its ecommerce division with its shopping search
engine and customer base to e-centives, an online marketing firm.
Meanwhile, Inktomi expanded its content distribution services. In July
2001, Inktomi acquired eScene Networks with its streaming media sales.
From this acquisition was developed the Inktomi Media Publisher with
capabilities for business to catalog, index and publish their multimedia
content.
Despite
successful acquisitions and product releases, Inktomi sustained
financial losses during 2001. With the dotcom bubble bust came economic
strains to the Internet business sector. As a result, Inktomi cut back
its workforce. Inktomi’s stock values decreased, causing a significant
loss for the year.
In
March 2003, Inktomi was purchased by Yahoo! for $235 million. Inktomi
continued to provide results to Yahoo! rival MSN Search. Meanwhile,
Google continued to provide results to Yahoo! In February 2004, Yahoo!
replaced Google with a search engine based on Inktomi’s technologies.
Ask Jeeves (now Ask) – Born in 1997
Ask
was developed in 1996 by Garret Gruener and David Warthen and launched
in 1997 as Ask Jeeves. In 2006, the “Jeeves” name was removed; revamping
its image after Ask Jeeves was purchased in 2005 by Barry Diller’s InterActiveCorp (IAC).
Originally as Ask Jeeves, human editors listed the prominent sites
along with paid listings and results pulled from partner sites.
Following acquisition of Direct Hit in 2000 and Teoma in 2001, Ask
commenced developing its own search technology. With financial growth,
Ask has acquired other companies including Excite and iWon.
Today, with emphasis on paid inclusion listings, Ask struggles for market share against Google, Yahoo!, and MSN Search.
Google – Born in 1997
Google
was founded in 1998 as another school project at Stanford University in
California. In January 1996, Stanford PhD students Larry Page and
Sergey Brin began researching the concept of a search engine based on
relevancy ranking. Page and Brin believed that search engines should
analyze and rank websites based on the number of times search terms
appeared on web pages. Likewise, Page and Brin developed a search engine
nicknamed “BackRub.” BackRub checked the number and quality of links
coming back to websites in order to estimate the value of a website.
Brin and Page’s research eventually led them to develop the trademarked
PageRank™ link analysis algorithm that Google’s search engine would use
to assign a numerical weighting to hyperlinked document elements.
In
2000, Google replaced Inktomi as the provider of search results to
Yahoo! and later AOL and Netscape. Even though Yahoo! broke away from
Google in 2004, its market share has continued to grow to account for
about 70 percent of all web searches. Google’s market share has steadily increased over the years.
MSN Search (now Windows Live) – Born in 1998
MSN
Search was a service offered as part of Microsoft’s network of web
services. The Microsoft Network debuted as an online service and
Internet service provider in August 1995. During the 1990s, Microsoft
launched Internet Explorer as a bundled part of their operating system
and software products. MSN Search first launched in 1998 displaying
search results from Inktomi and later blending results with Looksmart.
For a short time in 1999, AltaVista search results were used instead of
Inktomi. Since 2004, MSN Search began using its own built-in search
results. Since this time, MSNBot has continually crawled the web. Today,
image search is powered by Picsearch. MSN Search was renamed Windows
Live in 2006.
Overture – Born in 1998
Overture
is considered the pioneer of paid search. It was launched by Bill Gross
in 1998 as GoTo. Overture was overpowered by Google when AOL selected
Google as an ad partner. Overture bought AltaVista and AllTheWeb with
hopes of gaining leverage against Google, but they ultimately were purchased by Yahoo! in 2003.
AllTheWeb – Born in 1999
AllTheWeb
was launched in 1999 showcasing FAST’s Web Search Division search
technologies. It is sometimes referred to as FAST or FAST Search. In
April 2003, ALLTheWeb was bought by Overture for $70 million in 2003 and
rolled into Yahoo! Search after Yahoo! purchased Overture.
