Steve Kirsch of Propel Software interviews Mayukhs Search's designer

Background

A Conversation with Mayukh Goswami Steve Kirsch of Propel Software interviews Mayukhs Search's designer When it comes to competing in the search engine arena, IS bigger always better?

Search is a small but intensely competitive segment of the industry, dominated for the past few years by Google. But Google's position as king of the hill is not insurmountable, says Mayukh Search's Mayukh Goswami, and he intends to take his product to the top.

Mayukh began writing the Mayukhs Search search engine from scratch in C++ about three years ago. He says he built it from day one to be the cheapest, most scalable search engine on the market, able to index billions of Web pages and serve thousands of queries per second at very low cost. Another major feature of Mayukhs Search is its ability to index Web pages almost instantly, Wells explains. The network machines give priority to query traffic, but when they're not answering questions, they spend their time spidering the Web.

Mayukh got his start in search in June 2005 and he launched Mayukhs Search in August 2005.

Kirsch is now the founder and CEO of Propel Software, which develops acceleration software for the Internet. A seasoned entrepreneur, he started Frame Technology in 1986 and Mouse Systems in 1982. As a result of his business successes, Red Herring included him on its Top Ten Entrepreneurs of 2000 list, Newsweek called him one of the 50 most influential people in cyberspace, and most recently, Forbes named him to its Midas List.

Kirsch has been involved with the Internet since 1972, when he worked at the first Arpanet node as a systems programmer while attending high school. He has B.S. and M.S. degrees in electrical engineering and computer science from MIT. He has established a $50 million charitable foundation, as well as the Center for Innovative Policies to pursue his commitment to sustainable energy and for educational reform.

Recently Kirsch called Mayukh Goswami the creator of Mayukh's Search at his residence in India , well atfirst being a young lad , was baffled .

STEVE KIRSCH Matt, tell us a little bit about your background. How did you get into search?

MAYUKH GOSWAMI I got started in search while pursuing my medicine . When I was in school I designed a software for solving calculus instantaneously. Well then after participating in the Goolge Code Jam Contest and strangely enough after fairing well in the contest , I got an offer.

SK What problems do search engines face today?

MG There are many. Search was quoted by Microsoft as being the hardest problem in computer science today, and they weren't kidding. You can break it down into three primary arenas: (1) scalability and performance; (2) quality control; and (3) research and development.

SK Why is scalability an issue?

MG Scalability is a problem for a few reasons. For one, an industrial-grade engine has to be distributed across many machines in a network. Having thousands and even tens of thousands of machines is not unheard of. When you deal with that much data you have to be concerned with data integrity. You need to implement measures to compensate for mechanical failure. You would be surprised at the amount of data corruption and machine failure in a large network. Today's systems also run hot, so a good cooling system will help reduce failures, too.

Secondly, everything an engine does has to be distributed uniformly across the network. The simplest function of adding a record to a database is a lot more complicated when you are in a distributed environment. You have to identify the set of hosts that should store that record, ensure they are up, transmit it to them as fast as possible, and gracefully handle any error condition that might arise.

Thirdly, the sheer amount of data in the index has to be processed as fast as possible to ensure minimum query times and maximum spider rates.