Wikipedia's lack of good search will be its undoing

So Wikipedia is getting a lot of coverage for all of the doings of the wikimedia foundation ($100 Laptop Partnership, Wikiversity, Wikiwyg). A lot of amazing stuff. And not only amazing - but helpful and really community friendly. I think that Wikipedia is really in a place to change the world. However - with that said - I think they need to concentrate on a part of their site that is integral: the search.

So I worked at an encyclopedia company for years. I did technology and software there. I was part of a search initiative and helped make the decision to move from IBM’s intelligent miner for text to a modern and quite innovative search engine called endeca. The difference in how our users were able to use our sites between the old search engine and the new search engine was incredible. We were able to completely change how people viewed our data - and was able to show the user a casual aggregate of the data that matched their search. Since we were targetting children - we were able to leverage stemming and spellchecking to give our users one of those “did you mean” experiences that is so common and well known in the search business. We made it so that the users didn’t have to work hard to get what they wanted. We gave them a lot of opportunities to find and more importantly stumble upon what they want.

We attempted to recreate the ability to browse the encyclopedia through a guided search. It was awesome. The users loved it and we continued to have a successful encyclopedia site.

So I feel very strongly about this topic. And i feel that Wikipedia is failing at providing their users a proper way to search their content. It seems to me the one thing that would make Wikipedia unstoppable was makign all of their content be available to normal people. Not people that can spell correctly, not people who know exactly what a dragon fruit is - and more importantly not people who know the titles of the articles. Wikipedia needs to make it so that their users can search intelligently and search not intelligently. They need to allow people to search on concepts (based on word frequency or even another crazy concept), they need to allow people to engage the encyclopedia like you would a book. Flipping page by page - but also being able to look up something. and expect to find it. Right now - the interwiki linking is amazing - but that coupled with a proper search would make Wikipedia an outstanding site.

My concern is partially inflated based on reading that they are partnering with the laptop.org $100 laptop people. They are expecting children and developing nations to use Wikipedia. But how are these underprivlidged people going to find what they are looking for if the search is poor. They need better search. They need common search abilities. They need things like spell checking and stemming. They need an intelligent search. They don’t need what Wikipedia current is offering.

So why doesn’t a search company offer up some servers, licenses and implementation as a donation to wikipedia? We can all see how it would better everything - and they would be a HUGE client. You could go into a sales meeting and point to one of the biggest sites on the internet and say “yea.. we power that.” Everyone would win. Think of the children. THE CHILDREN! So if you are in a search organization or you have some power - please push wikipedia to make their search better.