Linking news stories to related ideas and conversations in real-time makes for a richer online news experience. The current front-runner in this field of experimentation has been Google News, which has quickly established itself as an innovative aggregator and created an online global newsstand. While we are big fans of Google News, it is at times is a bit flat. Its clustering algorithms are adept at finding similar stories from thousands of news sources. But the effect is a bit bland since many of the stories are redundant.
But what if Google gathered not only news stories but connected them with similar conversations happening around the Web? Steven Nieker and Martin Remy, an energetic and innovative team, has decided not to wait for Google and are the pushing the envelope a little farther. Their efforts have resulted in the recent launch of the The Waypath Project.
We asked Steve Nieker to share his thoughts on how such a project might effect how we get our news.
Q. Who are you and what got you interested in doing this?
We are two people who have been working on cutting-edge information-retrieval systems since the mid-nineties. A few years ago, we decided we couldn’t stand the state of information retrieval on the Web any longer, so we set out to build something that would provide a more intuitive user experience than keyword search, but that would scale without the enormous human involvement required of taxonomic systems.
I should point out that we happen to also be the original authors of Think Tank 23’s Nav4 technology, which is at the heart of the Waypath Project. (Think Tank 23 markets products based on this technology to corporate audiences.) However, the Waypath Project is an unfunded effort that we work on in our spare time. We’ve used our “influence” at Think Tank 23 to secure a few resources and licenses to the technology, but other than that, the Waypath Project is unfunded and receives no support from Think Tank 23.
We chose weblogs and online journals as a place to start because they comprise an active, educated community that we believe is as knowledgeable about the Web and as frustrated with its inefficiencies as we are, if not more so. With such an energized base of users and contributors, we hope the Waypath Project will grow into a community effort. So far, it looks promising; since our launch last week, we’ve already received a great deal of positive feedback and help tracking down and testing bugs.
Q: How you foresee larger news sites like CNN.com or nytimes.com taking advantage of this technology? How would that change the online news experience?
The technology we’ve developed is ideal for online publishers like CNN.com and nytimes.com. One of Think Tank 23’s partners, Clickability, serves this community and is using our technology to drive contextual navigation at some of the best-known publications on the Web. Log analysis has shown us that sites using our technology benefit from increased traffic, longer sessions, and reduced keyword search usage. For the publisher, that means their readers are finding what they want and sticking around to see more; it also means more opportunities to sell archived content.
From a user’s perspective, the impact of this technology on the news experience is significant. Now, instead of just getting the current news and a few hand-picked links to recent stories, you get one-click access to the whole context of the event; you can surf along conceptual paths to get in-depth information and commentary from every perspective and about every event leading up to today’s news.
We firmly believe that both contextual navigation and the general approach of fully automating its generation represent the future of information retrieval in large systems, from the Web down to individual publishers down to the average person’s 10K-document email client. Right now, the industry has to get over it’s dead-end fascination with taxonomic systems.
The recent launch of Google’s news site should help validate the approach we’re pioneering. Google News seems to use a similar approach to aggregate news of similar theme. However, we think that this is not a particularly useful application, because it simply groups news articles that contain largely the same information: if you’ve read one, you’ve read them all. More interesting is to apply this technology along the history of news, which is what news publishers using our technology are doing.
We’re hoping that the Waypath Project will encourage news sites to add a new dimension of information to their sites, weblogs and online journals. Weblogs represent the apex (or abyss, depending on your perspective) of the information revolution catalyzed by the Web. Weblogs take the news “authority” away from the news syndicates and give a voice to the people affected by the news. The Waypath Project provides a vehicle for quick, contextual access to the human aspect of the day’s news.
We’re planning to launch a news component of the Waypath Project in the next few weeks. It will show current headlines and what bloggers are saying about them. Right now, we’re working on aggregating a news feed. Interestingly, neither Moreover nor NewsIsFree wanted anything to do with us, so we’re building our own aggregated feed from resources we track down through Syndic8, another great community resource for locating and utilizing online information.
Q: What have you changed since launching?
Since launch (about a week ago), it’s been largely bug fixing. There was an argument added to the embedded results generator for excluding descriptions, but that’s hardly worth mentioning.
Within the next day or so, we’ll be adding a keyword search to the site. Keyword search is not what the Waypath Project is all about, but it seemed silly to make people go to two different sites for all their weblog-surfing needs, so we’ve added one. It will launch with a newly updated index comprising over 213,000 weblog entries; and now that keyword search is out of the way, we’ll be updating the index regularly, probably daily.
There are a few substantial efforts in the works, such as filters to exclude one’s specific weblog from one’s results or, alternatively, limit results to only one’s weblog, and the news feed tie-in I mentioned in the last round. Those may take a few weeks to see light.
Q: Do you think Google will try to do something similar to Waypath?
I’m afraid we don’t have any insight to offer regarding Google’s future. Of course, their search engine already includes weblogs; adding weblogs to Google News would be unusable with its current UI; which leaves the option of Google developing it’s own Waypath Project-like service.
This is a good place for me to plug the technology. The technology behind the Waypath Project is not trivial. In fact, Martin and I have been honing our engine (now in its second incarnation) since 1999. Ad-hoc, dynamic categorization is a complex field, in which there are many dead-end paths of exploration and in which the results are often dramatically different than your hypotheses predict. It takes experience, patience, and a fair amount of luck to come up with a system that works as well as ours does. We’re very proud of our work and have no doubt that it will remain unique (a patent is already pending) — and at the top of its field — for a good while to come.
Given the complexity of the technology, it’s difficult to say what the business case would be for Google to construct a service similar to the Waypath Project. There isn’t much opportunity for income and they already have the loyalty of the weblogging demographic, so there’s little revenue or market-share incentive for them to invest in the technologies and infrastructures necessary to pull it off.
A more sensible place for Google to focus its energies is on making their News service more useful, building an API, and licensing the feeds to businesses. This would let them use the technology they have built to compete with the established markets of companies like Moreover and NewsKnowledge. If Google doesn’t, and we have time, this is where the Waypath Project is likely to go next.
However, if Google wanted to build a Waypath Project of its own, the organization is certainly well funded enough to have a go at it. Of course, it would be easier for them to just buy the Waypath Project. Mention that, if you talk to them, will you?