Do we need the semantic web?

What kinds of applications do we need a semantic web for?  Is the semantic web practical?  These questions (among others) were posed by Jamie Taylor of Metaweb Technologies to a group of panelists at the Text Analytics Summit last week. The panelists were no lightweights.  They included Vladimir Zelevinsky from Endeca, Ron Kaplan from Microsoft, and Kathleen Dahlgren from Cognition.  I found this to be one of the most engaging segments of the Summit.

First of all, many people define the semantic web as a “web of meaning” or a “web of data” that will allow computer applications to exploit the data directly.  Check out the W3C webpage for more information about definitions.  The panelists at the Summit got into an interesting discussion about parsing data sources for the semantic web.  Here are a few of the highlights.  Please note that I asked some additional questions after the panel, itself, so if you’re reading information you didn’t hear on the panel this is the reason.

• What kind of applications is the Semantic Web good for?  It depends what you want to know.  For example, one of the panelists pointed out that you don’t need the semantic web to find a hardware store in Boston.  However, more unique queries might require it.   Most people have had the experience of knowing what they are looking for and using a five or six word query and still not finding it.  The panelists pointed out that entities (people, places, things) were relatively easy to extract; it is the relationships between the entities that is harder.   Vladimir Zelevinsky explained it like this in terms of information retrieval need/information retrieval technologies:

• Known Item Search -> Keyword Search (e.g., Google – where you need to find what you know exists);
• Unknown Item Search -> Guided Navigation (e.g., Faceted search – where you need to explore the data space);
• Unknown Relationship Search -> Semantic Web (where you are looking not for separate items in the repository, in this case the web, but for the connection(s) between them).

The semantic web could pay off in applications that require understanding the relationships between these entities. Ron Kaplan also noted that semantic web technology provides a standard way of merging data from different sources, and that will probably enable some useful new applications.

• Scaling the semantic web. Everyone seemed to agree that manually tagging documents is a brittle exercise. Vladimir Zelevinsky from Endeca suggested putting a parser on each machine.  He said that since you type slower than 1 sentence per second that at the moment of creation, semantics could be injected into the document.  Of course, it is a bit more complex than this, but  this was an interesting notion. Kathleen Dahlgren from Cognition said that NLP at scale was the wave of the future. NLP is complex but deeply distributed.  Computers are getting faster and cheaper, and this can make it fast and scalable.
• Is it practical?  There is a huge amount of data out there and it keeps changing. There is also a lot of duplicate information on the web.  Is it economically viable to think about parsing the web?  Ron Kaplan said he had done a back of the envelope calculation using the following assumptions:

“The simple order-of-magnitude calculation goes as follows:  There are roughly 2.5M seconds in a month, so an 8-core machine gives you 20M cpu seconds.  If it takes 1 second on the average to process a sentence (an upper bound), then you can do 20M sentences per month.  If a web page has on the average 20 sentences, you get 1M pages per month per machine. So, 1000 machines can do a billion pages per month. More if 1 second over estimates, less if 20 sentence/document underestimates.”

So this is economically feasible. If there is a need.  And that remains the question. Is it economically viable and necessary to try to find the information in the long tail?