Organizing the World's information

"My guess is (it will be) about 300 years until computers are as good as, say, your local reference library in doing search,"says Google's first employee and director of technology Craig Silverstein. "But we can make slow and steady progress, and maybe one day we'll get there." (Inside the Wide World of Google CBS News, March 28, 2004). According to CEO Eric Schmidt, people care a lot about information and the possibilities of the unfolding revolution in technology are greater than many of them even realize: “Imagine the scale of the kinds of questions you could ask that you could not ask before.”

The figure is a great compilation of key Google Facts by PingDom. Among the key technical facts not shown here are far-reaching inventions such as Programmable Search Engine (PSE, see patent application) based on the BigTable database (See Research Publication about it in PDF format) and other systems that could help it become more semantic.

The distributed storage system for managing structured data called Bigtable resembles a database sharing implementation strategies with parallel and main-memory databases. Instead of a full relational data model it uses a simple data model with data indexed using row and column names that can be arbitrary strings. A Bigtable is a sparse, distributed, persistent multidimensional sorted map. The map is indexed by a row key, column key, and a timestamp; each value in the map is an uninterpreted array of bytes, although clients often serialize various forms of structured and semi-structured data into these strings, controlling it through careful choices in their schemas.

PSE and other integration technologies may be providing a higher level of semantic analysis.
These techniques could figure out the meaning of content and “fill in the blanks” when an item of information is ambiguous or missing. The idea is to enrich an information object with additional tags so that queries about lineage (where something came from) and likelihood of accuracy (the “correctness” of an information element) can be used to generate a result.

Another new concept is a probabilistic mediated schema automatically created from the data sources. Semantic mappings between the schemas of the data sources are mediated by schemas with probabilities attached to each - to model uncertainty at its core. A deterministic mediated schema created from the probabilistic ones will be exposed to the user who could use the terminology of this mediated schema to interact with the system.

The Semantic Web is emerging to help us get the most out of the world's information. Many interesting applications are already here. Some of them already acquired by major search players - Bing, for example, is based on semantic technology from Powerset that Microsoft purchased in 2008. This blog article is only about one of the players organizing the world's information. Stay tuned for more.

Related articles by Zemanta

• Making Sense of Mountains of Data (technologyreview.com)
• NoSQL conference coming to Boston (radar.oreilly.com)
• Scalable Databases - From Relational Databases To Polyglot Persistence (slideshare.net)
• Is Semantic Technology Real? KM World Session Notes (fastforwardblog.com)
• Yahoo! looks beyond Google 's data cruncher (go.theregister.com)
• Creaking ahead toward the semantic web (matei.org)
• Cognition's Semantic Technology Contributes to Microsoft's Bing (prweb.com)
• Web 3.0 Conference: "Semantic Web? It's already here." (expertsystem.net)

Building dynamic Web 3.0 databases

This world is too complex to be described by rigid tables. The 40-year old relational technology may be here to stay, but not to rule.

A few new data storage technologies are currently getting attention, in particular graph databases such as AllegroGraph and Neo4j that allow to “structure on the fly” by using a multi-relational graph data model.

Check this slide share presentation: Graph Databases and the Future of Large-Scale Knowledge Management, by Marko Rodriguez from Los Alamos National Laboratory.

See also this short video on key technology dimensions and a longer talk by
Franz' Jans Aasman demonstrating some capabilities of his AllegroGraph RDF graph database:
NYCSW GeoSpatial, Temporal Reasoning with AllegroGraph from Morton Swimmer on Vimeo.

More on Neo4j:

Neo4j -- graph databases presentation

View more Microsoft Word documents from emileifrem.

Aurametrix is an early phase company working on Web3.0 & Health 4.0 technologies