Here's the state of play as I see it: it is expensive and difficult to borrow and this shows no sign of change; the US debt is rising instead of falling, propelled by the Iraq War and the reliance on China for material goods unreciprocated by a reliance from China on American goods; and this adds up to difficult times for business in America for at least three years and possibly longer. From these premises, it's possible to cautiously guess at what the future will hold. (Bearing in mind that every day brings new revelations about the grim state of world finance, so the crystal ball is murky at best)

First, this recession will be good for innovation because recessions generally are. During boom times, companies direct development and occupy great talent with at best evolutionary improvements over the state of the art. Companies are great chasers of new things, but aren't great at making new things. A recession means technologists cease to be paid vast amounts to duplicate the work of others. The Great Tech Bust of Ought Two gave us 37Signals, Flickr, and del.icio.us and there's a strong argument to be made that many companies spent the next six years chasing what they created.

Second, this recession will be great for free and open source because of the shortage of cash. Last recession saw the mainstream legitimisation of open source operating systems (youngsters, take note: there was a time when it wasn't automatically okay for an IT department to use Linux) because it was clear and away the most cost-effective choice. The saying I use is, "come for the price, stay for the quality". Perhaps this recession will legitimise many of the applications (CRM, finance, etc.) higher up the stack. (However, I'm not about to stick my neck out and predict 2009 as The Year of the Linux Desktop)

Third, open source services and cloud computing will benefit from the tight financial situation where conditions will favour opex and not capex. It wil be nigh impossible to borrow to buy hardware or a major software license. An open source software product is free to get through the door, and services around it are delivered from opex not capex. Similarly, cloud computing lets a company pay a little to use someone else's enormous capital investment. It looks like, if the rumours are true, Microsoft will launch Windows Cloud just in time. Don't expect to see anyone else putting in new data centres any time soon—in fact, the days of deep-pocketed investors covering high burn rates are over for a while.

Most consumer apps will be a harder sell with the US dollar in the gutter while the country haemorrhages cash overseas. This is bad but won't make profit impossible, you just have to really be making something consumers need. Apps like Wesabe might find a whole new audience in a recession (disclaimer: O'Reilly is an investor in Wesabe). The conditions don't suit speculative acquisitions, so expect a return to the focus on the bottom line that (very briefly) characterised the fallout from the '01 tech bust. Sorry, dreams of getting people to pay for your toothpick collector social network may have to wait until the return of the stupid money in 2013.

As Phil Torrone said, people will have more time than money. This is good for open source software, but also for hardware and Make-style reconnection with the objects around us. The low-cost high-impact physical events we've created (Ignite, hacker meetups, coworking spaces, foo/bar camps) will thrive even as big-ticket conferences feel the effects of pinched pennies. The killer app in the "web meets world" space may just come from a Maker with spare time who sees a great need.

That's how I see the world and what I think it might favour and disadvantage. How do you see it? What am I missing? Share your views in the comments, and a Head First SQL fridge magnet set for the commenter whom I find the most insightful.

Always know where you are.

You’ve arrived in a new city, a new continent, a new coffee shop. You don’t really know where you are, and are looking for a good place to eat. You pull out your laptop, fire up Firefox, and go to your favorite review site. It automatically deduces your location, and serves up some delicious suggestions a couple blocks away and plots directions there.

In order for this to be a possibility, your browser needs to know where you are.

To do this, future versions of Firefox plan on supporting the new W3C Geolocation Specification, which adds the native ability for Web sites to request, and you to optionally grant access to, your location.  We’re still working out the specifics, but we’re hoping that location will be provided by one or more user selectable service providers and methods, e.g. GPS-based, WiFi-based, manual entry, etc. You’ll be able to play with this in the upcoming beta releases of Firefox 3.1, as well as alpha releases of Fennec.

We realized, though, that some of our Firefox 3 users might also want to get a head start playing with gelocation Today.

Introducing Geode, an experimental add-on to explore geolocation in Firefox 3 ahead of the implementation of geolocation in a future product release. Geode provides an early implementation of the W3C Geolocation specification so that developers can begin experimenting with enabling location-aware experiences using Firefox 3 today, and users can tell us what they think of the experience it provides. It includes a single experimental geolocation service provider so that any computer with WiFi can get accurate positioning data.

The potential here is for more than just resturant lookups. For example, imagine an RSS reader that knows the difference between home and work and automatically changes it’s behavior appropriately. Or a news site whose local section is, in fact, actually local. Or Web site authentication that only allows you to login from certain physical locations, like your house.

What else will location make possible? Even if you can’t code, you can share your thoughts by commenting on this post or via the Concept Series, a forum for surfacing, sharing, and collaborating on new ideas and concepts.

How It Works & Privacy Implications

With Geode when a web site requests your location a notification bar will ask how much information you want to give that site: your exact location, your neighborhood, your city, or nothing at all.

We’re using Skyhook’s Loki technology to map the Wifi signals in your area to your location. Unlike normal GPS-based methods which can take upwards of 45 seconds for a lock, Geode works both inside and outside with an accuracy of between 10 to 20 meters, normally within a second.

Please note that in this early implementation, both location and IP information is sent to the current provider, Skyhook, everytime a website is granted access to your location. Skyhook’s privacy policy is that they do not store or use any personal identifying information, and they promise to only keep data in anonymized agregate. The ultimate plan for Firefox is that service providers and geolocation methods will be pluggable and user selectable — to provide users with as many choices and privacy options as possible.

As an experiment, Geode is also the beginning of a conversation about location-based privacy and integrating services that share personal data into Web browsers.

Download & Try It Out

You can download Geode here.

To see Geode in action you can check out the demo Food Finder, which shows you the cafes and restaurants within walking distance.

To kick off Geode, two other websites have started innovating with location. Both require accounts before you can try them out.

Pownce is a service that makes it easy to send stuff (music, photos, events, and messages) to your friends. Adding the power of location — where you are when you uploaded a photo or sent a message — paints a compelling picture that helps you discover friends and activities around you. Head there to see it in action (you’ll have to sign up for an account).

Yahoo! Fire Eagle is a service that acts as a broker for your location, creating a single place where any web service, from any device can get access to your last updated location. Fire Eagle integrates with Geode so that you’ll never have to manually enter your location again. Once you have a Fire Eagle account, you can see Geode working here.

Differences between Geode and Geolocation in Firefox 3.1

Geode and the Geolocation Services in Firefox 3.1 will use the same W3C API for Geolocation, meaning that the same Javascript code will work in both. The still-in-developement Firefox 3.1 version will allow the user to choose a geolocation service provider, which can either be a peripheral device like a GPS, or a web-based service provider like we’ve used in Geode. We’ll be using the feedback we get from Geode, as well as the feedback we see on the upcoming Firefox 3.1 Beta and Fennec Alpha releases, to refine the feature before shipping it in a future Mozilla product release. We’re particularly interested in ensuring that the final implementation is as sensitive to user privacy and choice as possible.

Get Involved

We’ve implemented a portion of the tentative W3C Specification for Geolocation that we’ve been collaborating on. This means that if you add geolocation to your site for use with Geode, it’s future-proofed to work when Firefox (and other browsers) bundle geolocation.

Using Geode on your site as simple as:

navigator.geolocation.getCurrentPosition(function(pos) {
  alert( pos.latitude + ", " + pos.longitude );
})

For two more pedagogical examples, see here and here. For a run-down of exactly what’s implemented, see the Geode wiki page. You can also get involved with the Firefox 3.1 feature either by commenting on the W3C specification, or by participating in the mozilla.dev.apps.firefox mailing list.

Credits

Special thanks to Ryan Sarver at Skyhook Wireless, Leah Culver at Pownce, and Chris Martin at Yahoo! for participating in the development and launch of the Geode prototype.

Thanks to Doug Turner for developing Geolocation Services in Firefox 3.1. And Justin Dolske and Aza Raskin for developing Geode.

In one of my recent post, I mentionned LODr, a semantic-tagging application based on MOAT. While I started it a few months ago, it’s finally online now. I put the code in svn last friday and twitted about it, but did not make any official announcement yet, so here it is. I certainly should have released before, but as the source code involves lots of classes, I wanted to be sure of the architecture.

So, what is it about ?

LODr aims to apply to MOAT principles (in a few words, link your tags to concepts URIs - people URI, Musicbrainz artists, DBpedia resources … - , share those relationships in a community and then tag content with those URIs) to existing Web 2.0 content. So you can “re-tag” your existing Flickr pics, slideshare presentations, etc, using those principles and make your social data enter the LOD cloud. I think focusing on the existing word is important here, as LODr lets you keep your Web 2.0 habits by using your favourite tools, but provides a separate service to semantically-enrich it. I don’t want to go into too much details here, but in brief, some interesting points regarding the applications are:

• While tags / URIs relationships are shared within the LODr community in a central RDF-base (following the MOAT architecture principles), LODr is a personal application, so that you just need to install the software on your webserver to enjoy it. Moreover, as it’s local, you can re-use your data immediately for any mash-up;
• LODr is completely RDF-based. It might be a bit geeky, but as some were recently wondering where are all the RDF-based applications, here’s one. And of course RDF-based means using standard vocabularies, such as SIOC, FOAF, DC, the Tag Ontology and of course MOAT. The RDF-backend is powered by ARC2, so you can enjoy a SPARQL endpoint for your data. Last but not least, each item page features RDFa, using the previous vocabularies, even if you decide not to use MOAT for a particular item (so that any Web 2.0 item you aggregate is RDFa-ized);
• Aggregated data will provide you a complete tagcloud for your social activity (which might be SCOT-ed in the next updates), as seen here. Each tag link redirects to a list of items provided using Exhibit, and you can restrict by source (i.e. the service it’s from) or creation date. And if a tag have been assigned a URI, you’ll get a link to browse the related items using a similar interface;
• When browsing all items tagged with a particular URI, you’ll get suggested some related URIs. Related because of co-occurence as usually in tag-based applications, but also because they’re directly interlinked, or because they share a common property. To avoid information overload, only the URIs you used to re-tag some of your items will be shown;
• The application can be easily extended. LODr uses wrappers to retrieve your data, and each wrapper is only a few lines of code (e.g. 24 lines for the Flickr one). At the moment, wrappers use RSS to retrieve data and the feeds are automatically discovered from the user FOAF profile - dataportability rocks ! Yet, the architecture allows to use authenticated wrappers (to use services API) but also SIOC exports for those tools;
• As the MOAT process is more time-consuming that simple tagging (since you must define tag/URI relationships, at least at the first time as you can do automated tagging after) the URIs can be displayed as labels when you need to choose which one is relevant for your tag (using the inference capabilities described here as not all resources have a direct rdfs:label property ) . When you need a new URI, the application relies on the Sindice search widget, as done in the Drupal MOAT module. And the system then checks if the new URI is valid, but I’ll blog about that particular point later;
• Finally, in addition of the previous features, LODr can be used to discover all the community content. This feature is not provided by the local application, but by LODr.info, that aggregates your RDF data when you re-tag it to provide search capabilities. Then, you can directly list all items linked to a particular URI. Want to find content related to the Forbidden City ? Or to SPARQL ? And to be even more enjoyable, I added a Ubiquity command so that from any Wikipedia page (more services will be supported soon), you can get the list of all related items (through DBpedia in order to find the concept URI from a document page). While it provides a really-straightforward way to discover related Web 2.0 content when browsing the Web, I also hope it can convice people of the complete process.

So, you can simply download the code from the website and install it. For those who just want to have a look, you can check my LODr instance (while you won’t be able to edit it, you can check the display interfaces). As there might be some bugs and I’m still adding features, please consider using the SVN version instead of the tgz. And then, enjoy the power of Linked Data for your Web 2.0 content

Class-level Mappings Now Generalize Semantic Web Connectivity

We are pleased to present a complementary view to the now-famous linking open data (LOD) cloud diagram (shown to the left; click on it for a full-sized view) [1]. This new diagram (shown below) — what we call the LOD constellation to distinguish it from its notable LOD cloud sibling — presents the current class-level structure within LOD datasets.

This new LOD constellation complements the instance-level view in the LOD cloud that has been the dominant perspective to date. The LOD cloud centrally hubs around DBpedia, the linked data structured representation of Wikipedia. The connections shown in the cloud diagram mostly reflect owl:sameAs relations, which means that the same individual things or instances are referenced and then linked between the datasets. Across all datasets, linking open data (LOD) now comprises some two billion RDF triples, which are interlinked by around 3 million RDF links [2]. This instance-level view of the LOD cloud shown to the left was updated a bit over a week ago [1].

The objective of the Linking Open Data community is to extend the Web with a data commons by publishing various open datasets as RDF on the Web and by setting RDF links between data items from different data sources. All of the sources on these LOD diagrams are open data [3].

So, Tell me Again Why Classes Are Important?

In prior postings, Fred Giasson and I have explained the phenomenon of ‘exploding the domain‘. Exploding the domain means to make class-to-class mappings between a reference ontology and external ontologies, which allows properties, domains and ranges of applicability to be inherited under appropriate circumstances [4]. Exploding the domain expands inferencing power to this newly mapped information. Importantly, too, exploding the domain also means that instances or individuals that are members of these mapped classes also inherit or assume the structural relations (schema, if you will) of their mapped sources as well.

Trying to think through the statements above, however, is guaranteed to make your head hurt. When just reading the words, these sound like fairly esoteric or abstract ideas.

So, to draw out the distinctions, let’s discuss linked data that is based on instance (individual) mappings versus those mapped on the basis of classes. Like all things, there are exceptions and edge cases, but let us simply premise our example using basic set theory. Our individual instances are truly discrete individuals, in this case some famous dogs, and our classes are the conceptual sets by which these individuals might be characterized.

To make our example simple, we will use two datasets (A and B) about dogs and their possible relations, each characterized by their structure (classes) or their individuals (instances):

When datasets are linked based on instance mappings alone, as is generally the case with current practice using sameAs, and there are no class mappings, we can say that Rin Tin Tin is both a dog pet and a mammal. However, we can not say that Lassie, for example, is a mammal, because there is no record for Lassie in Dataset A.

So, we thus see our first lesson: to draw an inference about instances using sameAs in the absence of class mappings requires each record (instance) to exist in an external dataset in order to make that assertion. Instances can inherit the properties and structure of the datasets in which they specifically occur, but only there. Thus, what can be said about a given individual (linked via owl:sameAs) is at most the intersection of what is contained in only the datasets in which that individual appears and is mapped. Assertions are thus totally specific and can not be made without the presence of a matching instance record. We can call this scenario the intersection model: only where there is an intersection of matching instance records can the structure of their source datasets be inferred.

However, when mappings can be made at the class level, then inferences can be drawn about all of the members of those sets. By asserting equivalentClass for dog between Datasets A and B, we can now infer that Lassie, Clifford and Old Yeller are canids and mammals as well as Rin Tin Tin, even though their instance records are not part of Dataset A. To complete the closure we can also now infer that Rin Tin Tin (Dataset A) is a pet and a German shepherd from Dataset B. We can call this scenario the union model. The mappings have become generalized and our inferencing power now extends to all instances of mapped classes whether there are records or not for them in other datasets.

This power of generalizability, plus the inheritance of structure, properties and domain and range attributes, is why class mappings are truly essential for the semantic Web. Exploding the domain is real and powerful. Thus, to truly understand the power of linked data, it is necessary to view its entirety from a class perspective [5].

Thus, to summarize our answer to the rhetorical question, class mappings are important because they can:

• Generalize the understanding of individual instances
• Expand the description of things in the world by inheriting and reusing other class properties, domains and ranges, and
• Place and contextualize things by inheriting class structure and hierarchical relationships.

The LOD Constellation

So, here is the new LOD constellation of class-level linkages. The definition of class-level linkages is based on one of four possible predicates (rdfs:subClassOf, owl:equivalentClass, umbel:superClassOf or umbel:isAligned). Because of the newness of UMBEL as a vocabulary, only a few of the sources linked to UMBEL have the umbel:superClassOf relationship and one (bibo) has isAligned.

Note that some of the sources are combined vocabularies (ontologies) and instance representations (e.g., UMBEL, GeoNames), others are strict ontologies (e.g., event, bibo), and still others are ontologies used to characterize distributed instances (e.g., foaf, sioc, doap). Other distinctions might be applied as well:

[click for full size]

The current 21 LOD datasets and ontologies that contribute to these class-level mappings are (with each introduced by its namespace designation):

• bibo — Bibilographic ontology
• cc — Creative Commons ontology
• damltime — Time Zone ontology
• doap — Description of a Project ontology
• event — Event ontology
• foaf — Friend-of-a-Friend ontology
• frbr — Functional Requirements for Bibliographic Records
• geo — Geo wgs84 ontology
• geonames — GeoNames ontology
• mo — Music Ontology
• opencyc — OpenCyc knowledge base
• owl — Web Ontology Language
• pim_contact — PIM (personal information management) Contacts ontology
• po — Programmes Ontology (BBC)
• rss — Really Simple Syndicate (1.0) ontology
• sioc — Socially Interlinked Online Communities ontology
• sioc_types — SIOC extension
• skos — Simple Knowledge Organization System
• umbel — Upper Mapping and Binding Exchange Layer ontology
• wordnet — WordNet lexical ontology
• yandex_foaf — FOAF (Friend-of-a-Friend) Yandex extension ontology

The diagram was programmatically generated using Cytoscape (see below) [6], with some minor adjustments in bubble position to improve layout separation. The bubble sizes are related to number of linked structures (ontologies) to which the node has class linkages. The arrow thicknesses are related to number of linking predicates between the nodes. Two-way arrows are shown as darker and indicate equivalentClass or matching superClassOf and subClassOf; single arrows represent subClassOf relationships only.

Note we are not presenting any rdf:type relations because those are not structural, and rather deal with the assignment of instances to classes [7]. More background is provided in the discussion of the construction methodology [6].

At this time, we have not calculated how many individuals or instances might be directly included in these class-level mappings. The data and files used in constructing this diagram are available for download without restriction [8].

Finally, we have expended significant effort to discover class-level mappings for which we may not be directly aware (see next). Please bring any missing, erroneous or added linkages to our attention. We will be pleased to incorporate those updates into future releases of the diagram.

How the LOD Constellation Was Constructed

Our diligence has not been exhaustive since not all LOD datasets are indexed locally and others do not have SPARQL endpoints. The general method was to query the datasets to check which ontologies used external classes to instantiate their individuals using the rdf:type predicate. The externally referenced ontology was then checked to determine its own external class mappings.

Here is the basic SPARQL query to discover the the rdf:type predicate for non-Virtuoso resources:

select ?o where
{
?s a ?o.
}

And here is the SPARQL query for Virtuoso-hosted datasets (note Virtuoso supports the distinct non-standard extension to SPARQL which is a more efficient way to get listings):

select distinct ?o where
{
?s a ?o.
}

We then created a simple script to go to all of the ontology namespaces so listed in these external mappinigs. If there was an external class mapping in the source with one of the four possible predicates of rdfs:subClassOf, owl:equivalentClass, umbel:superClassOf or umbel:isAligned, we noted the source and predicate and wrote it to a simple CSV (comma delimited) file. This formed the input file to the Cytoscape program that created the network graph [6].

There are possibly oversights and omissions in this first-release diagram since not all bubbles in the LOD cloud were exhaustively inspected. Please notify us with updates or new class linkages. Alternatively, you can also download and modify the diagram yourself [8].

Conspicuous by Their Absence

We gave particular diligence to a few of the more dominant sources in the LOD instance cloud that showed no class mappings. These include DBpedia and YAGO. While these have numerous and useful rdf:type and owl:sameAs relationships, and all have rich internal class structures, none apparently map at the class level to external sources.

However, because of the unique overlap of instances (named entities) in UMBEL, which does have extensive external class mappings, and which have been mapped to DBpedia instances, it is possible to infer some of these external class linkages.

For example, go to the DBpedia SPARQL endpoint:

http://dbpedia.org/sparql/

And try out some sample queries by pasting the following into the Query text box and running the query:

define input:inference ‘http://dbpedia.org/resource/inference/rules/umbel#’
prefix umbel: <http://umbel.org/umbel/sc/>
select ?s
where
{
?s a umbel:Person
}

This example query applies the external class structure of UMBEL to individual person instances in DBpedia because of the prior specification of some mapping rules used for inferencing [9]. The result set is limited to 1000 results.

Alternatively, since UMBEL has also been mapped to the external FOAF ontology, we can also now invoke the FOAF class structure directly to produce the exact same result set (since umbel:Person owl:equivalentClass foaf:Person). We do this by applying the same inferencing rules in a different way:

define input:inference ‘http://dbpedia.org/resource/inference/rules/umbel#’
select ?s
where
{
?s a <http://xmlns.com/foaf/0.1/Person>.
}

UMBEL can thus effectively act as a class structure bridge to the DBpedia instances.

Since DBpedia is an instance hub, this bridging effect is quite effective between UMBEL and other DBpedia instances in the LOD cloud. However, because there is not the same degree of overlap of instances with, say, GeoNames, this technique would be less effective there.

Explicit class-level mappings between datasets will always be more powerful than instance-level ones with class mediators. And, in all cases, both of those techniques that explicitly invoke classes are more powerful than instance-level links alone.

The Linked Data Infrastructure is Now Complete

Though all of the available linkages have not yet been made in the LOD datasets, we can now see that all essential pieces of the linkage infrastructure are in place and ready to be exploited. Of course, new datasets can take advantage of this infrastructure as well.

UMBEL is one of the essential pieces that provides the bridging “glue” to these two perspectives or “worlds” of the instances in the LOD cloud and the classes in the LOD constellation. This “glue” becomes possible because of UMBEL’s unique combination of three components or roles:

• UMBEL provides a rich set of 20,000 subject concept classes and their relations (a reference structure “backbone”) that facilitates class-level mappings with virtually any external ontology with the benefits as described above
• UMBEL contains a named entity dictionary from Wikipedia also mapped to these classes, which therefore strongly intersects with DBpedia and YAGO, and therefore helps provide the individual instances <–> classes bridging “glue”, and
• UMBEL is also a vocabulary that enhances the lightweight SKOS vocabulary to explicitly facilitate linkages to external ontologies at the subject concept layer.

In fact, it is the latter vocabulary sense, in combination with the reference subject concepts, that enables us to draw the LOD class constellation.

So, we can now see a merger of the LOD cloud and the LOD constellation to produce all of the needed parts to the LOD infrastructure for going forward:

• A hub of instances (DBpedia)
• A hub of subject-oriented (”is about”) reference classes (Cyc-UMBEL), and
• A vocabulary for glueing it all together (SKOS-UMBEL).

This infrastructure is ready today and available to be exploited for those who can grasp its game-changing significance.

And, from UMBEL’s standpoint, we can also point to the direct tie-ins to the Cyc knowledge base and structure for conceptual and relationship coherence testing. This infrastructure is an important enabler to extend these powerful frameworks to new domains and for new purposes. But that is a story for another day.

House music made with Pi House Generator Paul Slocum is an artist and musician who combines art and digital culture. His newest work is on display at Dunn and Brown Contemporary in Dallas and is made up of a vintage amplifier hooked up to a laptop running software of his own creation.

This software randomly generates house music using the number Pi. Much like the number itself, the music stream never repeats and constantly continues to evolve.

From Paul : “The software progressively calculates the sequence of digits in pi, starting at 3.14 and progressing towards infinity. As the program calculates the digits, it feeds the results into an algorithmic music generator containing my structural criteria for house music. The resulting piece of house music is infinitely long and static and never repeats itself. The number of processor cycles required to calculate pi increase with the number of digits it is calculated to. After months or years of playing the song, any fixed computer hardware will be unable to calculate the digits fast enough for the song to play continuously. The rate that the number of processor cycles increase per pi-digits is bound by the formula Z*log(N); however based on Moore’s Law, processor power per dollar increases at an exponential rate - doubling every two years. BY upgrading computers regularly with market trends, the song can be played indefinitely.”

Paul may be making his software available for download soon.

MontaVista is hosting right now the annual Embedded Linux Developers Conference, aka MontaVista Vision 2008, in the Palace Hotel in San Francisco. As a MontaVista employee, I'm of course able to get there and attend the conferences, which is for me a great opportunity to meet a whole bunch of really interesting people but also learn a lot from them.

For example, I spent the whole day yesterday into the Device Drivers Hands-on bootcamp taught by Mike Anderson (from the PTR Group), which was a really good introduction to device driver development for Linux. It came as a very good complement to the Device Drivers chapter in Chris Hallinan's Embedded Linux Primer.

I actually met Chris himself today. Meeting the author of a book you're currently reading is a very interesting experience. And while I was giving him some feedback, he also got the chance to gave me a bit more background on the book itself.

Today had a couple of very interesting presentations as well. I guess the schedule is pretty much self-explanatory. And all that ended with a really nice diner party in the main hall :)

I'll attend tomorrow's conferences too, but I can already tell that this year's Vision was a really great conference. The venue may have been a bit too big for the number of attendants ; but the Palace Hotel is just that big. I can't really step back and consider the success of all this from a MontaVista point of view, but I personally liked it and I'm glad that I had the opportunity to attend these conferences at the very beginning of my internship.

The fan contest for making the video of “Reckoner” has ended. This time courtesy of Clement Picon! Here’s the finished version of Clement Picon who’s winning animated video for “Reckoner,” chosen by the Radiohead as the song’s official video.

Now it crawl and index the videos clip of Myspace & you have faceted sources filtering on interface.

Weeloop is a social network dedicated to electronic musics. They provide artists, labels, producers and DJs to promote their art towards listeners, sponsors, medias and night clubs. All the stuff networking are presents, forum, friendlist, video & music sharing, chat room… the new webdesign of this website are pretty well-made.

This is the seventh post in an article series about MIT’s lecture course “Introduction to Algorithms.” In this post I will review lecture eleven, which is on the topic of Augmenting Data Structures.

There are some programming situations that can be perfectly solved with standard data structures such as a linked lists, hash tables, or binary search trees. Many others require a dash of creativity. Only in rare situations will you need to create an entirely new type of data structure, though. More often, it will suffice to augment (to modify) an existing data structure by storing additional information in it. You can then program new operations for the data structure to support the desired application. Augmenting a data structure is not always straightforward, however, since the added information must be updated and maintained by the ordinary operations on the data structure.

This lecture discusses two data structures that are constructed by augmenting red-black trees (see the previous post on red-black trees).

The first part of the lecture describes a data structure that supports general order-statistic operations on a dynamic set. It’s called dynamic order statistics. The notion of order statistics was introduced in lecture six. In lecture six it was shown that any order statistic could be retrieved in O(n) time from an unordered set. In this lecture it is shown how red-black trees can be modified so that any order statistic can be determined in O(lg(n)) time. It presents two algorithms OS-Select(i), which returns i-th smallest item in a dynamic set, and OS-Rank(x), which returns rank (position) of element x in sorted order.

The lecture continues with general methodology of how to augment a data structure. Augmenting a data structure can be broken into four steps:

• 1. Choosing an underlying data structure,
• 2. Determining additional information to be maintained in the underlying data structure,
• 3. Verifying that the additional information can be maintained for the basic modifying operations (insert, delete, rotate, etc.) on the underlying data structure, and
• 4. Developing new operations.

The second part of the lecture applies this methodology to construct a data structure called interval trees. This data structure maintains a dynamic set of elements, with each element x containing an interval. Interval is simply pair of numbers (low, high). For example, a time interval from 3 o’clock to 7 o’clock is a pair (3, 7).

Lecture gives an algorithm called Interval-Search(x), which given a query interval x, quickly finds an interval in the set that overlaps it. Time complexity of this algorithm is O(lg(n)).

The lecture ends with the correctness proof of Interval-Search(x) algorithm.

You’re welcome to watch lecture eleven:

Topics covered in lecture eleven:

• [00:20] Concept of augmenting data structures.
• [02:00] Dynamic order statistics.
• [02:20] OS-Select operation on dynamic order statistics.
• [02:50] OS-Rank operation on dynamic order statistics.
• [03:49] Dynamic order statistics key idea - keep the sizes of subtrees in nodes of a red-black tree.
• [04:10] Example of a tree representing dynamic order statistic.
• [10:10] OS-Select algorithm.
• [16:40] Analysis of OS-Select.
• [17:30] OS-Rank algorithm.
• [20:15] Modifying operations of dynamic order statistics tree.
• [22:55] Example of inserting an element into the tree.
• [26:11] Example of rotating a tree.
• [29:30] Methodology of data structure augmentation.
• [36:45] Data structure augmentation applied to construct interval trees.
• [37:31] Example of time-intervals.
• [39:48] Query operation on interval trees - find an interval in the set that overlaps a given query interval.
• [41:15] Step 1, underlying data structure: red-black tree keyed on low endpoint.
• [45:10] Step 2, additional node information: largest value in the subtree rooted at that node.
• [50:24] Step 3, modifying ops: insert, delete.
• [56:55] Step 4, new ops: Interval-Search.
• [01:00:00] Example of Interval-Search algorithm.
• [01:06:30] Running time of Interval-Search — O(lg(n)).
• [01:07:20] List all overlaps (k of them) in O(k*lg(n)) time.
• [01:08:50] Best algorithm to find all overlaps to date os O(k + lg(n)).
• [01:09:11] Correctness proof of Interval-Search algorithm.

Lecture eleven notes:

Lecture 11, page 1 of 2.
Lecture 11, page 2 of 2.

Have fun augmenting data structures! The next post will be about a simple and efficient search structure called skip list!

PS. This course is taught from the CLRS book (also called “Introduction to Algorithms”):

In this web-enabled world of ours, you have to wonder why business cards are still so popular. Shouldn't there be a better way? A number of startups have attempted to address this problem with ingenious solutions that range from iPhone apps to custom URLs. Others are calling for the use of QR Codes for mobile data exchange. Unfortunately, no one service has hit the sweet spot just yet, but newcomer "E" thinks they have it figured out. Will "E" succeed where the others have failed? Or is this one industry that refuses to become digitized?

Sponsor

HelloMyNameIsE.com

You have to appreciate E's creative URL - it's memorable, but also makes you curious. E? What's E?, you wonder. When I first encountered the URL, it was in a tweet which read "I'm now using E to add friends to my Twitter account. More info on http://hellomynameise.com." Did I click though? You bet.

"E," as it turns out, is a new spin on digital contact exchange. Instead of using paper business cards, you use your phone to exchange data. At first, you may think that sounds very much like mobile contact service Dropcard, but it's not. The only similarity between E and Dropcard is that they both allow you to customize your profile online and share it with others, but the similarities end there.

To use Dropcard, you either text or use a mobile app which emails your contact info to the person you just met. With E, you go to a mobile web URL that lets you exchange a passcode with your new contact. The passcode is simply a five-digit code which is entered into the mobile web app itself. They show your theirs, you show them yours...that sort of thing. Once connected, you don't receive an email message with their contact info like with Dropcard. E goes a step further and actually adds that contact to all the services you've already integrated with E.

Service Integration

At the moment, E allows you to integrate Twitter, PICNIC (a network for the PICNIC conference), and Soocial. However, Delicious, European social portal Netlog, and LastFM are listed as coming soon. After you integrate these services with E, when you add a contact they're immediately added to all those other web services, too. And thanks to Soocial, an address book solution, E contact info can also synchronize with your email address book in Gmail, Highrise, your OSX address book, or the address book on your phone itself.

Barriers To Adoption

E faces one of the typical problems that many web 2.0 startups do - they don't work for you until a lot of people are using it. Just because you have a profile on E, that doesn't mean that those you meet do. And unlike a service like Dropcard, there isn't a way to use E without the other person's involvement.

In addition to the service itself, the developers of E came up with a crazy but interesting idea for a hardware device called the "Connector." With this device, you can exchange contact info with others just by touching the two connectors together. While gadget junkies and shiny object collectors may find this device appealing, it could easily remain a niche gadget that ends up sitting on the shelf next to your Chumby and Nazbaztag. To cross the adoption barrier, those at E would be smart to sponsor events where everyone gets a Connector at registration. After a few high-profile events, they would have industry movers and shakers on board, and that's always a good place to start. Sponsoring events may be just what the company is planning, though, since their site mentions that the "Connector will be released at large events in the near future."

Will It Work?

At present, the E service is very basic. Twitter integration is the only service of note that works yet. (Soocial looks great, but is in private beta). The profiles themselves are also not as flexible as those with Dropcard are. You can easily add and remove services with Dropcard, but with E, I wasn't even able to add a second company that represents my second job. The services section of the web site is confusing - it doesn't allow you to do anything more than customize which services are connected. The actual profile information is entered under "Settings," so you can't specify that only personal contacts get your home address, for example. It appears to be all-or-nothing.

E still has far to go to become a truly successful digital contact exchange service, but at least they're trying something different. Because they operate via mobile URL, not an app specific to any one device, they're better positioned for more universal adoption that a service that designates itself as iPhone-only, for example.

The service is in private beta testing now, but you have the opportunity to make an impassioned plea as to why they should invite you on the signup page here. (If you get in, feel free to add me: 17975.)

Check out the video below to see E in action:

Hello, my name is E from Renato Valdés Olmos on Vimeo.

Two computer engineers ‘throw down’ in an awkward dance-off that seems to echo the development of information technolgy and the internet from 1951 up to the present day. The videoclip features a catchy jingle by pop impresario Jim Guthrie.

This is the second videoclip by Superbrothers.

Filmed at night with the lovely I-5 and San Diego skyline in the background. The frames were photographed with a Canon Rebel using 20-30 second exposure time. The author used a small green LED keychain light to draw each frame. Once all the positions were photographed they were strung together and synchronized to the music in Adobe After Effects.

This webapp electronic music studio is an emulator of machines used by DJs, producers, and bands from all over the world. It includes two TB-303 Bass Line generators, the Roland TR-808 and TR-909 drum machines and two banks of effects pedals including three delays, crusher, detune, flanger, reverb, a parametric equalizer and a compressor.

You can drag virtual cables between any output and any input to customize the setup – like the Reason software. The available version is a demo, but the developers are working on new effects and tools, ways to record pieces easily.

Very simple of use, interested? Give it a shot ! 

Coherence is Needed for Continued Sustainability

Since early in 2008 my colleague, Fred Giasson, has been authoring a series of important blog posts on ‘exploding the domain.’  Exploding the domain means to make class-to-class mappings between a reference ontology and external ontologies, which allows properties, domains and ranges of applicability to be inherited under appropriate circumstances.  Exploding the domain expands inferencing power to this newly mapped information.

Fred first used the phrase in an April post that introduced the concept:

Since that time, Fred has continued to explore these implications.  In an August posting looking at UMBEL as a possible reference framework for mapping and exploding domains, Fred stated,

The thrust of this analysis was to show how UMBEL subject concepts can act to create context (his emphasis) for linked classes defined in external ontologies. Where the individuals in a dataset are instances of classes, and some of these classes are linked to UMBEL (or a similar contextual structure), these subject concept classes also give context to those individuals.

Finally, and most recently, Fred demonstrated how the use of UMBEL could explode DBpedia’s domain by linking classes using only three properties: rdfs:subClassOf, owl:equivalentClass and umbel:isAligned.  (And, as I noted in an earlier posting this week, those mappings have now been made bi-directional from DBpedia to UMBEL as well.)

As we discuss and apply these concepts we are starting to see some further guidelines emerge.  Presenting these is the purpose of this post in this ongoing exploding the domain series.

The Mere Existence of Classes is Not Enough

Since its inception, DBpedia has had a class structure of sorts, first from the native Wikipedia categories from which it was derived and then with the incorporation of the YAGO structure (based on WordNet concept relationships).  Yet we have claimed that class structure has truly only recently been brought to DBpedia with the mappings to UMBEL.  Why?  Does not DBpedia’s initial class structure meet the test?

(BTW, these same questions may be applied to some of the other large data structures beginning to emerge on the semantic Web such as Freebase.  But, those are stories for another day.)

There are really two answers to these questions.  First, the mere existence of classes is not enough; they must actually be used.  And, second, the nature of those classes and their structure and coherence are absolutely fundamental.  This subsection addresses the first point; the following the second; with both aided by the table below.

There has been a class structure within DBpedia from inception, which was then supplemented a few months after release with the YAGO structure.  The starting  Wikipedia structure showed early issues which began to be addressed through a cleaned Wikipedia category class (CWCC) hierarchy.  These relationships were established with the rdf:type predicate that relates an instance to a class.  The classes themselves were related to one another through the rdfs:subClassOf predicate.  These class relationships allowed the linked classes to be shown in a hierarchical (taxonomy-like) structure.

Initially, in the case of the beginning Wikipedia categories, the internal class relationships were weak.  This was somewhat improved with the addition of YAGO and its WordNet-based concept relationships (with better semantics).

However, these class relationships were (to my knowledge) never mapped to any external structures or ontologies.  If used at all, they were only implied for ad hoc navigation within the internal instance data.

Really, anyone could have approached DBpedia at this point and began an effort of mapping its existing class structures to external data.  Indeed, we (as editors of UMBEL) considered doing so, but chose a different structure (see next section) for reasons of context and coherence.

The net result is that DBpedia and the other instances of the linking open data (LOD) cloud have remained focused and useful at the instance level, and not yet at the class level.

As we brought in UMBEL to provide a class structure to DBpedia and linked data, this circumstance began to change, as this table indicates:

Though shown for comparison reasons, the number of classes probably has no real importance.

The key argument in this subsection is that classes matter.  Indeed, one of the current challenges before the linked data community is to understand and treat differently the issues of instances from classes.  But, the question of whether one class structure is better than another is moot if class mappings are neglected altogether.

Sustainability Requires Coherence

UMBEL’s reasons for not taking up the Wikipedia structure or the WordNet structure — that is, the initial structures within DBpedia — for its lightweight subject ontology was based on lack of coherence.  I have spoken earlier about When is Content Coherent? regarding these arguments.  Other analysis supports the conclusions in various ways [1].

A central (or “upper”) reference framework should be one that is solid and venerable.  Over time, many subsidiary ontologies and structures will relate to it.  Like a steel superstructure to a skyscraper or a structural framework to a large ocean-going tanker, this beginning structure needs to withstand many stresses and maintain its integrity as various subsidiary structures hang from it.

So long as we are still in “toy” mode with relatively few external mappings and relative few ontologies, simple class-to-class mappings without respect to the coherence of the underlying ontologies may be OK.  But, we will soon (if not already) see that structural flaws, like slight perturbations at the Big Bang, may propagate to create huge discontinuities.

At the pace of development we are now seeing, there will be tens to hundreds of thousands of ontologies within the foreseeable future.  Granted, for any given circumstance, only a minor few of those may be applicable.  But the potential combinations still can defy imagination in terms of complexity and potential scope at widely varying scales.

At the scale of the Web, of course, there will never be a central authority (nor should there be) for “official” vocabularies or structures.  Yet, just as certainly, those ontologies and structures that do share some conceptual and structural coherence and are therefore more likely to easily integrate and interoperate will (I believe) win the Darwinian race.  Without some degree of coherence, these disparate structures become like ill-fitting jigsaw pieces from different puzzles.

As we watch structures and relationships accrete like layers in a pearl, we should begin with a solid grain of common sense and coherence.  That is why we chose the Cyc structure as the basis of UMBEL — it provides one such solid, coherent framework for moving forward.

I am not sanguine that ad hoc, free-form ontological structures, created in the same manner as topic-specific articles in Wikipedia or as informal tags in bookmarking systems, will bring such coherence.  But who knows?  Perhaps on the Web where novelty and the joy of creation and exploration can trump usefulness, such could transpire.

But, when we look to linked data and semantic Web constructs finally achieving its potential in the enterprise to overcome decades of data silos, I suspect purposeful coherence will win the day.

Sustainable ontologies, which themselves can host and interoperate with still further ontologies and structures, will require coherent underpinnings to not collapse from the weight of keeping consistency.  Just as our current highways and interstates follow the earlier roads before them, as early trailblazers we have a responsibility to follow the natural contours of our applicable information spaces.  And that requires coherence and consistency; in other words, logic and design.

Vocabularies are Not Free Form

In the past few months there has been a remarkable emergence of interest in vocabularies and semantics (as traditionally understood by linguists).  Today, for example, marks the kick-off of the first VoCamp get-together in Oxford, England, with interest and discussion active about potentially many others to follow.  Peter Mika, Matthias Samwald and Tom Heath have each outlined their desires for this meeting.

I hope the participants in this meeting and others to follow look seriously at the issues of coherence and interoperability and sustainability.  My caution is as follows:  like tagging and Wikipedia, we have seen amazing contributions from user-generated content that have totally re-shaped our information world.  However, we have not yet seen such processes work for structure and coherent conceptual relationships.

I believe participation and UGC have real roles to play in the emergence of coherent structures and vocabularies to enable interoperability.  But I also believe they have not done so to date, and useful approaches to those will not emerge in a free-form fashion and without consideration for sustainability and coherence testing.

Putting this Context into Context

In these observations there is absolutely no criticism intended or implied with DBpedia or prior linked data practice.  A natural and understandable progression has been followed here:  first make connections between things, then begin to surface knowledge through the exploration of relationships.  We are just now beginning that exploration through the use of classes, vocabularies and ontologies to explicate relationships.  The fact that linked data and DBpedia first emphasized linking things and publishing things is a major milestone.  It is now time to move on to the new challenges of structure and relationships.

There is much to be learned from other pathbreaking efforts such as the Open Biomedical Ontologies efforts and their attempts at coordination and standardization.  As the demands and interests in interoperability increase, interfaces, consistency and coordination will continue, I believe, to come to the fore.

In another 18 months we will likely look back at today’s issues and thoughts as also naïve in light of new understandings.  The pace of discovery and learning is such that I believe best practices will remain fluid for quite some time.

[1] An exact analysis related to our arguments of coherence has not been made, but related studies point to these observations in part or in slightly different contexts.

Wordnets tend to be star-like in structure, with sparse relations, and dominated by a few hub clusters. c.f., Holger Wunsch, 2008. Exploiting Graph Structure for Accelerating the Calculation of Shortest Paths in Wordnets, in Proceedings of Coling 2008, Manchester, UK, August 2008. See http://www.sfs.uni-tuebingen.de/~wunsch/wn-shortest-paths.pdf.

The strict uncorrected structure of Wikipedia categories can also be inconsistent, inaccurate, populated with administrative categories, demonstrate cycles, and lack uniform coverage. c.f., Jonathan Yu, James A. Thom and Audrey Tam, 2007. Ontology Evaluation Using Wikipedia Categories for Browsing, in Proceedings of 16th Conference on Information and Knowledge Management (CIKM), 2007; see http://goanna.cs.rmit.edu.au/~jyu/publications/YuEtal07.pdf. This paper also presents a comprehensive framework for ontology evaluation.

This reference describes a new way to calculate semantic relatedness (not the same as coherence) in relation to Wikipedia, ConceptNet and WordNet: Sander Wubben, 2008. Using Free Link Structure to Calculate Semantic Relatedness. ILK Research Group Technical Report Series no. 08-01, July 2008, 61 pp. See http://ilk.uvt.nl/downloads/pub/papers/wubben2008-techrep.pdf.

Table 3 in this citation presents an interesting contrast between what the authors call collaborative knowledge bases (CKBs, like Wikipedia) and linguistic knowledge basis (LKBs, like WordNet), again however not really addressing the coherence issue: Torsten Zesch and Christof Müller and Iryna Gurevych, 2008. Extracting Lexical Semantic Knowledge from Wikipedia and Wiktionary, in Proceedings of the Sixth International Language Resources and Evaluation (LREC’08), May 28-30, Marrakech, Morocco. See http://elara.tk.informatik.tu-darmstadt.de/publications/2008/lrec08_camera_ready.pdf.  Also see Torsten Zesch and Iryna Gurevych, 2007. Analysis of the Wikipedia Category Graph for NLP Applications, in Proceedings of the Workshop TextGraphs-2: Graph-Based Algorithms for Natural Language Processing at HLT-NAACL 2007, 26 April, 2007, Rochester, NY, pp. 1-8. See http://www.aclweb.org/anthology-new/W/W07/W07-02.pdf.

Radiohead is back at it again with the mass remix project of Reckoner, a super track from In Rainbows. You can go to the remix site to buy the song ’stems’, upload your own version, or vote for your favorite.  Radiohead is a band that pushes the boundaries again. This is probably the best idea I have seen in a long time !

This is the sixth post in an article series about MIT’s lecture course “Introduction to Algorithms.” In this post I will review lectures nine and ten, which are on the topic of Search Trees.

Search tree data structures provide many dynamic-set operations such as search, insert, delete, minimum element, maximum element and others. The complexity of these operations is proportional to the height of the tree. The better we can balance the tree, the faster they will be. Lectures nine and ten discuss two such structures.

Lecture nine discusses randomly built binary search trees. A randomly built binary search tree is a binary tree that arises from inserting the keys in random order into an initially empty tree. The key result shown in this lecture is that the height of this tree is O(lg(n)).

Lecture ten discusses red-black trees. A red-black tree is a binary search tree with extra bit of information at each node — it’s color, which can be either red or black. By contrasting the way nodes are colored on any path from the root to a leaf, red-black trees ensure that the tree is balanced, giving us guarantees that the operations on this tree will run on O(lg(n)) time!

PS. Sorry for being silent for the past two weeks. I am preparing for job interviews at a company starting with ‘G‘ and it is taking all my time.

Lecture 9: Randomly Built Binary Search Trees

Lecture nine starts with an example of good and bad binary search tree. Given a binary tree with n nodes, a good trees has height log(n) but the bad one has height close to n. As the basic operations on trees run in time proportional to the height of the tree, it’s recommended that we build the good trees and not the bad ones.

Before discussing randomly built binary search trees, professor Erik Demaine shows another sorting algorithm. It’s called binary search tree sort (BST-sort). It’s amazingly simple — given an array of n items to sort, build a BST out of it and do an in-order tree walk on it. In-order tree walk walks the left branch first, then prints the values, and then walks the right branch. Can you see why the printed list of values is sorted? (If not see the lecture ) [part three of the article series covers sorting algorithms]

Turns out that there is a relation between BST-sort and quicksort algorithm. BST-sort and quicksort make the same comparisons but in different order. [more info on quicksort in part two of article series and in “three beautiful quicksorts” post]

After this discussion, the lecture finally continues with randomized BST-sort which leads to idea of randomly built BSTs.

The other half of the lecture is devoted to a complicated proof of the expected height of a randomly built binary search tree. The result of this proof is that the expected height is order log(n).

You’re welcome to watch lecture nine:

Topics covered in lecture nine:

• [00:50] Good and bad binary search trees (BSTs).
• [02:00] Binary search tree sort tree algorithm.
• [03:45] Example of running BST-sort on array (3, 1, 8, 2, 6, 7, 5).
• [05:45] Running time analysis of BST-sort algorithm.
• [11:45] BST-sort relation to quicksort algorithm.
• [16:05] Randomized BST-sort.
• [19:00] Randomly built binary search trees.
• [24:58] Theorem: expected height of a rand BST tree is O(lg(n)).
• [26:45] Proof outline.
• [32:45] Definition of convex function.
• [46:55] Jensen’s inequality.
• [55:55] Expected random BST height analysis.

Lecture nine notes:

Lecture 9, page 1 of 2.
Lecture 9, page 2 of 2.

Lecture 10: Red-Black Trees

Lecture ten begins with a discussion of balanced search trees. Balanced search tree is search tree data structure maintain a dynamic set of n elements using tree of height log(n).

There are many balanced search tree data structures. For example: AVL trees (invented in 1964), 2-3 trees (invented in 1970), 2-3-4 trees, B-trees, red-black trees, skiplists, treaps.

This lecture focuses exclusively on red-black trees.

Red-black trees are binary search trees with extra color field for each node. They satisfy red-black properties:

• Every node is either red or black.
• The root and leaves are black.
• Every red node has a black parent.
• All simple paths from a node to x to a descendant leaf of x have same number of black nodes = black-height(x).

The lecture gives a proof sketch of the height of an RB-tree and discusses running time of queries (search, min, max, successor, predecessor operations) and then goes into details of update operations (insert, delete). Along the way rotations on a tree are defined, the right-rotate and left-rotate ops.

The other half of the lecture looks at Red-Black-Insert operation that inserts an element in the tree while maintaining the red-black properties.

Here is the video of lecture ten:

Topics covered in lecture ten:

• [00:35] Balanced search trees.
• [02:30] Examples of balanced search tree data structures.
• [05:16] Red-black trees.
• [06:11] Red-black properties.
• [11:26] Example of red-black tree.
• [17:30] Height of red-black tree.
• [18:50] Proof sketch of RBtree height.
• [21:30] Connection of red-black trees to 2-3-4 trees.
• [32:10] Running time of query operations.
• [35:37] How to do RB-tree updates (inserts, deletes)?
• [36:30] Tree rotations.
• [40:55] Idea of red-black tree insert operation.
• [44:30] Example of inserting an element in a tree.
• [54:30] RB-Insert algorithm.
• [01:03:35] The three cases in insert operation.

Lecture ten notes:

Lecture 10, page 1 of 2.
Lecture 10, page 2 of 2.

Have fun building trees! The next post will be about general methodology of augmenting data structures and it will discuss dynamic order statistics and interval trees!

PS. This course is taught from the CLRS book (also called “Introduction to Algorithms”):

The Linkage of UMBEL’s 20,000 Subject Concepts and Inferencing Brings New Capabilities

Thanks to Kingsley Idehen and OpenLink Software, DBpedia has been much enrichened with its mapping to UMBEL’s 20,000 class-based subject concepts. DBpedia is the structured data version of Wikipedia that I (among many) wrote about in depth in April of last year shortly after its release.

We have also recently gotten an updated estimate of the size of the semantic Web and a new release of the linking open data (LOD) cloud diagram.

A New Instance of the LOD Cloud Diagram

Since DBpedia’s release, it has become the central hub of linked open data as shown by this now-famous (and recently updated!) LOD diagram [1]:

[click for full size]

Each version of the diagram adds new bubbles (datasets) and new connections. The use of linked data, which is based on the RDF data model and uses Web protocols to name and access data, is proving to be a powerful framework for interconnecting disparate and heterogeneous information. As the diagram above shows, all types of information from a variety of public sources now make up the LOD cloud [2].

A Beginning Basis for Estimating the Size of the Semantic Web

The most recent analysis of this LOD cloud is by Michael Hasenblas and colleagues as presented at I-Semantics08 in September [3]. About 50 major datasets comprising roughly two billion triples and three million interlinks were contained in the cloud at the time of their analysis. They partitioned their analysis into two distinct types: 1) single-point-of-access datasets (akin to conventional databases), such as DBpedia or Geonames, and 2) distributed records characterized by RDF ontologies such as FOAF or SIOC. Their paper [3] should be reviewed for its own conclusions. In general, though, most links appear to be of low value (though a minority are quite useful).

Simple measures such as triples or links have little meaning in themselves. Moreover, and this is most telling, all of the LOD relationships in the diagram above and the general nature of linked data to date have based their connections on instance-level data. Often this takes the form that a specific person, place or thing in one dataset is related to that very same thing in another dataset using the owl:sameAs property; sometimes it is that one person knows another person; or, it may be in other examples that one entry has an associated photo. Entities are related to other entities and their attributes, but little is provided about the conceptual or structural relationships amongst those entities.

Instance-level mapping is highly useful to aggregate various attributes or facts about given entities or things. But, they only scratch the surface of the structure that can be made available through linked data and the conceptual relationships between and amongst all of those things. For those relationships to be drawn or inferred a different level of linkages needs to be made: what is the class or collection or schema view of the data.

The UMBEL Subject Concept ‘Backbone’

UMBEL, or similar conceptual frameworks, can provide this structural backbone.

UMBEL (Upper Mapping and Binding Exchange Layer; see http://www.umbel.org) is a lightweight reference ontology of about 20,000 subject concepts and their logical and semantic relationships. The UMBEL ontology is a direct derivation of the proven Cyc knowledge base from Cycorp, Inc. (see http://www.cyc.com).

UMBEL’s subject concepts provide mapping points for the many (indeed, millions of) named entities that are their notable instances. Examples might include the names of specific physicists, cities in a country, or a listing of financial stock exchanges. UMBEL mappings enable us to link a given named entity to the various subject classes of which it is a member.

And, because of relationships amongst subject concepts in the backbone, we can also relate that entity to other related entities and concepts. The UMBEL backbone traces the major pathways through the content graph of the Web.

The UMBEL backbone provides structure and relationships at large or small scale. For example, in its full extent, the structure of UMBEL’s complete structure resembles:

But, we can dive into that structure with respect to automobiles or related concepts . . .

. . . all the way down to seeing the relationships to Saab cars:

It is this ability to provide context through structure and relations that can help organize and navigate large datasets of instances such as DBpedia. Until the application of UMBEL — or any subject or class structure like it — most of the true value within DBpedia has remained hidden.

But no longer.

Some Example Queries

UMBEL already had mapped most DBpedia instances to its own internal classes. By a simple mapping of files and then inferencing against the UMBEL classes, this structure has now been brought to DBpedia itself. Any SPARQL queries applied against DBpedia can now take advantage of these relationships.

Below are some sample queries Kingsley used to announce these UMBEL capabilities to the LOD mailing list [4]. You can test these queries yourself or try alternative ones by using a standard SPARQL query.

For example, go to one of DBpedia’s query endpoints such as http://dbpedia.org/sparql and cut-and-paste one of these highlighted code snippets into the ‘Query text’ box:

Example Query 1

Example Query 2

Example Query 3

Example Query 4

Example Query 5

Example Query 6

Creating Your Own Mapping

By going to UMBEL’s technical documentation page at http://umbel.org/documentation.html, you can download the files to create your own mappings (assuming you have a local instance of DBpedia).

The example below also assumes you are using the OpenLink Virtuoso server as your triple store. If you are using a different system, you will need to adjust your commands accordingly.

1. Load linkages (owl:sameAs) between UMBEL named entities and DBpedia resources

File: umbel_dbpedia_linkage.n3

2. Load inferred DBpedia types (rdf:types) based on UMBEL named entities

File: umbel_dbpedia_types.n3

3. Load Virtuoso-specific file containing the rules for inferencing

File: umbel_virtuoso_inference_rules.n3

4. Load UMBEL External Ontology Mapping into a Named Graph (owl:equivalentClasses)

File: umbel_external_ontologies_linkage.n3

5. Create UMBEL Inference Rules

Conclusion

A new era of interacting with DBpedia is at hand. Within a period of just more than a year, the infrastructure and data are now available to show the advantages of the semantic Web based on a linked Web of data.  DBpedia has been a major reason for showing these benefits; it is now positioned to continue to do so.

http://searchenginerapbattle.com/