Evaluation of the Music Ontology framework

Evaluation of the Music Ontology framework

Yves Raimond, BBC R&D
Mark Sandler, Queen Mary, University of London
Heraklion, Crete, Greece, 29 May 2012

How the BBC makes websites

Start by identifying your domain objects and the relationships between them

Ontologies are increasingly used to power bbc.co.uk

The Programme Ontology

The Music Ontology

The Music Ontology

Problem

If these ontologies are such a fundamental building block for our websites:

What is "good"?

How well would a system backed by the evaluated ontology handle real-world user information needs?

e.g. "Who composed 'Peter Grimes'?"

Evaluation methodology

Both task-based [Porzel2004] and data-driven [Brewster2004]

Ontology fit (1)

Who composed 'Peter Grimes'?

Ontology fit (2)

Ontology fit (3)

Who composed "Peter Grimes"?

Ontology fit: 1

Ontology fit (4)

What was the name of the artist who was on the cover of that magazine?

Ontology fit (5)

Ontology fit (6)

What was [the name of the artist] who was [on the cover of that magazine]?

Ontology fit: 0.5

Use-case: evaluation of the Music Ontology framework

Casual users

Evaluation against previous studies of user needs

Two previous studies:

Evaluation on a dataset of verbalised user needs

Manual evaluation

On a random subset of the dataset (0.5%):

Evaluation at the lexical level (1)

Evaluation at the lexical level (2)

Evaluation at the lexical level (3)

Evaluation at the conceptual level

Example inferred topics

Resulting ontology fit: 0.723

Users of music libraries

Manual evaluation

Discussion

Main features not expressible within the Music Ontology

Towards continuous evaluation?

Future work

Thank you!