SEMANTICAL ENRICHMENT OF WEB USER INTERFACES IN THE CROWD

Structured data on the Web have become very important in recent years and offer machine-readable semantics of the data contained. Schema.org is the most recognized vocabulary for structuring data on websites and serves beyond that also to organize and query Google's knowledge graph. However, Schema.org is mainly used today to semantically describe real-world entities and their relationships to one another. To describe the functionality and handling of websites has not been the key application area of Schema.org yet, though it also includes an appropriate vocabulary. In addition, most Schema.org tools are limited in terms of the Schema.org classes and properties they offer and do not support the semantical enrichment of web user interfaces. In this article we motivate the potential of the sematic enrichment of web user interfaces regarding their functionality and handling and present possible application areas to consume these structured data. We analyze the requirements on structured user interface data and investigate the suitability of Schema.org as a vocabulary to fulfil them. Since there are hundreds of classes and properties in Schema.org, we present a conceptual model of Schema.org classes and properties that are appropriate to enrich web user interfaces. We investigate to what extent interactive elements on HTML websites or applications can be automatically mapped to our conceptual model elements. As a result, we present a method to semi-automatically produce structured user interface data and illustrate our approach with a continuous use case. As a proof of concept, we introduce Schemator, a comfortable structured user interface data tool supporting our method, which can be used to crowdsource knowledge about the functionality and handling of websites and applications.