Evaluation of User Interfaces Description Languages to Augmented Reality

Augmented reality is a rapidly advancing technology as it is widely used in several applications and domains, from the aerospace industry to medical-surgical applications, particularly in learning and training. A fundamental part of the educational use is the graphical user interface as a vehicle to deliver information to students. Within the augmented experiences development process, it is necessary to identify the most appropriate way to describe the user interface. This article 1 Universidad Industrial de Santander (Bucaramanga-Santander, Colombia). camilo.cruz1@correo.uis.edu.co. ORCID: 0000-0002-5267-8572 2 M. Sc. Universidad Industrial de Santander (Bucaramanga-Santander, Colombia). luis.bautista@correo.uis.edu.co. ORCID: 0000-0001-5852-311X 3 Ph. D. Universidad Industrial de Santander (Bucaramanga-Santander, Colombia). gpedraza@uis.edu.co. ORCID: 0000-0003-4706-1540 Evaluation of User Interfaces Description Languages to Augmented Reality Revista Facultad de Ingeniería (Rev. Fac. Ing.) Vol. 30 (57), e13304. July-September 2021. Tunja-Boyacá, Colombia. L-ISSN: 0121-1129, e-ISSN: 2357-5328. DOI: https://doi.org/10.19053/01211129.v30.n57.2021.13304 evaluates languages describing augmented reality user interfaces. In this paper, several languages are analyzed, and new criteria are proposed to extend the evaluation framework. Finally, the results show the current limitations and the slowdown of development and adaptation of new languages for augmented reality platforms.


I. INTRODUCTION
Augmented Reality (AR) is a technology that allows users to observe virtual elements such as 3D objects in a real-world context. The virtual and real elements are part of the user interface (UI) and can be manipulated in real time. The development of augmented reality applications has grown over time, becoming highly impactful in domains such as engineering, medicine, and education [1][2][3]. Part of these applications are designed from scratch by using low-level toolkits, but when dealing with the creation of complex augmented reality systems, this approach becomes error-prone and more time-consuming to develop [4].
The user interface is one of the main components of an augmented reality system whose design occasionally becomes a challenge for the developer [5]. Its implementation is a time-consuming task when performed at a low programming level so tools that facilitate and speed up its development become necessary [6].
Oliveira and Araujo [5] mention two main approaches to developing user interfaces for augmented reality: frameworks and authoring tools. Frameworks try to cover the whole process of implementing AR applications although they focus on application logic instead of user interface.
Authoring tools offer a customized set of interface and interaction components to develop applications without needing programming skills. Some tools use high-level description languages to describe user interfaces in text files. Generally, the syntax of these languages is XML-based [7]. This paper aims to identify languages used to describe user interfaces in AR development. To achieve this, a systematic literature review was carried out. This paper presents the review results of the languages used to describe augmented reality user interface. Also, new criteria were proposed to extend the framework presented by Córdova Gonzales [8], and a comparative evaluation of the languages is performed.

II. METHODOLOGY
The research aim is to find languages used to describe user interfaces in augmented reality applications, and, through a comparison, select the language that is best suited for the creation of three-dimensional interfaces for augmented reality. To find relevant papers, the method proposed by Kitchenham and Charters [9] was applied in the review. The authors suggest the review implementation in three phases: planning, conducting, and reporting.

A. Planning the Review
Firstly, a search in several databases was carried out by using associated keywords with search objectives. Once a set of articles is obtained, a filter based on the criteria was applied to identify the papers that can provide direct evidence to the review question: What languages have been developed to describe the user interfaces in augmented reality applications? The following databases were used in the search for papers: Springer, IEEE, ACM, and EBSCO Discovery Service. A set of search strings was used, as seen in table 1. The items used were title, abstract, keywords.
Results by each database can be seen in Table 1.

Search Strings
Springer IEEE EBSCO ACM "augmented reality" AND "user interface" AND "description language" 121 8 13 74 "augmented reality" AND "user interface" AND "markup language" 291 7 17 113 "augmented reality" AND "authoring tools" AND "languages" 327 12 20 69 "augmented reality" AND "xml languages"   The following selection criteria were applied to a collection of 88 papers: papers that involve the use of languages in the user interfaces development; papers that present methods or tools that contribute to the development of augmented reality applications; papers not related to augmented reality. Following the criteria, 56 papers are excluded, and 8 papers are included due to snowballing, resulting in 40 papers total.

C. Reporting the Review
After analyzing the papers obtained in the review, 10 languages were identified, most of them to specify or store AR content. To have an overview, a synthesis of each language is presented, focused on features, limitations, and application examples. Table 2. Languages identified in the review.

1) Augmented Reality Markup Language (ARML). ARML
Anchors describe the spatial relationship between the physical and virtual object.
The current specification focuses on visual objects, and its implementation was limited to mobile devices [25]. Córdova Gonzales [8] presents the development of an augmented reality browser for vision-based applications using markers where each scene is described under the ARML 2.0 standard. [12] is an XML-based language for creating augmented reality presentations within the Studiertube framework [26]. By using APRIL, authors can specify multimedia objects, behaviors, and possible user interaction. UML state diagrams are used to specify the flow of an augmented reality presentation; they are stored as XMI files and then translated into APRIL format. From this description, Studiertube configuration files are generated that allow running the augmented reality presentation on that platform. Ledermann and Schmalstieg [27] present several application examples, such as an interactive storybook for children, a virtual display system for archaeological artifacts, and an outdoor tourist guide application. APRIL is not based on task-centric domains [28] and some design aspects are also not considered [4,15] -such as the 3D content structure and the way real and virtual objects are organized within the hierarchical structure of the user interface. Although Ledermann and Schmalstieg state that it should be possible to use other platforms to reproduce presentations created with APRIL [27], Vitzthum and Hussmann mention that the presentations are not intended to be platform-independent as they are designed to run on the Studiertube platform [28]. [13] was designed to enable the modeling of Contextual Augmented Reality Environments (CARE) [29]  that support both directional and natural augmentation. CARL allows the specification of content objects, trackables, and interfaces by using XML syntax.

3) Contextual Augmented Reality Language (CARL). CARL
Unlike other languages such as ARML and APRIL that help simplify the development of augmented reality environments, CARL was intended for dynamic AR contextual information visualization systems. A case study [30] presents a CARL browser application for a bookstore. Ruminski [31] introduces the concept of spatial sound modeling implemented in the CARL browser allowing augmented reality presentations enriched with 3D sound. [14] is an extension of MR-ISL (Mixed Reality Interaction Scenario Language) [32] which is a high-level language for defining interaction scenarios in augmented reality environments. AREML was developed following the AREM (Augmented Reality Environment Modeling) approach for creating AR environments inspired by the object-oriented paradigm. Its development is based on ARSM (Augmented Reality Scene Model) which formalizes the concepts of AR-Class, AR-Object, and its components through the AREML language.

5) SSIML extension for Augmented Reality (SSIML/AR). Scene Structure and
Integration Modeling Language [33] is a language aimed at the integration of 3D scenes in complex applications. SSIML/AR [15] is an extension of SSIML adapted to structured specification and development of AR applications, focusing on AR user interfaces. An example application presented in the article shows a simplified scenario of an AR system that guides the user through installing a graphics card on a desktop computer [15].  [17] is an XMLbased specification language that enables the description of both generic and specific 3DICs (3D Interaction Components). It encapsulates a conceptual framework for 3DICs encompassing 3D widgets, 3D gadgets, and 3D interaction techniques. ICDL implements elements from other XML languages such as InTml [36] for status graphs, bibliographic references, and documentation. A ring menu defined in XML is shown as an example of the implementation of a specific 3DIC.

8) Augmented Reality-based User Interaction Markup Language (ARUIML).
ARUIML [18] is an XML-based descriptive data format that facilitates the development of the presentation layer of AR applications with a user-centric approach. It provides communication mechanisms to support meaningful and engaging interactions with end users. ARUMIL is based on an interaction model composed of users, client systems, server systems, spatial layers, and non-spatial layers. For the adaptability of user interfaces to different situations, ARUIML proposes a structure of context-sensitive layers following a model-based approach.

9) Mixed Reality Interface Markup Language (MRIML). MRIML [19] is a language
created to define cross-platform user interfaces, object behavior for interaction prototypes in virtual reality or augmented reality, and object-oriented scene diagrams for the development of complex application scenarios. MRIML provides a vocabulary for defining UI elements in a document that is sent to the rendering unit which create UIs for individual platforms. As an example, the article presents two rendering results (2D and 3D) of a user interface from the same sample scenario.

D. Related Work
Visser [7] shows a study of XML languages used in augmented reality to store content, which sought the possibility of standardizing the format of AR content to allow interoperability between different AR browsers. Ahn [37] presents a content structure for AR application development in HTML5 for mobile devices, which seeks a separation between AR content and application logic. The article also shows a comparison between seven AR markup languages. The Open Geospatial Consortium (OGC) [35] introduced the concepts of ARML 2.0 and a comparison with other AR browser content formats. Córdova Gonzales [8] analyses the differences between ARML 2.0 and other augmented reality languages.

III. COMPARATIVE ASSESSMENT OF AUGMENTED REALITY LANGUAGES
Córdova Gonzales [8] proposes seven evaluation criteria to compare augmented reality languages, based on the studies conducted by Visser [7] and Ahn [37] who propose eight and five criteria, respectively. The criteria proposed by Visser focus on the requirements that an augmented reality language must meet to be a candidate for standardization, while in Ahn [37], they focus on the content structure of each language. The authors decided to use the seven criteria of the first work since they derive from the two previous works. The criterion "TESTAR -Test Definition" was discarded as the documentation is insufficient to evaluate it, which resulted in the addition of the criterion "DOC -Amount of Documentation". The criterion "FOCUS -Language Focus" was added since some of the languages found have a specification more focused on content description than on user interfaces.
Finally, the criterion "YEAR -Year of publication" allows to organize them by date. : Full documentation of language specification, implementation details; : Incomplete documentation, e.g., the vocabulary of the language is not fully defined; : Insufficient/no documentation available.

YEAR -Year of release:
The year in which the version of the language was published.

IV. RESULTS
ARTYPE: 50% of the languages support geolocation-based augmented reality.
Likewise, 60% support AR based on markers, and 80% AR based on vision.  the other hand, Córdova Gonzales [8] states that "very few browsers and augmented reality platforms have made efforts to implement this standard (ARML), nor is there any news about future implementations". Since its approval, ARML 2.0 did not achieve the expected adoption or commercial impact that it was promising. However, Liao [39] states that this reaction is frequently for standards, especially those created by consortia that make biased assumptions about future user practice and markets, which in some cases may be outweighed by the rate of technological change at the time the standard is approved.

VI. CONCLUSIONS
Considering the current development of augmented reality and the increase in the development of products based on this technology, knowing the languages that allow describing the user interfaces for this technology is a necessity. Due to this, a systematic review of scientific literature was carried out, where the existence of ten (10) languages used to describe the user interface for augmented reality applications was established. Likewise, a detailed analysis of each language was performed, focusing on the main characteristics, limitations, and some examples of their application. A comparative evaluation was conducted to establish the robustness of the language in terms of current development requirements. For this purpose, a comparative evaluation framework was adapted by adding the necessary criteria for the adequate and updated use of the language. The main findings of this study show that the lack of documentation is one of the major limitations since it prevents the indepth evaluation of some languages. In addition, it was identified that these languages slowed down their development in 2015, and the adoption of the ARML standard has not been as expected. Finally, ARML 2.0 and APRIL are the languages that best met the evaluation criteria.

ACKNOWLEDGMENTS
This paper is part of the results of the project titled "Model-based method for the design of graphical user interfaces for reducing extraneous cognitive load due to split spatial attention", part of the research project titled "Integración de técnicas ópticas