Benchmark Framework for Mobile Robots Navigation Algorithms
Marco de referencia para algoritmos do navegación de robots móviles
Nelson David Muñoz-Ceballos*
Jaime Alejandro Valencia-Velásquez**
* Politécnico Colombiano Jaime Isaza Cadavid (Medellín-Antioquia Colombia). ndmunoz@elpoli.edu.co
** Universidad de Antioquia (Medellín-Antioquia, Colombia).
Fecha de Recepción: 02 de Enero de 2014 Fecha de Aprobación: 03 de Febrero de 2014
Abstract
Despite the wide variety of studies and research on mobile robot systems, performance metrics are not often examined. This makes difficult to establish an objective comparison of achievements. In this paper, the navigation of an autonomous mobile robot is evaluated. Several metrics are described. These metrics, collectively, provide an indication of navigation quality, useful for comparing and analyzing navigation algorithms of mobile robots. This method is suggested as an educational tool, which allows the student to optimize the algorithms quality, relating to important aspects of science, technology and engineering teaching, as energy consumption, optimization and design.
Keywords: Educational robotics, Mobile robots, Navigation algorithms, Performance metrics.
Resumen
A pesar de la amplia variedad de estudios e investigaciones sobre los sistemas de robots móviles, a menudo las métricas de rendimiento no son examinadas; esto hace difícil establecer una comparación objetiva de los logros. En este trabajo se evalúa la navegación de un robot móvil autónomo; se describen varias métricas, que en conjunto proporcionan un indicador de la calidad de la navegación, útil para comparar y analizar los algoritmos de navegación de robots móviles. Este método se propone como una herramienta educativa que permite al estudiante optimizar la calidad de los algoritmos, relacionando aspectos importantes de la ciencia, la tecnología y la enseñanza de la ingeniería, como el consumo de energía, la optimización y el diseño.
Palabras clave: Robótica Educativa, Robots Móviles, Algoritmos de Navegación, Métricas de Rendimiento.
I. Introduction
An autonomous mobile robot has to combine mission execution with fast reaction to unexpected situations. To overcome this problem, various types of control architectures for mobile robots have been designed, with the aim to improve performance of the navigation system of a mobile robot in the execution of the mission. Despite the wide variety of studies and research on robot navigation systems, quality metrics are not often examined, turning the objective comparison of performance into a challenge [1].
This paper presents some suggestions for the assessment of navigation algorithms in educational robotics. This method is suggested as an educational tool to help the student with the optimization of the algorithms quality, association of important aspects of science, technology and engineering teaching as energy consumption, optimization and design.
II. Performance Metrics for Robot Navigation
The navigation system gives a robot the capability to move between given locations. There are several metrics that can be used to evaluate the performance of a navigation system, but none of them are suitable to indicate the quality of the whole system. Therefore it is necessary to use a combination of different indexes quantifying different aspects of the system. Having a good range of performance measurements is useful for: Optimizing algorithm parameters, testing navigation performance within a variety of work environments, making a quantitative comparison between algorithms, supporting algorithm development and helping with decisions about the adjustments required for a variety of aspects involved in system performance [3, 5].
Typical performance criteria in navigation and obstacle avoidance are [8, 11, 12]:
Mission success: number of successful missions.
Path length: distance traveled to accomplish the task.
Time taken to accomplish the task.
Collisions: number of collisions per mission, per distance and per time.
Obstacle clearance: minimum and mean distance to the obstacles.
Robustness in narrow spaces: number of narrow passages successfully traversed.
Smoothness of the trajectory: relative to control effort.
Navigation performance metrics can be classified in the following importance order:
Metrics that consider the security in the trajectory or proximity to obstacles.
Metrics that consider the trajectory towards the goal.
Metrics that evaluate the smoothness of the trajectory.
A. Security metrics
These metrics describe the robot security while it travels through a trajectory, taking into account the distance between the vehicle and the obstacles in its path [2].
Security Metric-1 (SM1): Mean distance between the vehicle and the obstacles through the entire mission measured by all the sensors; the maximum value will be produced in an obstacle free environment. If the deviation of the index from its maximum value is low, it means that the chosen route had fewer obstacles.
Security Metric-2 (SM2): Mean minimum distance to obstacles. This is taken from the average of the lowest value of the n sensors. This index gives an idea of the risk taken through the entire mission, in terms of the proximity to an obstacle. In an obstacles free environment SM1 = SM2 is satisfied.
Minimum Distance (Min): Minimum distance between any sensor and any obstacle through the entire trajectory. This index measures the maximum risk taken throughout the entire mission.
B. Dimension metrics
The trajectory towards the goal is considered in its time and space dimensions. In general, it is assumed that an optimal trajectory towards the goal is, whenever possible, a line with minimum length and zero curvature between the initial point (xi,yi) and the final point (xn,yn), covered in the minimum time.
Length of the Covered Trajectory (PL) is the length of the entire covered path by the vehicle from the initial point to the goal. For a trajectory in the x-y plane, composed of n points, and assuming the initial point as (x1 f(x1)) and the goal as (xn, f(xn)), PL can be calculated as:
Where (xi, f(xi)), i = 1, 2, . . . , n are the n points of the trajectory in Cartesian coordinates [6].
The length of a trajectory given by y = f(x), in the x-y plane between the points (a, f(a)) and (b, f(b)), can also be calculated as [10].
Mean distance to the goal (Mgd): This metric can be applied to robots capable of following reference trajectories. An important aspect when determining the quality of the robot navigation system is the ability to follow a trajectory that aims to reach a goal. To evaluate the quality of the trajectory execution, the mean distance between the vehicle and goal is analyzed. The difference is more significant if the covered distance is shorter [9]. The mean distance to the goal is defined by the square of the proximity to the goal (ln), integrated across the length of the trajectory and normalized by the total number of points n:
Control Periods (LeM): It is the amount of control periods. This metric is related to the number of decisions taken by the planner to reach the goal, if the robot moves with lineal and constant speed (v). This gives an idea of the time needed to complete the mission [2].
C. Smoothness metrics
The smoothness of a trajectory shows the consistency between the decision-action relationship taken by the navigation system, as well as the ability to anticipate and to respond to events with rapidly enough [9]. The smoothness of the generated trajectory is a measure of the energy and time requirements for the movement; a smooth trajectory translates into energy and time savings [4]. A smooth trajectory is also beneficial to the mechanical structure of the vehicle.
Bending Energy (BE): This is a function of the curvature (k) used to evaluate the smoothness of the robot's movement. For curves in the x-y plane, the curvature at any point (xi, f(xi)) across a trajectory is given by:
The bending energy (BE) can be understood as the energy needed to bend a rod to the desired shape [1]. BE can be calculated as the sum of the squares of the curvature at each point of the line k(xi,f (xi)), along the length of the line L. So, the bending energy of the trajectory of a robot is given by:
Where k(xi, f(xi)) is the curvature at each point of the trajectory of the robot and n is the number of points in the trajectory.
The value of BE is an average and does not show with enough clarity that some trajectories are longer than others. Therefore, TBE can be used instead. This metric takes into account the smoothness and length of the trajectory simultaneously.
TBE is defined by 
and numerically, 
As the trajectory gets straighter, the values BE and TBE will be lower, which is desirable since the energy requirement are increased according to the increase in the curvature of the trajectory.
Smoothness of Curvature (Smoo) is defined by the square of the change in the curvature k of the trajectory of a vehicle with respect to the time, integrating along the length of the trajectory and normalized by the total time (t) [9].
III. Robot and navigation algorithm
Navigation algorithms provide basic capabilities for the mobile robot, such as the ability to evade obstacles and to generate a trajectory towards a goal.
A. Navigation Algorithm
This is a reactive algorithm based on the potential field method, which produces two different behaviors: first, goal attraction, and second, obstacles repulsion (keeping away from objects). The planning of the movement consists in the proper combination of both behaviors in such a way that the robot reaches the goal without collisions, figure 1. This combination is achieved using a vector sum [7].
Where
(x,y,Φ): current position and orientation
(x,y)f : final goal position
Si: sensors information (length measurement sensors)
(V,δ): velocity and orientation angle commands
(v,w): linear and angular velocity
B. Mobile robotics platform
The robot was simulated according to the characteristics of the common robotics platforms used for education or research. This has a cylindrical structure, differential drive system and distance measuring sensors distributed equally around the robot's circumference (the robot was simulated with 16 sensors, each one with a distance range equal to 50cm and detection cone angle equal to 10 degree), figure 2.
C. Simulations
A 6 m x 4 m frame, structured environment with static obstacles was created for the execution of a navigation mission between two points (towards a goal); figure 2. The paths generated by the algorithms in the scenario are shown in figure 3 and 4. Table 1 summarizes the results obtained from the simulation using both control algorithms according to the quality metrics described above.
D. Metrics selection
Taking into account that the objective is to execute a navigation mission from a starting point to a final point (navigation mission towards a goal), an order of importance can be established in order to evaluate the navigation characteristics, as follows:
The mean distance between the vehicle and the obstacles during the trajectory.
The distance covered by the vehicle between the starting point and the goal.
The time needed to complete the mission.
The smoothness of the trajectory.
The first point considers the security of the trajectory and measures the risk taken by the robot in its movement towards the goal. The second and third points measure aspects related to the planning of the trajectory, and the fourth point considers the quality of the trajectory according to the energy and time required for the movement.
These characteristics can be analyzed using the following set of performance metrics:
SM1, SM2 and Min are proposed for evaluating security.
PL and LeM are proposed for evaluating the trajectory.
TBE is proposed for evaluating the smoothness of the trajectory.
IV. Tests and Results
The code for navigation algorithm is shown in figure 5. This algorithm has two parameters (katr and krep). In all simulations, katr=1, but krep takes two values.
First simulation, krep=0.1,
Second simulation, krep=0.01
In the first simulation, the algorithm has krep=0.1, the repulsive force is bigger, and the navigation has an oscillatory pattern near obstacles. Consequently it takes more time to complete the mission.
In the second simulation, the algorithm has krep=0.01, the repulsive force is less important, and the navigation has a soft pattern near obstacles, uses less control periods. Consequently, it takes less time to complete the mission, and it covers a safer and shorter path.
The figure 3 shows that krep=0.1 produces a great oscillation for each control period. The figure 4 shows that krep=0.01 covers a smoother path, there is a smaller change in the orientation during each control period. Consequently, it implies energy saving and less structural stress on the robot.
From table 1, it can be deduced that the difference between both simulations in trajectory and time is 20.5% and 20% respectively. The robot programmed with krep=0.01 passed minimum 12 cm from any obstacle, which is acceptable for a 25 cm diameter robot. It also showed approximately 89% less bending energy for krep=0.01 than for krep=0.1. For these reasons, krep=0.01 can be considered the best choice.
V. Conclusions
This suggestion for the assessment of navigation algorithms provides a tool for educational robotics. A very simple application example was presented. The obtained results demonstrate the need to establish a procedure to be used when analyzing and comparing control algorithms using several performance metrics. This is an open topic of research. It has become necessary to establish proper approaches and benchmarking procedures, for example, using a benchmarking standard framework for navigation algorithm assays and performance assessment.
This metrics can be applied in simulated environments, but the performance metrics evaluation is more important in real environments. Many of the challenges in robot navigation come from the challenges of real environments, such as uncertainty in the sensors and the errors in odometry, which are generally not considered in simulation.
Acknowledgements
This work was a partial result of the 2061080241 project, from the Colombian Polytechnic Jaime Isaza Cadavid.
References
[1] S. Wong, L. Middleton and B. MacDonald, "Performance metrics for robot coverage task", Proceedings Australasian Conference on Robotics and Automation ACRA, Auckland, New Zealand, 2002, pp. 7-12.
[2] J. Álvarez, Planificación del movimiento de vehículos autónomos basada en sensores. Tesis doctoral, Universidad de Oviedo, Oviedo, España, 1998, pp. 178.
[3] G. Cielniak, A. Treptow and T. Duckett, "Quantitative Performance Evaluation of a People Tracking System on a Mobile Robot", Proceedings of the European Conference on Mobile Robots (ECMR), Ancona, Italy, 2005.
[4] S. Dongqing, "Aerial robot navigation in cluttered urban environments", Ph.D. Thesis, The Florida State University, Florida, USA, 2006, pp. 87.
[5] J. Evans and E. Messina, "Performance Metrics for Intelligent Systems", Proceeding of the Performance Metrics for intelligent Systems Workshop, Gaithersburg, MD, August 14-16 of 2000.
[6] Y. Guo and J. Wang, "A new performance based motion planner for nonholonomic mobile robots", Proceedings of the 3rd performance metrics for the Intelligent Systems Workshop (PerMIS'03) NIST, Gaithersburg, MD, September 2003.
[7] J.C Latombe, "Robot Motion Planning", Kluwer Academic Publishers, 4th Edition, Boston, 1996.
[8] Munoz, N.; Valencia, J. & Londono, N. (2007). Evaluation of Navigation of an Autonomous Mobile Robot, Proceedings of the Performance Metrics for Intelligent Systems (PerMIS) Workshop, Washington, DC, EE.UU. http://www.nist.gov/el/isd/ks/upload/PerMIS_2007_Full_Proceedings.pdf.
[9] J. Rosenblatt, "DAMN: Distributed Algorithm for Mobile Navigation. Ph.D. Thesis", Carnegie Mellon University Robotics Institute, Pittsburg, PA, 1997.
[10] M. Selekwa, E. Collins and J. Combey, "Multivalued versus univalued Reactive Fuzzy Behavior Systems for Navigation Control of Autonomous Ground Vehicles", Proceedings from the 17th annual Florida Conference on the Recent Advances in Robotics FCRAR2004, May 20, 2004.
[11] J. Minguez, (2008). Robot Obstacle Avoidance Papers Using Experiments, In: Good experimental methodology guidelines, Bonsignorio, F.; Hallam J., & del Pobil, A. P. (Ed), Special Interest Group on Good Experimental Methodology in Robotics European Robotics Research Network (EURON), Tech. Rep., 2008. Available from: http://www.heronrobots.com/EuronGEMSig/Downloads/GemSigGuidelinesBeta.pdf.
[12] N. D. Munoz-Ceballos, J. A. Valencia and N. Londono-Ospina (2010). Quantitative Performance Metrics for Mobile Robots Navigation, Mobile Robots Navigation, Alejandra Barrera (Ed.), ISBN: 978953-307-076-6, InTech, Available from: http://www.intechopen.com/books/mobile-robots-navigation/quantitative-performance-metrics-for-mobile-robots-navigation.