AUTHOR’S CONTRIBUTION: Rodriguez-Bastidas carried out the problem statement, documentary review, software development, usability tests, analysis, and discussion of the obtained results. He also wrote the manuscript. Vargas-Rosero delimited the problem, advised throughout the development of the investigation, reviewed the techniques for processing the results, and reviewed the final document.
Competing interests: The authors have declared that no competing interests exist.
Medical images are essential for diagnosis, planning of surgery and evolution of pathology. The advances in technology have developed new techniques to obtain digital images with more details, in return this has also led to disadvantages, such as: the analysis of large volumes of information, long time to determine an affected region and difficulty in defining the malignant tissue for its later extirpation, among the most relevant. This article presents an image segmentation strategy and the optimization of a method for generating three-dimensional models. A prototype was implemented in which it was possible to evaluate the segmentation algorithms and 3D reconstruction technique, allowing to visualize the tumor model from different points of view through virtual reality. In this investigation, we evaluate the computational cost and user experience, the parameters selected in terms of computational cost are the time and consumption of RAM, we used 140 MRI images each with dimensions 260x320 pixel, and as a result, we obtained an approximate time of 37.16s and consumption in RAM of 1.3GB. Another experiment carried out is the segmentation and reconstruction of a tumor, this model is formed by a three-dimensional mesh made up of 151 vertices and 318 faces. Finally, we evaluate the application, with a usability test applied to a sample of 20 people with different areas of knowledge. The results show that the graphics presented by the software are pleasant, they also show that the application is intuitive and easy to use. Additionally, it is mentioned that it helps improve the understanding of medical images.
Las imágenes médicas son imprescindibles para la realización del diagnóstico, planificación de cirugía y evolución de la patología. El avance de la tecnología ha desarrollado nuevas técnicas para obtener imágenes digitales con más detalles, esto a su vez ha llevado a tener desventajas, entre ellas: el análisis de grandes volúmenes de información, tiempo prolongado para determinar una región afectada y dificultad para definir el tejido maligno para su posterior extirpación, entre las más relevantes. Este artículo presenta una estrategia de segmentación de imágenes y la optimización de un método de generación de modelos tridimensionales. Se implementó un prototipo en el que se logró evaluar los algoritmos de segmentación y técnica de reconstrucción 3D permitiendo visualizar el modelo del tumor desde diferentes puntos de vista mediante realidad virtual. En esta investigación, se evalúa el costo computacional y la experiencia del usuario, los parámetros seleccionados en términos de costo computacional son el tiempo y el consumo de RAM, se utilizaron 140 imágenes MRI cada una de ellas con dimensiones de 260x320 píxeles, y como resultado, se obtuvo un tiempo aproximado de 37.16s y el consumo de memoria RAM es de 1.3GB. Otro experimento llevado a cabo es la segmentación y reconstrucción de un tumor, este modelo está formado por una malla tridimensional que contiene 151 vértices y 318 caras. Finalmente, se evalúa la aplicación con una prueba de usabilidad aplicada a una muestra de 20 personas con diferentes áreas de conocimiento, los resultados muestran que los gráficos presentados por el software son agradables, también se evidencia que el software es intuitivo y fácil de usar. También mencionan que ayuda a mejorar la compresión de imágenes médicas.
As imagens médicas são imprescindíveis para a realização do diagnóstico, planificação de cirurgia e evolução da patologia. O avanço da tecnologia tem desenvolvido novas técnicas para obter imagens digitais com mais detalhes, isto, por sua vez, tem algumas desvantagens, entre elas: a análise de grandes volumes de informação, tempo prolongado para determinar uma região afetada e dificuldade para definir o tecido maligno para sua posterior extirpação, entre as mais relevantes. Este artigo apresenta uma estratégia de segmentação de imagens e a optimização de um método de geração de modelos tridimensionais. Implementou-se um protótipo no qual logrou-se avaliar os algoritmos de segmentação e técnica de reconstrução 3D, permitindo visualizar o modelo do tumor desde diferentes pontos de vista mediante realidade virtual. Nesta pesquisa, avalia-se o custo computacional e a experiência do usuário; os parâmetros selecionados em termos de custo computacional são o tempo e o consumo de RAM. Utilizaram-se 140 imagens MRI cada uma delas com dimensões de 260x320 pixels, e como resultado, obteve-se um tempo aproximado de 37.16s e o consumo de memória RAM é de 1.3GB. Outro experimento realizado é a segmentação e reconstrução de um tumor; este modelo está formado por uma rede tridimensional que contém 151 vértices e 318 caras. Finalmente, avalia-se a aplicação com uma prova de usabilidade aplicada a uma amostra de 20 pessoas com diferentes áreas de conhecimento, os resultados mostram que os gráficos apresentados pelo software são agradáveis, também se evidencia que o software é intuitivo e fácil de usar. Também mencionam que ajuda a melhorar a compressão de imagens médicas.
The fundamental objectives of medicine are the prevention and promotion of health, pain relieving, assistance to incurable diseases and prevention of premature death [
Doctors rely on various tools that provide them with information about the status of the human body, in order to diagnose and determine the evolution of the pathology and decide on the treatment to follow [
Different techniques have emerged through technological developments to observe the interior of the body, among them is the computed tomography (CT), where the image is constructed from multiple values of the X-ray beam, which crosses the study area [
Medical images are presented in 2D, 3D, and 4D. Three dimensions are associated with space and fourth is the time, in two-dimensional images the information represents the width and the height of an organ, by adding one more dimension, it represents depth and it is known as three-dimensional image [
The delimitation (segmentation) of the affected area must be done by a specialist doctor, and must take into consideration the additional knowledge of anatomy, pathology, etc. Bokde et al. [
By the incorporation of computation, the analysis of the region of interest in medical images is significantly improved, there are several investigations to perform segmentation, but there are still drawbacks, authors such as Weese and Lorenz [
The document is structured as follows, the first section explains the materials and methods presenting preliminary concepts of medical images, describing the algorithms of segmentation and generation of 3D models, also, it shows the stages of software development. The next section shows the result after a computing cost assessment, three-dimensional models, and an end-user experience (usability). The last section presents the conclusions and future works.
Currently, technological development has allowed obtaining real-time information of the current state of the organs, facilitating the portability of reports between diverse entities of health. Medical images have also evolved since their invention, presenting improvements in techniques and procedures, but in return, they also increase the amount of data to be processed and analyzed.
As a solution to the handling of information, the DICOM (Digital Imaging and Communications in Medicine) standard appears, which is the most widespread standard. It was created by the ACR (American College of Radiology) and by the NEMA (National Electrical Manufacturers Association). The DICOM standard is open-source designed to organize images for visualization, storage, printing and transmissions [
Among the information collection techniques of the human body, there is Computed tomography (CT) and Magnetic Resonance (MRI). Hounsfield, in 1971 develops TC, which uses the intrinsic characteristic of X-rays to pass through the tissues of the human body, and thus form the image that represents the organ. Ramirez et al. [
Artificial intelligence is revolutionizing the way of processing massive volumes of data (Big Data) [
In this work, we used the K-means algorithm because of its simple implementation and great results. K-means developed by MacQueen in 1967 is the most widespread and used [
In virtual environments, the 3D or three-dimensional model is the technique that allows modifying the perception of the observer, providing depth to two-dimensional images. Representing an object on a screen requires an analysis of the morphological characteristics of things. In computing, three-dimensional models are discrete, that is, they are constructed from points located in 3D space, and are interconnected by edges to form graphically visible structures.
Computational graphic models consist of meshes. Authors such as Boening et al. [
Medical images are under the DICOM standard, and the source of generation can be CT or MRI. The files are stored in a folder, where they are properly labeled. Perform delimitation of the region of interest using established characteristics. Perform the generation of three-dimensional models, using the segmentation of medical images. Interconnect a virtual reality visualization device, using virtual reality glasses and a mobile device. Allow interaction with objects within the work environment.
Block diagram of the program.
The module Read Image establishes a one-dimensional array with pixel values of the image. Initially, a data-stack loads the names of the files located in the directory defined by the user. Next, a function receives the stack with names and extracts the pixel values. Finally, the data-array reshapes dimension to nx1.
The next module (Segmentation) aims to group pixels with similar characteristics. This application implements clustering algorithms (k-means, FCM, and SFCM), these algorithms are iterative and unsupervised, they seek patterns in the data, without having a prediction as aim. As input data needs an array with population to segment, and it also requires cluster number, the initialization of centroids is random.
Next, the module Mesh is responsible of generating the 3D mesh, it receives the data from the Segmentation module, and it delivers the file with extension *.
The module Navigate is responsible of reading and loading the meshes, it is also responsible of receiving the information of the joystick or gyroscope sensors of the mobile device, with this data it performs movements of the objects or the camera.
Finally, the VR module is responsible of establishing the gateway between the application running on the computer and the mobile device, where the viewing of the objects happens. This module uses a client-server model to build a bidirectional channel.
Class diagram of the program.
Main window of the application.
For improving the navigation experience, the SIM3D software performs movements from two perspectives. The first of these is the movement of the camera, allowing it to perform rotational and translational changes, which makes it easy to observe from various perspectives. The second way to perform moves is to keep the observer static and move the objects present in the scene, additionally allowing translation and rotation.
This section presents the strategy for the evaluation of the designed software. The analysis includes the computational cost, time, and RAM usage during mesh generation and viewing of a three-dimensional model. Finally, we perform an experiment with a group the people to define the usability of the computational application. Three computers were used to run the tool developed in this work, as indicated in
Processor
Core i5-5200 CPU 2.2GHz
Core i7-6700 CPU 3.5 GHz
Core i5-2400 CPU 3.1 GHz
RAM
6 GB
8 GB
4 GB
Graphic card
Intel HD
GTX 1050 4GB
Intel HD
Operative system
Win 10
Win 10
Win 10
We used a medical MRI image with dimensions of 260x320 pixels for the evaluation of time and RAM metrics. The protocol for the experiment was the following: we performed 10 independent tests and each of them ran 100 iterations, also the number of images in each new inspection increases.
Time and RAM are related to the number of images processed in each run.
1
001
0.265
0.01
0.010
0.01
0.243
0.02
0.010
0.01
0.423
0.02
0.010
0.01
2
002
0.592
0.20
0.028
0.01
0.537
0.05
0.026
0.01
0.951
0.08
0.026
0.01
3
010
2.467
0.50
0.100
0.02
2.358
0.11
0.097
0.09
4.609
0.31
0.097
0.09
4
020
5.049
0.30
0.190
0.01
4.486
0.30
0.190
0.01
8.012
0.45
0.190
0.01
5
030
6.404
0.60
0.250
0.01
5.957
0.31
0.251
0.01
11.48
0.50
0.251
0.01
6
040
9.057
1.09
0.300
0.01
8.252
0.41
0.310
0.01
15.71
1.80
0.310
0.01
7
050
11.51
1.03
0.500
0.10
9.923
1.01
0.493
0.10
20.87
1.57
0.494
0.10
8
100
23.07
0.90
0.900
0.08
21.18
1.10
0.910
0.09
43.67
2.09
0.910
0.09
9
120
27.50
1.01
1.100
0.09
23.82
1.50
1.089
0.10
47.66
2.90
1.099
0.10
10
140
30.44
1.10
1.300
0.10
25.63
2.00
1.281
0.20
55.43
4.01
1.291
0.20
For the reconstruction of the tumor, we used four MRI studies that show a brain tumor.
Primary Glioblastoma layer: (a) Layer 15, (b) Layer 16, (c) Layer 17 and (d) Layer 18 [
Clustering techniques present a very high sensitivity to the selection of centroids [
Rendering is done using the Unity graphics engine.
Segmented image: (a) Layer 15, (b) Layer 16, (c) Layer 17 and (d) Layer 18.
Tumor 3D.
The meshes generated by the computational application use triangles. Figure 7 shows the reconstruction of the tumor seen from various viewpoints, also, the illustration includes views of several sections aiming for the best view.
On the whole, the model 3D generated by the application SIM3D contains vertices than form polygons. It includes meshes generated from medical multilayer images; the segmentation delimits the region in groups or several clusters, and it separates in parts.
Views of the glioblastoma tumor: (a) and (b) top view, (c) back view with rotation, (d) front view with rotation and (e) bottom view.
a
29.462
26.040
26.040
24.932
14.560
67.384
59.588
59.588
58.352
32.044
b
29.666
28.508
24.769
23.195
12.248
68.060
64.682
55.769
53.458
27.024
c
8.555
12.112
7.320
11.696
7.166
18.076
24.430
15.320
24.702
14.828
d
15.202
5.082
0.151
1.883
0.153
32.566
11.042
0.318
3.976
0.316
e
26.837
15.639
09.297
8.928
7.611
60.268
34.616
20.138
12.294
16.486
In the developed system, the visualization is done through virtual reality using stereoscopy technique, that is to say, the models are seen through VR glasses. Moreover, the interaction with the objects visible in the scene uses a video game control. The system starts with the choice of DICOM images directory and the selection of the segmentation algorithm, indicating respective parameters, and with the help of the control it moves the objects, both in translation and rotation.
Authors such as Barajas et al. [
Views of k clusters (a) k = 1, (b) k = 2, (c) k = 3, (d) k = 4 y (e) k = 5.
SIM3D user.
We used a document with ten questions, with nine closed questions and one open question. The questionnaire is easy to answer, and the questions are shown in
P1
Was it easy to make a mental representation of the tumor before using the software?
P2
After using the software, was it easier to understand the two-dimensional images presented at the beginning of the exercise?
P3
Are you satisfied with the scenes shown in the SIM3D application?
P4
Is the degree of difficulty in detecting a tumor in medical images high?
P5
During or after the experiment, did you feel any discomfort?
P6
In general, are you satisfied with the navigability of the software?
P7
Before, during and after finishing the test, were there any difficulties interacting with the SIM3D program?
P8
Is the interface suitable for the application?
P9
How do you rate the ergonomics of the software?
P10
What do you think are the opportunities for software improvement?
The protocol for the execution of the test was divided into three phases: induction, developing, and finally survey answering. In the introductory stage, we inform aspects related to the handling of the SIM3D software, then, each participant observes the images that later load in the computational application. The user must make an imaginative description of the tumor shown in the pictures. In the development stage, the participants must define initial parameters, a directory that includes the images and a cluster number. After that, they perform the following actions: first, move the visible objects in the scene, in the three dimensions, using the video game control (Joystick). Secondly, the head moves to adjust the position of the camera. Thirdly, they perform the incorporation or rejection of objects that uses the buttons in the application or the video game control (Joystick). In the final phase, they must answer the survey.
The participants gave their opinion using the five point Likert scale, where they can select: totally agree (TA), agree (D), neutral (N), disagree (ED) and totally disagree (TD). Question 10 is an open question in which they capture their opinion without limitations. Figure 10 shows the results of the questionnaire. We located in the abscissa axis the index of the nine questions that shows in
Although the results of the questions identified with numbers one and four testify the difficulty in the interpretation of 2D images (traditional method), the answers of the second question let us infer that the implemented application improves the perception of the information contained in medical images, since 80% completely agree, 15% agree, and only 5% remain neutral (see
Result of the questionnaire.
Besides, the simplicity for the user to scroll through the screen and use the icons of the application makes it attractive to use. Questions six, seven and nine allowed us to obtain the perception of software navigability, and we found that 60% strongly agree, 25% agree, and 15% remain neutral (see
Using screens at a close visual range causes effects on the human body such as eyestrain, dizziness, nausea, among others. To determine any change in the participants, we asked about any discomfort presented during or after the test. Here, 15 % agreed (see
The open question allows the respondent to answer using their own words and reveal additional information about improvements to the computer program. There is a tendency to improve three-dimensional graphics, some of the answers were: "Eliminate pointed angles between faces"; "objects must be smooth surfaces"; "non-square graphics"; among others. There are also suggestions about the implementation of a security protocol to protect clinical information. Also, users showed interest in the application being multi-user.
The difference between the research presented by Indraswari
In this article, we presented the design and implementation of the software named SIM3D, for the segmentation and generation of three-dimensional models based on multilayer medical images. The delimitation of the region of interest was performed using clustering algorithms (K-means) and the Delaunay triangulation algorithm was used to generate the 3D mesh, as a rendering engine for Unity 3D. The process was implemented in C++ programming language.
We evaluated the SIM3D application in three computers with different hardware resources, and Windows 10 operating system. The computer must have at least 4Gb available in RAM for proper operation. Other than the RAM, there are no significant changes in the three teams.
The evaluation of the program used MRI images from a patient diagnosed with primary glioblastoma. In the magnetic resonance imaging from the head divided in 24 sliders, the tumor is visible in slice 15 to 18. The k-means algorithm selected the region of interest (tumor), and generated three-dimensional meshes.
In the usability tests the participants were a group of multidisciplinary users such as engineering students, engineers, and medical students. The results show a great acceptance of the designed application.
As future work, it is proposed to optimize the mesh generation algorithm, evaluate using the operating system in real-time, use virtual reality glasses like those developed by Oculus VR, to estimate the performance of the application. Also, perform tests with a real-time source of images.