Fractographic classification in metallic materials by using 3D processing and computer vision techniques

Failure analysis aims at collecting information about how and why a failure is produced. The first step in this process is a visual inspection on the flaw surface that will reveal the features, marks


I. IntroductIon
Failure analysis intends to understand why and how a fracture is produced and the way to avoid it.Fractography is a discipline that studies the topographic characteristics resulted from a failure mode, and tries to reveal the superficial features on the fracture by visual inspections, looking for the propagation patterns and fracture origin [1].Through this characterization it is possible to explore and determinate previous conditions that would help find the causes of the rupture.The first step in this examination is spotting the macroscopic features on the surface, followed by a microscopic study with a stereoscope or a Scanning Electron Microscope (SEM), which will identify cleavage facets, transgranular fractures, intergranular fractures, grain boundary morphology, and fatigue striations, among others [2].
Mechanical failures happen when a mechanical element is divided into one or more fragments, followed by three steps: i) crack(s) nucleation, ii) crack(s) propagation, and iii) element's fracture.Depending on the velocity of these steps, the fracture can be classified into sudden, if the cracks spread on the material between 0.2 to 0.4 the speed of sound, or progressive, if the cracks have a slow propagation velocity (e.g., 1mm/day) [1].This study focuses on failures produced by effort solicitation, specifically in three modes: i) brittle fracture with granular texture and river or radial marks; ii) ductile fracture with fibrous texture and no marks; and iii) fatigue fracture, which is divided into two regions: one near the fracture origin with flat texture and Ratched marks (also called saw tooth marks), and another one that could be brittle or ductile [1].
Computer vision technics have been used in research on materials and failure analysis.For instance, SEM and fracture surface rebuilding using previous chosen sub-images, a process called auto-shape analysis, have been used to determine crack growth rate through the texture [3].In many cases, these techniques analyze the images of certain materials, for instance steel API5L-X52, in order to find the micro-cavity's morphology [4].In [5] and [6], they studied fractographic texture methods aiming at reconstructing the history of a fatigue mode fracture based on the relation between the features of the surface texture and the crack growth rate velocity.In 3D analyses, the stereo shape reconstruction is used through stereo images, utilizing an algorithm capable of finding homologue points in the two images [7]; this recovers the shape from multiple images in the same scene, using a fixed visual direction and different source light directions [8].The information given by images in three dimensions allows to determine important quantitative features such as the local roughness, and provide a deep comprehension of the failures mechanisms [9].This information has been used to study roughness and fractality [12], failure morphologic description [13], profile dimension, shape of the topographic surface, volumetric area, and the high steep depth, among others [14].Despite the lack of studies on mechanical surface classification using 3D data sets, there has been some research about classification and recognition of failure surfaces.In [10], the authors worked with images acquired with SEM in three kinds of sintered steel with fatigue fracture mode in order to recognize the morphology, metrics, and porosities in the surface fracture.In the characteristics calculations, they included the morphologic gradient in gray level images and search for the local neighbors to do the classification.In [11], they generated a software able to classify six different morphologies in fracture surfaces, using as descriptors the Fourier spectrum and the gray level co-occurrence matrix.The latter extracts three characteristics: contrast, uniformity, and correlation.The acquisition of the images was made using SEM, and the grouping method was the K-mean algorithm.However, the classification percentage does not exceed 80 % in the majority of the cases.
Traditionally, the fracture classification process is a visual exam that reveals particular structures from each kind of fracture, generally done by nonexpert personnel that usually lack the knowledge or experience required on this topic.Fractographic analysis is the first step in failure analysis, but a wrong classification of the failure mode will make it very hard to find the failure causes, and search for a way to avoid future flaws.This process can be brought to a computer vision problem to support the failure analysis.
The method proposed in this paper acquires 3D images for training and testing in 5x scale from the laser scan confocal microscopy Zeiss LSM 700, which generates several images in different heights allowing to reconstruct 3D clouds by stocking 2D images.We took fifty-four 3D images for training and eighteen 3D images for testing.To characterize the failure surfaces, we used i) Haralick's features and ii) the fractal dimension, both proposed for 2D texture studies, and then expanded to different studies in the 3D space.To classify the failures, we evaluated two methods commonly used in computer vision: Artificial Neural Networks (ANN) and Support Vector Machine (SVM); in addition, we used the confusion matrix with four marginal metrics: accuracy, sensitivity, specificity, and precision to estimate three evaluation methods: Receiver Operating Characteristic (ROC), the Individual Classification Success Index (ICSI), and the Jaccard's coefficient.However, the results obtained by an expert and the 2D study presented in [27] are better than the results found here.This research attains a classification percentage near or over 60 % accuracy for the three analyzed failure modes, and it successfully exceeds the randomness line on the ROC space, which proves that it is possible to classify mechanical fractures based on their particular texture and computer vision techniques.Moreover, it allows future studies to test different methods of 3D surface reconstruction and textural features to classify failure modes.

II. ExpErImEntal procEdurE
To perform the 3D analysis, it was necessary to reconstruct the cloud points used for the training and testing of two classification methods.These clouds resulted from stocking 2D images taken from different heights in z direction.After that, we proceeded to filter the data in order to reduce noise and quantity of points in each cloud, and calculated the texture features.Finally, we classified the testing images into one of the three kinds of fracture examined in this investigation, and evaluated the performance using the confusion matrix and some marginal relations estimated from this matrix.Figure 1 shows the process block diagram, whose details will be explained more carefully in the next subsections.

A. 3D image acquisition and reconstruction
The point clouds were obtained with the confocal laser scanning microscopy Zeiss LSM 700, which acquire 2D images from a fracture surface at different heights and stock them to generate a 3D block from the analyzed surface.As the height changes, a process of focusing and defocusing takes place in different regions of the surface.As a result, a discrimination of the material topology is achieved.Figure 2 displays the 3D reconstruction with the software Zen 2009.
Due to software restrictions, it is not possible to save the 3D model of the surface; however, it is possible to obtain 2D images.From these images and using the C++ libraries, OpenCV, and Point Cloud Library (PCL), we built the point clouds of training and testing.To accomplish this, it was necessary to do an image binarization to determinate the points with information and save its gray level and its coordinates (x, y, z). Figure 3 shows a fracture surface with fatigue fracture, indicating the binarized and gray level images (Figure 3a).It can be observed that, as long as the height changes, new regions of the surface are focused.Figure 3b presents the reconstruction of a point cloud on the left, and the same cloud after we applied a voxel grid filter that reduces the quantity of points from 478.102 to 36.001 on the right.

B. Features and descriptors
Although a fracture mode differentiates from others by its marks and texture, these marks will not be found in all fractured mechanical elements, so we took the texture as the principal descriptor and used features that characterize it.Many 2D analyses have been expanded to 3D.It is important to highlight, as it is showed in [28] and [29], that, even though many methods have been proposed to describe texture, there is not yet a mathematical precise definition for this kind of problem due to the complexity and variation real world textures are put through, such as scale, view point, light, among others.
One of the most common methods to characterize textures is the Haralick's features [15].This technique obtains a Gray Level Co-occurrence Matrix (GLCM) of an image and extracts features from its pixels.The GLCM is a spatial relation between two pixels located at a distance (d) and a direction(θ) between them.In order to take all the directions from a reference pixel into account, it is necessary to convert this matrix into a symmetric one by adding its transposed matrix.Finally, it is also necessary to obtain a probabilistic matrix with , where i is the number of rows, j is the number of columns, is the value of the cell in position (i, j), and N is the total number of rows or columns.There are many possible combinations for distance (d), but it is common to set d = 1 due to its computational simplicity [17].In 3D analysis there are thirteen directions for each reference voxel, since this is surrounded by twenty-six neighbors.Mathematically, it is defined by (1), where is the gray level in the reference voxel, and is the gray level in the neighbor voxel [16].
(1) Considering Haralick's features is one of the most used methods due to its computational simplicity in 2D texture analysis, it was transferred to a 3D space in some studies such as [30], where they proved that texture must be treated as a three-dimensional problem in order to keep the details of the structural elements.
With the co-occurrence matrix, we calculated the features suggested in [15] and listed below, where is a probabilistic matrix, the total number of gray levels on the image, and : • Angular second moment: .
• Contrast: • Correlation: , where are the mean and the standard deviation of .
• Sum of squares or variance: .
• Difference entropy: .Finally, the average was calculated with (2) and the range with (3) in every direction, both are the input features to the classifier, where is each feature and is the total number of directions., . ( The fractal dimension allows to get useful information about the geometric structure in the image [18].While the topological dimension is defined by an integer value that describes the dimension's number an object belongs to (1D, 2D, 3D), the fractal dimension uses real values to describe an object in terms of selfsimilarity and space occupancy [20].The Differential Box Counting method is one of the most used to generate the fractal dimension.This method divides the object in voxels of size and sets a gray level threshold; if the gray level of the reference voxel is over the value of the threshold, then this voxel counts plus one in the total occupancy.The fractal dimension is given by (4), where is the total number of voxels that contain the studied texture [19].

C. Confusion matrix
This matrix, which is developed for each class, allows to evaluate the classification algorithm.The cells in Table 1 are the true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN).Among the marginal relations that can be obtained with this matrix are the accuracy (ACC), the sensitivity (TPR), the specificity (SPC) and the precision (PPV), given by ( 5), ( 6), ( 7) and (8).

D. Artificial Neural Networks (ANN)
The , where β = 0.6 and α = 0.6.The total number of neurons on the hidden layer (NO) were selected taking into account that where .Discriminability (i.e., how a classifier reacts faced with unknown datasets), using the confusion matrix and the accuracy, was used to evaluate the performance of the selected architecture (5).In Table 2, the performance analysis is showed according to the neural network in the hidden layer, with 12 neurons, in column Acc.12N; the best performance is achieved taking into account two out of three failure modes.

E. Support Vector Machine (SVM)
SVM is a discriminant classifier defined by a separating hyperplane, which means that given an input dataset, with its labels, the algorithm generates an optimal hyperplane able to categorize new samples [21].Given a training set of N data points , with as the input pattern and as the output pattern, as it is showed in (9), where and b is a real constant [22], a radial base was selected as a parametric function (RBF), as suggested in [21], , and as a parameter for the optimization problem 4 [23].
The input data for the classifier was previously normalized in order to avoid additional effects over the algorithm, using (10), where is the value to be normalized, are the range of values in the vector, and ) are the range to do the normalization [24].(10)

III. rEsults
The designed algorithm allows to classify fractures in mechanical materials by using computer vision techniques.In this section, the running test of this algorithm is presented using different performance metrics: the ROC space, the individual classification success index, and the Jaccard's coefficient.Additionally, a comparison between the results found here and the ones obtained by experts on the topic is presented.

A. Algorithm test
Previously, a classification of different pieces with the three kinds of fractures mentioned above was made using 2D images [27].The results showed that the approach the authors used can be compared to the classification percentage made by an expert.In order to analyze the contribution of 3D point clouds, we decided to use the same training and testing pieces used in [27]; however, some of the pieces were too big or heavy to be analyzed with the microscope.Finally, 27 elements were selected for the training and 12 pieces for testing.For each piece, two regions of interest (ROI) were obtained.According to the obtained resolution and denoising, and the roughness of the piece, the microscope LSM 700 can take from 10 minutes up to a whole day to obtain a point cloud.For this research, an average of 25 minutes per ROI was selected considering the acquisition time and the resolution of the piece.Figure 5 shows three pieces, one for each fracture mode, with their 3D reconstruction, the classification given by the expert (one of the authors of this research), and the one given by the algorithm.

FIg. 5.
Comparison of three analyzed pieces between the algorithm classification using ANN and the one given by the expert.
Table 3 presents the classification percentage for the 24 testing images, taking into account measures through the confusion matrix -equations ( 5), ( 6) and (7).It can be observed that, for surfaces with ductile sudden fractures and progressive fractures due to fatigue, the ANN classifier discriminated the true positives better than SVM, represented by the sensitivity relation TPR; in addition, for brittle sudden fractures, ANN classifier discrimination rate of true negatives was 100 %, represented by the specificity SPC metric.In a global way, a good performance was achieved for brittle sudden and progressive fractures due to fatigue with 75 % accuracy, and an acceptable performance for ductile sudden fractures, whose discrimination of true negatives is 56 %.SVM discriminated among ductile and brittle fractures with a 58 % and 50 % accuracy respectively; however, the results for progressive fractures due to fatigue are not correct.The sensitivity TPR percentage is evidence of this.

B. Performance evaluation
In order to evaluate the classifiers, we used: i) the ROC space (Receiving Operating Characteristic), where the sensitivity vs one minus the specificity is ploted (a perfect classifier must be located in coordinates (0,1) on ROC plane) [25]; and the metrics showed in [26]: ii) the individual classification of success index (ICSI) (11), and iii) Jaccard's coefficient (12).These methods depend on the marginal relations of sensitivity (TPR) (6) and precision (PPV) (8).In Figure 6, the ROC space is displayed for the three kinds of fractures.With ANN, the randomness line is exceeded for the three modes and is much better than SVM with ductile and fatigue fractures -even fractures due to fatigue do not exceed the randomness line-.However, SVM has a better discrimination for brittle fractures.For this kind of fracture, both classifiers had a discrimination of 100 % for true positives.
Table 4 shows the results with the performance indicators for the classification according to each failure mode.ANN performed better than SVM taking into account the three fracture classes, and SVM performed better for brittle sudden fractures.Even though the results obtained with 3D images in texture analysis for fracture classification are not better than the ones obtained by an expert or previous research using 2D images [27], it is interesting to analyze them.With SVM, it is possible to better discriminate brittle fractures; however, the accuracy percentages for the other two fracture modes are below 59 %, and the ICSI has a negative or zero value; whereas, with ANN, an accuracy percentage of 58 % for ductile fractures and 75 % for the other two types was obtained.The three failure modes exceeded the randomness line in the ROC space, which proves that it is possible to classify these kinds of fractures by using 3D images.Furthermore, the initial hipothesis of using texture features in order to describe fracture surfaces is correct, which allows future works to aproximate themselves to generate an expert concept on the topic, testing different texture features and aqcuisition methods.

C. Expert validation
The research presented here had the support of three experts on materials engineering.The first expert, one of the authors of this paper, had physical access to the piece and made the initial classification of the surface fractures.The other two experts observed the 2D images of each piece and classified them into one of the three modes.The results obtained from the experts and the algorithm in terms of total pieces correctly classified are presented in Table 5. Considering that the total of testing pieces was 12, it can be observed that the classification range obtained by experts is between 83.3 % and 66.6 % These percentages allows to conclude that the fracture classification is not a simple problem and developing a computational tool that is able to do this classification would support the failure analysis.

Iv. conclusIons
This research aims at classifying failure modes in: i) Ductile sudden fracture, ii) Brittle sudden fracture, and iii) progressive fracture due to fatigue, in order to support the fractographic study in failure analysis.To analyze the contribution of 3D images, the acquisition of the point cloud was made with the confocal laser scanning microscope Zeiss LSM 700 with a target of 5x, and representative ROIs were selected in order to be used as testing and training in the algorithm.In total, 27 pieces were used for training, 9 for each fracture mode, and 12 testing pieces, 4 for each failure mode.The first step to do a fractographic analysis corresponds to a visual inspection of the fractured surface, where the observation of particular features (marks and textures) is made to classify the flaw in one of the failure modes.In this work, the fractures were characterized based on their particular texture because there are no marks in every fracture surface, and descriptors were generated by the gray level cooccurrence matrix, Haralick's features, and the fractal dimension.The obtained results allow classifying the textures in ductile sudden fractures (58 %) and brittle sudden and progressive fractures due to fatigue (75 %), according to the accuracy percentage, and the performance evaluation with ROC space, which exceeded the randomness line of this space.The results were compared to the classification made by two experts on the topic.The experts observed the pieces through 2D images, with a classification of total pieces between 83 % and 66 %, which allows to conclude that the classification of fractured surfaces is not a trivial problem and that developing a computational tool could support failure analysis.The 3D study is significant and deserves to be evaluated.For future researches, we suggest to test different methods of characterization and use more point clouds for testing and training the algorithm.

Figure 4
exhibits the results for two of these features of the training images; figure4ashows the average of the inverse difference moment, which is related to the homogeneity of the image (low values mean low homogeneity and vice versa), and figure4bshows the fractal dimension of training images.The classification is difficult and requires the use of classifiers that allow to discriminate among classes.For this purpose, two classifiers where evaluated: ANN y SVM.

4 .
(a) Inverse difference moment average.(b) FD.FIg.Behavior of two features obtained by the training images.
of the neural network implemented here are: training method: Backpropagation; stop criteria: 1000 Iterations + Épsilon: 0.00001; Number of layers: three, one at the input (NC), one hidden (NO), and one at the output (Nclas); Number of neurons per layer: 23 in NC, 12 in NO, and 3 in Nclas; Activation function: SIGMOID_SYM

FIg. 7 .
FIg. 7. ROC space of evaluation for the classifiers ANN and SVM, where SD: ductile sudden, PF: Progressive due to Fatigue, and SF: Brittle Sudden.

tablE 4
results of evaluation performanCe of ann Classifier and svm aCCording to eaCh mode