AR/VR applications for Rapid Decision-Making During Inspection & Maintenance

By Nandu Vellal and Satish Mantripragada


Over the last few years, data captured during industrial inspection of parts has advanced due to advancements in sensor technologies, this calls for improvements in data presentation to the operator in which he/she can quickly grasp the extent of the severity of the defect, and make rapid and accurate decisions to not only prevent failure of the component but also to optimize the life of the asset.  

The objective of this whitepaper is to evaluate the inspection process and data processing workflows used currently, and provide suggestions for data presentation techniques using next generation user-interfaces like AR and VR that enables the operator to make rapid and effective decisions.  

While the scenario presented here is for non-destructive evaluation (NDE) of a component using ultrasound sensing system, such analyses can be applied to other NDE techniques as well depending on the nature of the data captured. 


  • Data captured from Ultrasound sensors is processed by control electronics on the device and is presented to the user on a display screen. 
  • Ultrasound scan display data is presented in the form of charts or graphs using A, B, C-scan on the display unit and the operator either based on experience or by comparing against a database of such similar defects categorizes these as low-priority or high-priority (fix immediately).  This consumes considerable amount of time in analyzing the defect data and interpreting it. Operator training, skill and fatigue play key role in identifying, detecting and classifying the defects. 
  • Currently, there is no ability to access the expertise of a senior NDT professional during inspection when the operator is unable to make proper inferences with the data and classify the defects. 
  • Additionally, with current inspection processes there is no continuous recording of data and even the ones that are recorded are defects in a format that is not conducive to 3D representation and difficult to perform audit of the inspection process. 
Existing NDE Process


  • By incorporating the usage of next generation sensors during inspection along with A, B, C-scans a 3D rendering of the defect can be overlayed on the real-time view (image) of the physical component to provide a better visualization to the operator using Augmented Reality (AR). This will increase the speed of decision-making capability of the inspector as he/she can quantify the defect using a 3D image instead of graphs/charts.  
  • A continuous recording of the inspection data will enable detailed analysis by Inspection Engineers in real-time [or at a later time] at control room using VR technology for better visualization and taking necessary corrective maintenance action.  
  • The data recorded can also be used for future audit purposes and training new professionals – first, as a self-learning tool using VR and then as a tool to understand the real-life defect scenarios. 
  • Further, the solution can be enhanced by damage growth studies coupled with AI/ML and Finite Element Analysis to predict defect growth based on various criteria like historical defect data, load on the component, material of the component, location of the defect, ambient conditions, etc.  and assess the severity of the situation by simulating the failure of the component.
Proposed NDE Process

Benefits of Proposed Solution:

  • Better documentation and record keeping of defects and progression of defects using AI/ML [learn and adopt]. The data can be used for auditing the inspection process and used as data sets for future training procedures. 
  • Provides state-of-the-art inspection procedures by leveraging assistance from NDE experts remotely, to address the problems by incorporating better maintenance practices before it becomes a critical issue thereby leading to effective utilization of asset and optimization of the life of the asset. 
  • Faster and effective training of operators by using real-life historical data captured during actual inspection along with VR technology there-by providing faster skills development of inspection personnel. 
  • High clarity data presentation for operator to make quick and rapid decision with guidance from AI/ML (learn from past data) and AR/VR along with existing simulation models. AR for on-site inspection protocols and VR for remote inspection. 

Industry Applications: 

In the image 1 on the right, you see an inspector using AR to identify the components in an electrical control panel to troubleshoot a problem. The visual indicators on the tablet screen are hinting the inspector where to look for, and this leads to faster and better troubleshooting of problems in the field. 


In the image 2 on the left, you see a senior level inspector reviewing the inspection data remotely from a control room using VR. A senior NDT professional can guide a junior professional in the field to quickly solve complex problems in the field. 

AR Application

In a prototype application we are currently building similar to the image 3 above, an inspector, instead of viewing Ultrasound A, B & C-scans (in the left) will be able view AR image of defect overlayed on the part during inspection (in the right). This enables the inspector to quickly quantify the defect and assess the severity of the situation. While such applications are being explored within the industry there are numerous challenges that need to be addressed with respect to technology like a) integration of different types of data from multiple sources [in real-time] is a significant technological challenge b) while the example presented here is using the mature Ultrasound technology for sensors, such 3-dimensional data from other sensing technologies need to be researched and validated c) as newer materials are developed, generating failure models will take significant time, effort and money to collect valid data and perform research AI/ML analysis. 



This article aims to provide insights as to how core tenets of NDE 4.0 - interoperability, decentralized decisions, information transparency, remote technical assistance - can be incorporated as part of next generation NDE solutions by implementing core tenets of NDE 4.0. The rapid innovation underway in sensor technologies and their adoption as part of NDE 4.0, will lead to capture of massive amounts of data from multiple sensor modalities, which are ideal applications for Bid Data and AI/ML. By leveraging these new technologies, the inspectors will be able to perform better, faster and efficient inspection by leveraging the decision-making capabilities of the machine, and lessening the cognitive burden on themselves. 

Leave a Reply

Your email address will not be published. Required fields are marked *