Abstract

Augmented reality (AR) has already helped manufacturers realize value across a variety of domains, including assistance in maintenance, process monitoring, and product assembly. However, coordinating traditional engineering data representations into AR systems without loss of context and information remains a challenge. A major barrier is the lack of interoperability between manufacturing-specific data models and AR-capable data representations. In response, we present a pipeline for porting standards-based design and inspection data into an AR scene. As a result, product manufacturing information with three-dimensional (3D) model data and corresponding inspection results are successfully overlaid onto a physical part. We demonstrate our pipeline by interacting with annotated parts while continuously tracking their pose and orientation. We then validate the pipeline by testing against six fully toleranced design models, accompanied by idealized inspection results. Our work (1) provides insight on how to address fundamental issues related to interoperability between domain-specific models and AR systems and (2) establishes an open software pipeline from which others can implement and further develop.

References

1.
Egger
,
J.
, and
Masood
,
T.
,
2020
, “
Augmented Reality in Support of Intelligent Manufacturing—A Systematic Literature Review
,”
Comput. Indus. Eng.
,
140
, p.
106195
.
2.
Pedone
,
G.
, and
Mezgár
,
I.
,
2018
, “
Model Similarity Evidence and Interoperability Affinity in Cloud-Ready Industry 4.0 Technologies
,”
Comput. Indus.
,
100
, pp.
278
286
.
3.
Kwon
,
S.
,
Monnier
,
L. V.
,
Barbau
,
R.
, and
Bernstein
,
W. Z.
,
2020
, “
Enriching Standards-Based Digital Thread by Fusing As-Designed and As-Inspected Data Using Knowledge Graphs
,”
Adv. Eng. Inform.
,
46
, p.
101102
.
4.
Hanke
,
A.
,
Vernica
,
T.
, and
Bernstein
,
W. Z.
,
2020
, “
Linking Performance Data and Geospatial Information of Manufacturing Assets Through Standard Representations
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Virtual, Online
,
Aug. 17–19
.
5.
Wang
,
X.
,
Ong
,
S. K.
, and
Nee
,
A. Y.
,
2016
, “
A Comprehensive Survey of Augmented Reality Assembly Research
,”
Adv. Manufact.
,
4
(
1
), pp.
1
22
.
6.
Radkowski
,
R.
,
2016
, “
Object Tracking With a Range Camera for Augmented Reality Assembly Assistance
,”
ASME J. Comput. Inf. Sci. Eng.
,
16
(
1
), p.
011004
.
7.
Palmarini
,
R.
,
Erkoyuncu
,
J. A.
,
Roy
,
R.
, and
Torabmostaedi
,
H.
,
2018
, “
A Systematic Review of Augmented Reality Applications in Maintenance
,”
Rob. Comput.-Int. Manufact.
,
49
, pp.
215
228
.
8.
Young
,
C.
, and
Rai
,
R.
,
2021
, “
Automated Procedure Reconfiguration Framework for Augmented Reality-Guided Maintenance Applications
,”
ASME J. Comput. Inf. Sci. Eng.
,
21
(
6
), p.
061011
.
9.
Ong
,
S. K.
,
Zhang
,
J.
,
Shen
,
Y.
, and
Nee
,
A. Y.
,
2011
, “Augmented Reality in Product Development and Manufacturing”
Handbook of Augmented Reality
,
B.
Furht
, ed.,
Springer
,
New York
, pp.
651
669
.
10.
Jiang
,
S.
, and
Nee
,
A.
,
2013
, “
A Novel Facility Layout Planning and Optimization Methodology
,”
CIRP. Ann.
,
62
(
1
), pp.
483
486
.
11.
Ong
,
S.
,
Yuan
,
M.
, and
Nee
,
A.
,
2008
, “
Augmented Reality Applications in Manufacturing: A Survey
,”
Int. J. Prod. Res.
,
46
(
10
), pp.
2707
2742
.
12.
Liu
,
R.
,
Duan
,
G.-j.
, and
Liu
,
J.
,
2019
, “
A Framework for Model-Based Integrated Inspection
,”
Int. J. Adv. Manuf. Technol.
,
103
(
9
), pp.
3643
3665
.
13.
ASME Y14.5-2009
,
2009
.
Dimensioning and Tolerancing.
American Society of Mechanical Engineers
,
New York
.
14.
Kwon
,
S.
,
Mun
,
D.
,
Kim
,
B. C.
,
Han
,
S.
, and
Suh
,
H.-W.
,
2019
, “
B-Rep Model Simplification Using Selective and Iterative Volume Decomposition to Obtain Finer Multi-Resolution Models
,”
Comput.-Aided Design
,
112
, pp.
23
34
.
15.
Pratt
,
M. J.
,
Anderson
,
B. D.
, and
Ranger
,
T.
,
2005
, “
Towards the Standardized Exchange of Parameterized Feature-Based CAD Models
,”
Comput.-Aided Design
,
37
(
12
), pp.
1251
1265
.
16.
ISO 10303-242:2014
,
2014
,
Industrial Automation Systems and Integration – Product Data Representation and Exchange - Part 242: Application Protocol: Managed Model-based 3D Engineering
,
International Organization for Standardization
,
Geneva, Switzerland
.
17.
Feeney
,
A. B.
,
Frechette
,
S. P.
, and
Srinivasan
,
V.
,
2015
, “
A Portrait of an ISO STEP Tolerancing Standard as an Enabler of Smart Manufacturing Systems
,”
ASME J. Comput. Inf. Sci. Eng.
,
15
(
2
), p.
021001
.
18.
AP242.org
,
2009
. Development of a Convergent Modular STEP Application Protocol Based on AP 203 and AP 214: STEP AP 242 –Managed Model Based 3D Engineering. Version 1.0.
19.
ISO 1101:2012
,
2012
,
Geometrical Product Specifications (GPS) –Geometrical Tolerancing – Tolerances of Form, Orientation, Location, and Run-out
,
International Organization for Standardization
,
Geneva, Switzerland
.
20.
ISO 23952:2020
,
2020
,
Industrial Automation Systems and Integration – Quality Information Framework (QIF) – An Integrated Model for Manufacturing Quality Information
,
International Organization for Standardization
,
Geneva, Switzerland
.
21.
Trainer
,
A.
,
Feeney
,
A. B.
, and
Hedberg
,
T. D.
,
2015
, “
Validation for Downstream Computer Aided Manufacturing and Coordinate Metrology Processes
.”
NIST GCR 16-003.
22.
Hall
,
S.
, and
Takahashi
,
R.
,
2017
, “
Augmented and Virtual Reality: The Promise and Peril of Immersive Technologies
.”
Media & Entertainment.
23.
Urbas
,
U.
,
Vrabic
,
R.
, and
Vukašinović
,
N.
,
2019
, “
Displaying Product Manufacturing Information in Augmented Reality for Inspection
,”
Proc. CIRP
,
81
, pp.
832
837
.
24.
Kleverud
,
F.
,
2018
, “
Interactive Visualization of CAD Data in Real Time Rendering
,” Master's Thesis, Department of Industrial and Materials Science, Chalmers University of Technology, Gothenburg, Sweden.
25.
Fang
,
F. Z.
,
Li
,
Z.
,
Arokiam
,
A.
, and
Gorman
,
T.
,
2016
, “
Closed Loop PMI Driven Dimensional Quality Lifecycle Management Approach for Smart Manufacturing System
,”
Proc. CIRP
,
56
, pp.
614
619
.
26.
Polvi
,
J.
,
Taketomi
,
T.
,
Moteki
,
A.
,
Yoshitake
,
T.
,
Fukuoka
,
T.
,
Yamamoto
,
G.
,
Sandor
,
C.
, and
Kato
,
H.
,
2017
, “
Handheld Guides in Inspection Tasks: Augmented Reality Versus Picture
,”
IEEE Trans. Visualiz. Comput. Graphics
,
24
(
7
), pp.
2118
2128
.
27.
Runji
,
J. M.
, and
Lin
,
C. -Y.
,
2020
, “
Markerless Cooperative Augmented Reality-based Smart Manufacturing Double-check System: Case of Safe Pcba Inspection Following Automatic Optical Inspection
,”
Rob. Comput.-Int. Manufact.
,
64
, p.
101957
.
28.
Li
,
S.
,
Zheng
,
P.
, and
Zheng
,
L.
,
2020
, “
An Ar-Assisted Deep Learning-Based Approach for Automatic Inspection of Aviation Connectors
,”
IEEE Trans. Indus. Inform.
,
17
(
3
), pp.
1721
1731
.
29.
Fiorentino
,
M.
,
Monno
,
G.
, and
Uva
,
A.
,
2009
, “
Tangible Digital Master for Product Lifecycle Management in Augmented Reality
,”
Int. J. Inter. Design Manufact. (IJIDeM)
,
3
(
2
), pp.
121
129
.
30.
Fiorentino
,
M.
,
Uva
,
A. E.
, and
Monno
,
G.
,
2011
, “
Product Manufacturing Information Management in Interactive Augmented Technical Drawings
,”
World Conference on Innovative Virtual Reality
, Vol.
44328
, pp.
113
122
.
31.
Urbas
,
U.
,
Ariansyah
,
D.
,
Erkoyuncu
,
J. A.
, and
Vukainovi
,
N.
,
2021
, “
Augmented Reality Aided Inspection of Gears
,”
Tehnički vjesnik
,
28
(
3
), pp.
1032
1037
.
32.
Sharp
,
M.
,
Hedberg Jr.
,
T.
,
Bernstein
,
W.
, and
Kwon
,
S.
,
2021
, “
Feasibility Study for An Automated Engineering Change Process
,”
Int. J. Prod. Res.
,
59
(
2
), pp.
1
16
.
33.
Schroeder
,
W. J.
,
Zarge
,
J. A.
, and
Lorensen
,
W. E.
,
1992
, “
Decimation of Triangle Meshes
,” Proceedings of the 19th annual conference on Computer graphics and interactive techniques, pp.
65
70
.
34.
ISO 10303-21:2002
,
2002
,
Industrial Automation Systems and Integration - Product Data Representation and Exchange - Part 21: Implementation Methods: Clear Text Encoding of the Exchange Structure
,
International Organization for Standardization
,
Geneva, Switzerland
.
35.
ISO/IEC 19775:2013
,
2013
,
Information Technology–Computer Graphics, Image Processing and Environmental Data Representation–Extensible 3D (X3D)
,
International Organization for Standardization
,
Geneva, Switzerland
.
36.
Lipman
,
R.
, and
Kwon
,
S.
,
2021
, “
STEP File Analyzer and Viewer User Guide (Update 7)
”. Advanced Manufacturing Series (NIST AMS), 200-12.
37.
Stefan
,
L.
,
Hermon
,
S.
, and
Faka
,
M.
,
2018
, “
Prototyping 3D Virtual Learning Environments With X3D-Based Content and Visualization Tools
,”
BRAIN. Broad Res. Artifi. Intel. Neurosci.
,
9
, pp.
6
20
.
38.
Consortium
,
V.
,
1997
, ISO/IEC 14772-1 1997 the Virtual Reality Modeling Language (vrml97).
39.
Ishii
,
H.
,
2008
, “
The Tangible User Interface and Its Evolution
,”
Commun. ACM
,
51
(
6
), pp.
32
36
.
40.
Sobel
,
W.
, and
Eckenrode
,
R.
,
2017
, Mxd final report project 14-10-01. avm standards development and promulgation. Technical Report, System Insights.
41.
Jauhar
,
T. A.
,
Han
,
S.
, and
Kwon
,
S.
,
2021
, “
Downstream Computer-aided Design, Engineering, and Manufacturing Integration Using Exchangeable Persistent Identifiers in Neutral Re-imported Computer-Aided Design Models
,”
ASME J. Comput. Inf. Sci. Eng.
,
21
(
1
), p.
014501
.
42.
Sanjiv
,
K.
,
2016
, “
How Augmented Reality Can Revolutionize Manufacturing
.”
Industry Week.
43.
Carpendale
,
M. S. T.
,
2003
, “
Considering Visual Variables as a Basis for Information Visualisation
”.
44.
Regazzoni
,
D.
,
Rizzi
,
C.
, and
Vitali
,
A.
,
2018
, “
Virtual Reality Applications: Guidelines to Design Natural User Interface
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Quebec City, Quebec, Canada
,
Aug. 26–29
, Vol. 51739, American Society of Mechanical Engineers, p. V01BT02A029.
45.
Ghazwani
,
Y.
, and
Smith
,
S.
,
2020
, “
Interaction in Augmented Reality: Challenges to Enhance User Experience
,” Proceedings of the 2020 4th International Conference on Virtual and Augmented Reality Simulations, pp.
39
44
.
You do not currently have access to this content.