Graphical Abstract Figure
Graphical Abstract Figure
Close modal

Abstract

In the contemporary era of engineering education, the integration of large language models, offers a novel perspective on enhancing the design process. This study investigates the impact of ChatGPT-3.5 on mechanical engineering design education, focusing on concept generation and detailed modeling. By comparing outcomes from artificial intelligence (AI)-assisted groups to those without AI assistance, our research reveals that AI significantly broadens concept generation diversity but also introduces bias for existing popular designs. Additionally, while AI aids in suggesting functions for computer-aided design (CAD) modeling, its textual nature and the occurrence of unreliable responses limit its usefulness in detailed CAD modeling tasks, highlighting the irreplaceable value of traditional learning materials and hands-on practice. The study concludes that AI should serve as a complement to, rather than a replacement for, traditional design education. Additionally, it highlights the necessity for further specialization within AI to enhance its effectiveness.

1 Introduction

In the ever-evolving landscape of our era, humanity is perpetually confronted with fresh challenges. Consequently, nurturing the ability to tackle unforeseen problems within engineering education is important. Designers frequently draw inspiration from pre-existing designs, leveraging this strategy to substantially lower the expenses linked to initiating designs from the ground up. However, this ease of access might unintentionally fixate their thinking, limiting their creative potential and hindering the development of innovative design concepts [1].

To address this fixation challenge, innovative approaches such as brainstorming [2], mind mapping [3], and design principles [4] have been developed. These techniques aim to facilitate designers in transcending traditional thought processes and exploring beyond the boundaries of existing design paradigms. In the current digital age, where a vast array of engineering designs is easily accessible, researchers are delving into numerous data-driven methods to foster innovation in design from general design heuristic workflow [5] to more specific domain-focused machine learning models for specific design questions [68]. However, these works tend to be highly specialized, making them difficult for the broader audience to utilize and comprehend, thereby diminishing their overall influence.

The emergence of large langauage models (LLMs) might change this situation and exacerbate this limitation of design innovation significantly. LLMs are language models that have been trained with large datasets of textual information and can be applied to deal with various language-related tasks [9]. For example, the LLM discussed in this work, ChatGPT-3.5 [10], was trained on a 400 billion tokens dataset. These huge datasets offer LLMs not only the capability to chat with human users but also the knowledge of the textual dataset, which is the reason why LLMs have been applied in many downstream domains like medical suggestions [11], advertising [12], and education [13].

Given that LLMs derive their learning entirely from pre-existing databases, they intrinsically face a shortage of learning material for unfamiliar challenges, leading to feedback that is essentially anchored to analogous instances within their training data. Furthermore, the interactive capabilities of LLMs might foster a misleading perception among designers that the outputs are custom-crafted for their unique problems. This phenomenon could further entrench designers’ thought processes, impeding the pursuit of genuinely innovative design concepts.

In addition to concept generation, we aim to assess the performance of LLMs in detailed computer-aided design (CAD) modeling tasks, which follow the concept generation phase. Concept generation relies heavily on creativity and broad ideation, where the focus is on generating innovative ideas, exploring various design possibilities, and thinking outside of established norms. This stage often involves brainstorming, conceptual sketches, and considering different design alternatives without being constrained by technical limitations. At this stage, tools like ChatGPT can offer substantial support by providing inspiration, and offering insights into different design concepts based on broad knowledge.

On the other hand, detailed CAD modeling is a more technical and precision-driven phase of the design process. It involves translating the chosen concept into a fully defined model, where every component must meet specific functional and dimensional requirements. This stage requires a deep understanding of engineering principles, geometric constraints, and the specific tools available within CAD software. The training data for LLMs often include material related to popular CAD software such as solidworks and freecad.

This study delves into the shifting landscape of challenges and opportunities introduced by the advent of LLMs, specifically ChatGPT-3.5, within the realm of mechanical engineering design education. This paper contributes primarily to two distinct yet interconnected areas concerning the augmentation of engineering design through machine learning, focusing on concept generation and detailed CAD modeling. We assessed ChatGPT-3.5’s performance across three dimensions: the quality of design outcomes, user feedback, and the comparative analysis of designs produced with and without ChatGPT-3.5’s assistance. The remainder of this paper is organized as follows: Section 2 reviewed related works about LLMs in education and engineering design.solidworks and freecad. This material typically originates from online tutorials and forums that provide detailed operational guidelines. Given this context, we hypothesize that LLMs have the potential to support detailed CAD modeling tasks. However, the potential for unreliable responses [14] from LLMs raises concerns about their ability of detailed CAD modeling, which remains an open question.

This study delves into the shifting landscape of challenges and opportunities introduced by the advent of LLMs, specifically ChatGPT-3.5, within the realm of mechanical engineering design education. This paper contributes primarily to two distinct yet interconnected areas concerning the augmentation of engineering design through machine learning, focusing on concept generation and detailed CAD modeling. We assessed ChatGPT-3.5’s performance across three dimensions: the quality of design outcomes, user feedback, and the comparative analysis of designs produced with and without ChatGPT-3.5’s assistance. The remainder of this paper is organized as follows: Section 2 reviewed related works about LLMs in education and engineering design. Section 3 elucidates the methodologies employed in the concept generation and detailed CAD modeling studies. Section 4 presents the findings from these two case studies. Section 5 discusses the limitations of this study. Lastly, Sec. 6 synthesizes the research results, offering insights into the future trajectory of engineering design education in the age of LLMs.

2 Related Works

In education, scholars from various disciplines have been actively exploring the potential applications of LLMs. In journalism and media, for example, these tools can significantly accelerate news or report writing while also enhancing the quality of students’ work [15]. In medical education, ChatGPT has demonstrated its ability to pass the United States Medical Licensing Exam, highlighting its potential in advancing student performance in assessments [16]. These studies suggest that LLMs may influence student evaluation methods and the design of course materials [17].

Although most current LLMs are general-purpose models that have not yet specialized in the engineering domain, evaluating their impact on engineering education is still essential. Several studies have already begun to address this.

In the context of engineering thesis writing, Bernabei et al. [18] found that students were able to produce high-quality essays with well-distributed grades. The advantages included enhanced organization of language, effective summarization, and the ability to expand on thesis content. However, drawbacks were noted, such as the potential for generating incorrect or misleading information. Similarly, in chemical engineering education, Tsai et al. [19] conducted a study involving a project on calculating steam power plant efficiency. They discovered that LLMs provided well-structured code and demonstrated accurate comprehension of complex concepts. Despite these strengths, the LLMs occasionally produced errors, and significant teaching time was required to address and correct these mistakes. In the field of computer science, Liu et al. [20] explored a “guardrailed” approach to artificial intelligence (AI) use in education. This method proved more beneficial than outright banning AI tools, as it allowed for the advantages of AI assistance while helping to mitigate potential academic dishonesty—a concern also highlighted by Cotton et al. [21]. Expanding to geotechnical engineering, Kim et al. [22] examined how ChatGPT could support programming tasks through conversational prompts. While ChatGPT could not fully replace the programming process, it effectively reduced syntax errors and provided a foundational framework for logical programming, assisting students in developing their coding skills. Furthermore, in mechanical engineering, Tian et al. [23] investigated the ability of LLMs to answer conceptual questions in mechanics. Their study found that GPT-4 outperformed human students across various mechanics topics, demonstrating its significant potential in enhancing comprehension and academic performance within the field.

From the perspective of engineering design education, the impact of introducing LLMs to support students is still not fully understood. Our work offers unique advantages: (1) we have a representative sample of over one hundred students participating in our project; (2) we possess design data from Fall 2022 as a comparison group, collected before ChatGPT was released; and (3) our study focuses on both stages of the design process—concept generation and detailed CAD modeling—allowing us to examine different aspects of LLM performance.

3 Methodologies

This section delineates the research methodologies utilized in this study. Before the case study carried out in the design course, we conducted an introductory lecture on prompt guidance to ensure a uniform level of proficiency in using ChatGPT across the class, as elaborated in Sec. 3.1.

Subsequently, Sec. 3.2 delves into the exploration of ChatGPT-3.5’s role in the concept generation process for pipe inspection robots. As shown in Fig. 1 case study 1, this exploration unfolds in four stages: (1) division of participants into groups, (2) facilitation of in-class team-based concept generation sessions, (3) collection and comparison of feedback on designs aided by ChatGPT-3.5 versus those researched through Google, and (4) summary and analysis of the generated design concepts.

Fig. 1
Overall workflow of investigation on applying LLM in engineering design education
Fig. 1
Overall workflow of investigation on applying LLM in engineering design education
Close modal

Finally, Sec. 3.3 evaluates the effectiveness of ChatGPT-3.5 as a tool in guiding students to create a specific object in CAD software. As shown in Fig. 1 case study 2, this assessment is structured in three phases: (1) tasking students with creating an uncommon geometry not covered in the course syllabus, (2) allowing students the autonomy to utilize any resources at their disposal, and (3) evaluating the outcomes based on design accuracy and student feedback.

3.1 Prompt Guidance.

To effectively introduce lower-division undergraduates to using ChatGPT for this project, we adopted the concept of progressive prompts as outlined by Zheng et al. [24]. The efficacy of LLMs largely depends on how prompts are designed. Techniques like chain-of-thought prompting [25] and self-consistency are crucial for enhancing this capability, helping users receive more structured and thoughtful responses. Progressive prompts leverage previous answers generated by the LLM to incrementally guide users toward more accurate and refined responses, gradually improving their understanding and decisions.

In this project, we specifically provided guidelines for using progressive prompts to help students design pipe inspection robots. The prompts were systematically structured to follow a three-phase design approach [26] consisting of (1) concept generation, (2) concept refinement (sketching), and (3) final concept selection for further design development with solidworks. These phases map closely to the progressive prompt approach, guiding students step-by-step through the design process.

We structured our prompt approach into four steps to align with both progressive and structured prompt engineering

  1. Outlining general ideas and concepts: This step is aligned with concept generation. Here, students guide the LLM to brainstorm innovative ideas, ensuring they cover various possible design approaches.

  2. Providing insights into components and materials: This step helps students sketch out specific details of their concepts, such as choosing components and materials that meet the design criteria. The progressive prompts ensure that the feedback of the LLM are self-consistency.

  3. Conducting a comparative analysis and discussing the advantages and disadvantages: In this phase, the prompts assist students in evaluating different design options by comparing their strengths and weaknesses. This stage corresponds to refining the design concepts, helping students make informed decisions about which design is most suitable for their needs.

  4. Assessing various concepts: This final step relates to the future development of the final design in solidworks. Here, the focus is on assessing concepts based on the overall preference from students to determine the one which they will develop in the future.

By using progressive prompts, students systematically guided the LLM through the design process, including concept generation, concept refinement, and final concept selection. Each step built upon the previous one, ensuring a structured approach that helped students improve their design solutions iteratively.

We introduced the use of personas in prompting within our project, where students directed ChatGPT to assume roles such as mechanical or manufacturing engineers. Both personas are critical during the engineering design process. Mechanical engineers play a pivotal role in the early stages of design, where they conceptualize innovative ideas and create the blueprint for what the product could become. They ensure that the design meets the required functional specifications. Mechanical engineers are also deeply involved in prototype development, testing, and refining concepts through simulations, models, and physical testing. In contrast, manufacturing engineers focus on transitioning the designs conceived by mechanical engineers into large-scale production. They optimize manufacturing processes, select appropriate materials, and establish production workflows to ensure that the product is produced efficiently and cost-effectively [27].

The students involved in this project are first- or second-year undergraduates who have limited knowledge of engineering design and production workflows. As such, actual physics, such as component bending under force or wheel torque, were not considered in this design project. However, students were still required to apply the basic manufacturing knowledge they had learned, such as prioritizing the use of standard components (e.g., screws and nuts) and considering geometric limitations of processes like milling and turning. To include these factors, we introduced the personas of mechanical and manufacturing engineers to guide the generation of responses, significantly influencing the type of feedback students received.

When interacting with a mechanical engineer persona, the responses were more likely to focus on the early design stages, offering suggestions for conceptual designs such as snake-based or inflation-based mechanisms. The system provided detailed insights into how specific components, such as robotic arms or camera holders, could be designed to meet performance criteria. Conversely, when the students interacted with a manufacturing engineer persona, the responses shifted toward practical implementation, production efficiency, and scalability. The suggestions included recommendations for selecting cost-effective materials (e.g., metal sheets and standard components) and improving assembly processes (e.g., incorporating snap-fit designs).

By using progressive prompt engineering and personas, students were able to systematically guide the design process, receiving comprehensive feedback that bridged the gap between theoretical ideas and practical implementation. This method not only improved their understanding of the mechanical design process but also empowered them to use AI tools effectively for engineering insights.

3.2 Concept Generation of Pipe Inspection Robots.

When integrating LLMs into engineering processes, leveraging them for high-level tasks such as concept generation becomes a logical choice. Design engineering is frequently required to synthesize knowledge across various disciplines. LLMs, with their training on diverse datasets that span multiple fields, are inherently equipped to support the investigation of multifaceted knowledge. Their strength lies in ideating and formulating concepts by drawing upon the extensive data they have been trained on. Within an engineering framework, this translates to LLMs’ ability to promptly offer a broad spectrum of solutions, methodologies, and concepts upon receiving a problem statement or design parameters. Such a capability is valuable during the initial phases of design, where the objective is to explore a wide array of possibilities before focusing on the most feasible options. Therefore, a concept generation case study is formulated to understand the impact of LLMs in concept generation.

3.2.1 Project Formulation.

This project is formulated based on the MECH290 Design Graphics for Mechanical Engineering course in the Department of Mechanical Engineering of McGill University. Typically, this course enrolls around 100 students, with 94 students in the fall of 2022 and 110 students in the fall of 2023, providing a substantial sample size for our study. To effectively evaluate the impact of ChatGPT, it is crucial to include a comparison group unaffected by ChatGPT. The most feasible approach to establish such a group was to use the outcomes from a previous semester (Fall 2022), given that ChatGPT was released toward the end of 2022. Consequently, we replicated the same design project from that semester, which focused on developing pipe inspection robots and adhered to the following six design requirements:

  • — The robot should have a camera-holding mechanism/structure.

  • — The robot should have a locomotion mechanism that allows movement of the robot like turning, for example, in T-connections.

  • — The robot should have space left for NEMA8 motors wherever needed (to enable movements).

  • — A camera (GoPro HERO5 Black) should be attached to the robot to enable visual inspection of the interior of a pipe (the camera model will be obtained from the Reverse Engineering part of the project).

  • — All parts should be connected mechanically.

  • — The minimum pipe diameter is 500 mm, and the maximum is 750 mm. The robot should be able to move vertically in the pipe with friction from the pipe wall, as shown in Fig. 2.

Fig. 2
The pipe to be inspected
Fig. 2
The pipe to be inspected
Close modal

3.2.2 Group Deviation.

This project is structured as a team-based endeavor, with each team comprising 4 to 5 students. Therefore, it is imperative to ensure that student groups are balanced in terms of design preferences and capabilities. To this end, we initiated the process with a survey employing a five-point Likert scale questionnaire [28] among the students, the results of which are detailed in Table 1. This table is designed to evaluate the students’ creative preferences, drawing upon adaptations from a previous work [29]. It assesses individual preferences for creativity in engineering design across four key dimensions. The first dimension, team centrality, evaluates the influence of team dynamics and an individual’s role within the team. The second dimension, risk tolerance, measures how comfortable an individual is with risk and uncertainty in the design process. The third, creative confidence/preference, assesses a person’s self-perceived creativity and their preference for innovative solutions over conventional ones. Lastly, the motivation dimension looks at the level of effort and persistence an individual is willing to invest in creative tasks. Together, these dimensions provide a thorough framework for understanding what drives or hinders creative concept selection in engineering design. In addition, we assessed design capabilities using the grades from two in-class quizzes. We posit a positive correlation between students’ design capability and their quiz scores.

Table 1

Questionnaire for assessing design preferences and capabilities

Team Centrality
I prefer to submit my ideas anonymously
I play a central role in teams that I am a part of
I feel comfortable presenting my ideas to my team members
I believe that I do generate valuable ideas
I believe that I influence the direction and progress of projects that I am a part of
Risk Tolerance
I like dealing with ambiguous or unknown elements in the design process
I believe that risky design concepts will lead to positive design outcomes
I tend to have positive experiences when taking risks during projects
I prefer taking risks during design projects
Creative Confidence
I believe that I am a creative individual
I intend to increase creativity during the design process
I tend to pay more attention to creative ideas
I prefer creative designs over conventional designs
Motivation
I do care about getting good grades
I am not easily discouraged when I am being criticized
It is easy for me to stay focused on the task at hand during design projects
Team Centrality
I prefer to submit my ideas anonymously
I play a central role in teams that I am a part of
I feel comfortable presenting my ideas to my team members
I believe that I do generate valuable ideas
I believe that I influence the direction and progress of projects that I am a part of
Risk Tolerance
I like dealing with ambiguous or unknown elements in the design process
I believe that risky design concepts will lead to positive design outcomes
I tend to have positive experiences when taking risks during projects
I prefer taking risks during design projects
Creative Confidence
I believe that I am a creative individual
I intend to increase creativity during the design process
I tend to pay more attention to creative ideas
I prefer creative designs over conventional designs
Motivation
I do care about getting good grades
I am not easily discouraged when I am being criticized
It is easy for me to stay focused on the task at hand during design projects

We commenced by normalizing all scores to ensure comparability and then calculated the cumulative score of ever team member for each team. Following this, we ranked all teams based on their averaged scores of all team member. To facilitate an equitable distribution, we alternately assigned the ranked groups between two sets. This approach aimed to maintain a balance, not only in terms of individual capabilities and design preferences but also to ensure that each set represented a cross section of the skill and creativity spectrum of the entire class.

3.2.3 ChatGPT Evaluation.

This study was meticulously designed to evaluate the impact of AI-assisted design processes, particularly focusing on the creativity and efficiency improvements introduced by ChatGPT-3.5. Participants were systematically divided into two primary cohorts: the AI group that leveraged ChatGPT-3.5 as their principal tool and the control group that relied on traditional online resources, including general search engines like Google and specialized design-oriented websites. The AI group harnessed the conversational interface of ChatGPT-3.5 to ideate, refine, and troubleshoot design concepts in real time. In contrast, the control group navigated through the more conventional route of information retrieval. Their process involved querying search engines and reading through design-related web pages to gather the necessary data and inspiration for their design tasks.

To gain in-depth insights into the comparative effectiveness of each methodology, participants were requested to document their interaction histories. Those using ChatGPT-3.5 submitted full chat logs, while members of the non-AI group provided the web pages used as references. Supplementing these data, a questionnaire was administered to capture the subjective experiences and strategies employed by each participant throughout the design exploration, as shown in Table 2.

Table 2

Questionnaire for students in concept generation

What is your first thought when using ChatGPT? (for AI group)Ideation
Improvement
Divide the project into several specific design requirements
What is your first strategy in this project? (for non-AI group)Independent ideation
Brainstorm (discussion)
Search reference
In the following aspects, how has ChatGPT improved your design? (for both AI and non-AI groups)Realistic concern
Design complexity
Novelty
Simulations and prototyping
Documentation
Manufacturability
What is the most challenging part during the design? (for both AI and non-AI groups)Ideation
Communication as a team
Search reference
Explain your design in the submission
Understand the design requirements
Time management
What is your first thought when using ChatGPT? (for AI group)Ideation
Improvement
Divide the project into several specific design requirements
What is your first strategy in this project? (for non-AI group)Independent ideation
Brainstorm (discussion)
Search reference
In the following aspects, how has ChatGPT improved your design? (for both AI and non-AI groups)Realistic concern
Design complexity
Novelty
Simulations and prototyping
Documentation
Manufacturability
What is the most challenging part during the design? (for both AI and non-AI groups)Ideation
Communication as a team
Search reference
Explain your design in the submission
Understand the design requirements
Time management

This detailed questionnaire was specifically designed to assess and compare the conceptual design approaches of two distinct student groups: those utilizing AI assistance through ChatGPT (AI group) and those who did not use AI tools (non-AI group). The survey is structured to probe into the initial reactions and strategic choices made by the students at the outset of their projects, as well as the diverse challenges they faced throughout the design process. Additionally, the questionnaire delves into how students across both groups address various aspects of design such as realism, complexity, and manufacturability. By capturing nuanced feedback on the students’ experiences, this questionnaire aims at the tangible impacts of AI integration on student creativity, problem-solving capabilities, and overall project management in the context of engineering design education.

Although the first two questions differ for the two groups, they were both designed to capture the students’ initial reactions and strategic choices during the concept design phase. In the AI group, where ChatGPT was used as a consultant, the goal was to observe how students interacted with the AI and leveraged its capabilities. Specifically, we aimed to see how students approached ChatGPT for assistance. Several possibilities emerged: (1) Students provided all design requirements to ChatGPT, asking it to generate ideas directly; (2) Students had an initial idea and used ChatGPT to refine and improve it to meet design requirements; and (3) Students broke the project into smaller tasks (e.g., tuning, vertical movement) and sequentially sent requirements to ChatGPT.

On the other hand, the non-AI group did not have access to this LLM consultant. Therefore, concept generation was conducted entirely within the group, either independently by each member or through group discussions (brainstorming). Additionally, they may search for examples or references, then generate their concept based on that. The difference in approach between the two groups reflects the distinct methodologies each employed, which is why the first two questions were tailored differently. However, both questions aimed to gauge the students’ preferences when faced with design problems, providing insights into their natural inclinations in problem-solving.

3.2.4 Effect of ChatGPT.

Alongside its utility in aiding the generation of concepts, we are interested in the impact of ChatGPT-3.5 on the final design product. To investigate this, a comparative analysis of final design submissions from the fall sessions of 2023 and 2022 has been undertaken.

This analysis focuses on two specific areas. First, design consistency: we examine whether the final designs align with and originate from the initially submitted concepts. This evaluates the students’ performance in maintaining their design vision from inception to completion, which is critical for the integrity of design projects. Second, design preferences: we assess trends in the types of designs favored by students across these two periods. By identifying shifts in design inclinations, we can infer how ChatGPT-3.5 might influence the design choices of engineering students.

3.3 Detailed Computer-Aided Design Modeling.

Given that ChatGPT-3.5 is an LLM primarily engineered for conversational interactions, it naturally lends itself to ideation processes such as brainstorming. This format effectively facilitates the initial generation of ideas. However, following the brainstorming phase, the next step involves translating these concepts into CAD files, which is part of the core teaching content of MECH 290. We also explored ChatGPT-3.5’s capabilities in producing detailed CAD modeling.

3.3.1 Design Formulation.

In this research, to ensure a fair basis for evaluation, we opted to supply participants with the engineering drawings of a specific object rather than relying on concepts they generated for the detailed CAD modeling phase. The provided object is depicted in Fig. 3. The chosen object incorporates complex features such as curves and lofts, techniques that students have not been exposed to in the course content. This deliberate choice aimed to simulate a scenario where students encounter unconventional geometric designs—ones that they have not yet learned to model in CAD software (solidworks, for this study).

Fig. 3
The object provided for detailed CAD modeling evaluation
Fig. 3
The object provided for detailed CAD modeling evaluation
Close modal

The essence of this exercise was to push students into a situation where they must navigate the process of converting an abstract design idea into a tangible, detailed CAD modeling without prior direct instruction on the specific methods required. Given this gap in their knowledge, students are expected to find external resources for guidance. This search could lead them to traditional avenues such as Google searches or instructional videos that offer step-by-step tutorials on similar projects. However, with the advent of ChatGPT, a new avenue for seeking assistance has emerged. Unlike traditional resources, ChatGPT offers interactive, conversational support, potentially guiding users through the CAD modeling process with personalized advice. This study, therefore, also explored how effectively ChatGPT can serve as a resource for students grappling with the task of realizing unfamiliar designs in CAD software, comparing its utility against more conventional forms of guidance.

3.3.2 Evaluation Process.

In this case study, students were granted the freedom to choose any assistance tool for their homework, recognizing that the authors cannot oversee the specific tools utilized in completing their assignments. Considering that feedback about ChatGPT-3.5’s detailed modeling capability is required, it’s reasonable to hypothesize that students may prioritize ChatGPT-3.5 as their initial go-to resource, seeking alternatives only if ChatGPT fails to provide satisfactory guidance. To assess the effectiveness of guidance in detailed CAD modeling, our evaluation is structured around three critical aspects:

Accuracy of CAD Files: This criterion examines the precision with which the CAD models align with the provided specifications and engineering drawings. It reflects the students’ ability to translate guidance into practical outputs.

Support Questionnaire: A carefully designed survey was designed to capture detailed insights into the types of support students sought during their design process, the sources of this support, and its helpfulness, as shown in Table 3. This questionnaire seeks to understand not just the frequency of ChatGPT’s use but also how it compares with other resources in terms of utility and relevance to the students’ specific needs.

Table 3

Questionnaire for detailed CAD modeling

Please rate how ChatGPT assisted you in your design process in the following aspects.Figuring out solidworks functions to use
Creating a base sketch
Assigning dimensions and constraints
Detailed geometric modeling steps with the solidworks function
Please rate the usefulness of the following tools during your design process.ChatGPT
Official solidworks website (official documentation and tutorials)
Online forums or communities (like Reddit, Stack Overflow)
YouTube (video tutorials and guides)
Discussions with colleagues or peers
Personal experimentation and practice
Please rate how ChatGPT assisted you in your design process in the following aspects.Figuring out solidworks functions to use
Creating a base sketch
Assigning dimensions and constraints
Detailed geometric modeling steps with the solidworks function
Please rate the usefulness of the following tools during your design process.ChatGPT
Official solidworks website (official documentation and tutorials)
Online forums or communities (like Reddit, Stack Overflow)
YouTube (video tutorials and guides)
Discussions with colleagues or peers
Personal experimentation and practice

General Feedback: Beyond the questionnaire, we solicit feedback on the experience of employing ChatGPT-3.5 for detailed CAD modeling. This feedback is necessary for estimating overall satisfaction, pinpointing areas where ChatGPT excels, and identifying other potential feedback from students.

4 Result

4.1 Concept Generation.

In this case study, we required students to document their generated design concepts through sketches, allowing them to submit an unrestricted number of concepts. This approach aimed to foster an environment, where students felt free to explore a wide range of solutions without constraint. Alongside these sketches, we also systematically gathered questionnaires and chat histories. By examining the questionnaires, we can understand the students’ perceptions and experiences with the design process, while the chat histories offer a detailed view of the interactive guidance received. Together, these elements will enable a deeper exploration of how digital tools and resources influence design thinking and concept development among engineering students.

4.1.1 Concept Sketch.

Given the project’s objective to design a pipe inspection robot, we recognized that while there are numerous commercially available designs, the inclusion of specific functionalities—namely, the capability for directional changes and vertical movement—might challenge the full applicability of these pre-existing solutions. This necessitated the generation of innovative concepts by the students, which, although possibly less conventional or reliable, adeptly met the requirements in novel ways.

As detailed in Sec. 3, each group was provided with a 35-minute window to conceptualize and illustrate their design ideas. Following this creative phase, we compiled the sketches and organized them into several distinct categories, as outlined below:

  • Gravity-based: These designs function similarly to ground vehicles, using wheels or tracks on one side to move with the assistance of gravity.

  • Circular: These designs have wheels mounted around a circular frame for movement.

  • Cube: These designs feature a cube-shaped main body with wheels attached to all six faces, allowing for omnidirectional movement.

  • Flying: These designs resemble quadcopters and are capable of flying through the pipe.

  • Hinge: These designs have cylindrical main bodies equipped with retractable legs and hinges at the center for maneuverability.

  • Non-hinge: These designs also have cylindrical bodies with retractable legs but lack hinges, relying on other mechanisms for turning.

  • Composite joint: These designs feature multiple joints and wheels on all sides, allowing the robot to adopt a flexible, zigzag shape for enhanced maneuverability.

  • Suction: These designs use a leg-based suction mechanism to move within the pipe.

  • Worm: These designs also have multiple joints but feature wheels only on one side, mimicking the movement of a worm.

The compilation of design concepts is visualized in Fig. 4. Analysis reveals that most teams not utilizing AI typically formulated three design concepts. In stark contrast, the diversity in the number of concepts developed by AI-assisted teams was more pronounced, with figures ranging from a solitary concept to as many as five. To encapsulate this variability, we employed the median absolute deviation (MAD), a robust metric for assessing variability [30]. MAD is computed as the median of the absolute deviations from the overall median of the dataset, offering insights into the dispersion of the number of design concepts across different teams. Equation (1) for calculating MAD around the median is delineated as follows:
(1)
where Xi represents each value in the dataset, median(X) is the median of the dataset, and |Ximedian(X)| calculates the absolute deviations from the median.
Fig. 4
The compilation of design concepts generated for pipe inspection robot: the distribution of concept generation counts
Fig. 4
The compilation of design concepts generated for pipe inspection robot: the distribution of concept generation counts
Close modal

In our analysis, the MAD values for the AI and non-AI groups were found to be 1.0 and 0.5, respectively. The difference in MAD values highlights a distinct level of consistency in design choices between the two groups. The non-AI group’s design selections tend to cluster around a typical value more closely, while the AI group’s choices show a broader distribution.

In addition to quantifying the number of concepts generated, our study also delved into an analysis of the different types of concepts conceived, with the findings presented in Fig. 5. The sketches are summarized in the following types based on their looks and mechanisms, as shown in Fig. 6. We discovered distinct preferences in the conceptual approaches between the two groups: the non-AI group demonstrated a predilection for designs incorporating car and hinge mechanisms, whereas the AI group exhibited a preference for non-hinge designs. This trend was not merely confined to the initial concept generation phase but was also evident in the final design submissions. This phenomenon, indicating a divergence in design strategy and innovation between the groups, is further explored and discussed in Sec. 4.1.3.

Fig. 5
The compilation of design concepts generated for pipe inspection robot: design type occurrence counts
Fig. 5
The compilation of design concepts generated for pipe inspection robot: design type occurrence counts
Close modal
Fig. 6
Examples of sketch types of pipe inspection robot concepts
Fig. 6
Examples of sketch types of pipe inspection robot concepts
Close modal

4.1.2 Questionnaire Feedback.

We developed two parallel questionnaires tailored to the distinct groups participating in the study. The detailed questions and the feedback received from students are summarized in the following.

The initial query directed at the AI group was about their initial thoughts upon using ChatGPT, whereas for the non-AI group, the question concerned their initial strategy for the project. According to the results as shown in Fig. 7, significantly 73.9% of students in the AI group delegated the ideation tasks to ChatGPT. In contrast, a majority of 84.3% of students in the non-AI group undertook the ideation process themselves, with 76.5% engaging in brainstorming sessions and 7.8% opting for independent ideation. We interpret the high reliance on AI for ideation as analogous to the preference for brainstorming observed in the non-AI group; students naturally gravitate toward easily accessible external resources for support. Talking with ChatGPT or brainstorming with teammates represents tapping into these readily available resources. In comparison, conducting research via Google involves an additional layer of effort to check all search results and extract relevant, actionable insights.

Fig. 7
Questionnaire feedback of initial thought/strategy in the concept generation
Fig. 7
Questionnaire feedback of initial thought/strategy in the concept generation
Close modal

The second question is about which design aspect (in addition to course requirements) is considered during the design generation. As shown in Fig. 8, we identified six key aspects for consideration: realistic concerns, design complexity, novelty, prototyping, documentation, and manufacturability. Notably, the AI group exhibited greater attention to these aspects, despite them not being mandatory for the assignment or contributing to the grading. This heightened consideration was particularly evident in areas such as realistic concerns, design complexity, and manufacturability. We attribute this difference to a shift in roles within the conceptualization process; whereas students in the non-AI group assumed the role of solution providers, their counterparts in the AI group positioned themselves as discerning clients. Consequently, the AI group adopted a more critical stance during concept generation, delegating the task of solving the problem to ChatGPT, rather than tackling it directly.

Fig. 8
Questionnaire feedback of design process concerns and challenges in AI and non-AI groups
Fig. 8
Questionnaire feedback of design process concerns and challenges in AI and non-AI groups
Close modal

The last question explored the most challenging aspects of the design process, including ideation, communication, reference searching, documenting concepts for submission, understanding design requirements, and managing teamwork. As shown in Fig. 8, it was observed that the AI group faced greater challenges in documenting concepts for submission and managing their time effectively. This can be attributed to the nature of responses from ChatGPT-3.5, which are text-based, whereas the required submissions needed to be in sketch format. Thus, the AI group needed to translate textual suggestions into visual representations, a process that did not apply as directly to the non-AI group, who could more straightforwardly sketch out concepts conceived in their minds. Additionally, the challenge of time management for the AI group could be traced back to these additional translation steps, indicating a need for more efficient processes in integrating AI-generated ideas into tangible design submissions.

4.1.3 Effect of ChatGPT in Final Designs.

In addition to the aspects previously discussed, another crucial factor in evaluating a design concept is the consistency between the initial design concept and the final product. The final designs are categorized into four types: hinge-based designs, non-hinge designs, composite joint designs, and non-categorized designs.

  • Hinge-based designs: These designs incorporate hinges that guide the robot’s movement through T-connections in pipes. They are equipped with retractable legs with wheels or tracks that enable the robot to move within vertical pipes using friction.

  • Non-hinge designs: These designs do not include hinges and instead rely on differential steering to navigate turns. Like hinge-based designs, they are also equipped with retractable legs with wheels or tracks.

  • Composite joint designs: These feature multiple joints, allowing the robot to form a flexible zigzag shape. This adaptability enables the robot to adjust to varying pipe diameters, with the leading joint guiding the robot through turns.

  • Non-categorized designs: This category includes designs that do not conform to the specific characteristics of the aforementioned types.

As shown in Fig. 9, the distinction between these categories lies in how each design handles movement, turning, and adaptability within the pipes.

Fig. 9
Examples of four categories of final pipe inspection robot designs
Fig. 9
Examples of four categories of final pipe inspection robot designs
Close modal

Our analysis revealed that within the non-AI group, merely 3 out of 14 teams proposed new concepts in their final design submissions, in contrast to the AI group, where 6 out of 14 teams deviated from their original concepts. Delving deeper into this observation, we discovered that in the non-AI group, two teams moved to hinge-based designs, while one team went to non-categorized designs. Meanwhile, in the AI group, three teams shifted toward non-hinge designs, one team adopted a hinge-based design, and the remaining two teams explored unique, non-categorized designs. This consistency analysis leads to two notable insights. First, there is a tendency among students to abandon AI-generated concepts more readily compared to those conceived independently. This phenomenon could be attributed to the psychological principle of the ”sunk cost fallacy,” which posits that individuals are more likely to continue investing in a project after dedicating significant resources to it [31]. Second, there appears to be a distinct preference for non-hinge designs among the AI group, whereas the non-AI group showed a proclivity for hinge-based designs. This trend was further corroborated by a comparison between the 2022 and 2023 cohorts, details of which will be explored in the subsequent paragraph.

Furthermore, the final designs can be juxtaposed with those from the previous semester. As outlined in Sec. 4.1.1, the spectrum of designs ranges from commercialized options with standard functions (e.g., car and non-hinge mechanisms) to highly customized features (e.g., hinge-based systems and complex joints). Figure 10 illustrates the distribution of designs across these categories. Our analysis reveals that for both the 2023 non-AI group and the 2022 cohort—who also did not utilize AI—designs incorporating hinge mechanisms were overwhelmingly favored. Conversely, in the AI-assisted group, non-hinge designs prevailed, albeit with hinge-based designs also being significantly represented.

Fig. 10
The distributions of final robot designs in different categories
Fig. 10
The distributions of final robot designs in different categories
Close modal

This trend suggests that AI-assisted designs tend to align more closely with commercialized, mainstream designs. Given that AI is trained on extensive existing textual datasets, it is likely to reflect a bias toward well-established designs suited for mass production. While innovative designs are also accessible online, their textual data—found in forums, handbooks, Reddit, etc.,—is considerably less prevalent compared to that of commercial products. Consequently, LLMs may not prioritize these innovative designs in their responses, leading to a predominance of more conventional design concepts among AI-assisted groups. This observation underscores the influence of training data on the output of LLMs and highlights the potential for AI to reinforce existing design norms rather than fostering novel innovation.

We also observed a notable increase in the number of non-categorized designs among the 2023 cohort compared to 2022. Upon examining the chat histories, it became apparent that these distinctive designs predominantly originated from teams employing an “incremental” chat strategy within the 2023 AI group. This “incremental” strategy involves students progressively incorporating additional requirements into their pipe inspection robot discussions. Consequently, ChatGPT responds with a variety of robot features tailored to each new requirement. The synthesis of these robot features results in unique, non-categorized designs. We hypothesized similar influence happened to the 2023 non-AI groups as well, following our recommendation for every team to experiment with ChatGPT-3.5 after the in-class concept generation session.

4.2 Detailed Computer-Aided Design Modeling.

Given that the detailed CAD modeling task does not specify the tools that students must use, our initial step will be to summarize the design accuracy, focusing specifically on the aspects of modeling where students tend to make errors. Following this, we will integrate the accuracy analysis with feedback from the students’ questionnaires to discern the underlying relationship between design accuracy and students’ tool preferences. Lastly, we will summarize the written feedback to identify the key advantages and disadvantages of using ChatGPT during the detailed CAD modeling phase.

4.2.1 Design Accuracy.

As highlighted in Sec. 3.3, the CAD file under discussion was created utilizing “loft” functions with a “combined curve” operation in solidworks. Consequently, the errors encountered can be categorized into three distinct types: (1) wrong function, (2) correct function but with incorrect curve, and (3) other minor design mistakes.

The first type of error, choosing the wrong function, indicates that the student could not discern the correct function required. Identifying the “loft” function is relatively straightforward, as the geometry visibly demonstrates a smooth transition between profiles. However, recognizing the “loft” necessitates an additional step—creating the guide curve, which merges curves from two separate sketch planes. Errors occurring in this phase represent the second type of error. This step presents a greater challenge, as students must deduce that the “loft” function cannot engage two discrete guide curves independently. Instead, they must ascertain the technique to merge these two curves into one and subsequently apply this singular, combined curve.

Additionally, there are other minor oversights such as undefined sketches, incorrect dimensions, and omissions of fillets. Given that these errors are not pertinent to the guidance capabilities of ChatGPT, they have not been extensively discussed here.

We observed that 27.27% of students made no mistakes at all, 6.36% of students failed to recognize the correct function, and 15.45% identified the correct function but did not apply it properly. The rest made minor mistakes. These outcomes reinforce the notion that pinpointing an unfamiliar function is less challenging than effectively employing it, even with ChatGPT’s assistance. We delve into the rationale behind this in the subsequent section.

4.2.2 Questionnaire Feedback.

In the scenario we hypothesized, students attempt to identify a specific function and subsequently learn how to implement it. Therefore, our questionnaire was divided into two sections: (1) assessing the utility of various tools throughout the design process, and (2) evaluating how ChatGPT facilitated the design across different facets. Detailed questions and results are presented in Fig. 11.

Fig. 11
Questionnaire feedback of detailed CAD modeling evaluation
Fig. 11
Questionnaire feedback of detailed CAD modeling evaluation
Close modal

The findings indicate that ChatGPT’s support was not as pronounced in the detailed CAD modeling tasks as it was during the concept generation phase. The first set of questions revealed that the overall impression of ChatGPT-3.5 was moderately positive. Many students found that tutorial videos and a combination of interactive communication with hands-on practice were significantly more beneficial. Meanwhile, official documentation from solidworks and online forums such as Reddit and Stack Overflow were less useful in these highly customized scenarios. The limited utility of official documentation and online forums also sheds light on why ChatGPT’s performance was suboptimal, given that its training data predominantly derives from these sources.

Regarding the second set of questions, we assessed ChatGPT’s performance in four areas: identifying the correct function, creating sketches, assigning dimensions and constraints, and providing detailed, step-by-step guidance for specific functions. The results indicated that ChatGPT offered minimal assistance with sketches, dimensions, and constraints, possibly because these tasks fell within the students’ existing modeling capabilities, reducing the need to consult ChatGPT. ChatGPT received moderately positive feedback for identifying solidworks functions, likely because this information is accessible in function descriptions, video titles, or online forum queries. However, for providing step-by-step guidance on function usage, ChatGPT received moderately negative feedback. This may be due to the specificity of operation steps varying significantly across different objects, with online guidance typically integrating text with visuals for clearer instruction. Despite these limitations, ChatGPT struggled to offer effective step-by-step suggestions for creating complex geometries.

Inspired by other research on ChatGPT [32], we also examined ChatGPT-3.5’s efficacy in generating step-by-step instructions for simpler geometries, such as tables and chairs. The outcomes were notably promising. However, since the students involved already possessed basic modeling skills, they did not require ChatGPT’s assistance for these simpler tasks. We chose not to include these basic modeling tasks in our study, and we encourage readers to consult related publications for more details.

4.2.3 General Feedback.

The student feedback on using ChatGPT-3.5 for detailed CAD modeling tasks in solidworks reveals several benefits:

  • Clarity in Instructions: A notable strength of ChatGPT is its ability to provide clear, step-by-step instructions for specific operations in solidworks, enhancing task guidance.

  • Understandable Language and Format: ChatGPT stands out for its use of comprehensible language and formatting, which demystifies complex instructions and makes technical information more approachable.

  • Rapid Responses: An appreciated feature is ChatGPT’s promptness in offering responses, offering a quick albeit sometimes cursory starting point for deeper exploration.

However, several limitations of ChatGPT in supporting 3D modeling tasks were also identified:

  • Limited Utility in 3D Modeling: ChatGPT’s effectiveness in 3D modeling support is constrained, primarily due to its textual nature and inability to process or visualize intricate geometrical configurations.

  • Dense Text Responses: The density and complexity of ChatGPT’s text-based responses can pose understanding challenges, rendering it less effective than visual instructional resources, such as YouTube tutorials.

  • Questionable Reliability: Instances of unreliable responses from ChatGPT have led to concerns over its dependability, occasionally misguiding users and leading to time inefficiencies.

  • Ambiguous Guidance: ChatGPT often presents advantages and disadvantages without definitive recommendations, which can be problematic for novices in need of explicit direction.

  • Text-Based Limitations: The intrinsic shortcomings of a text-based AI, like ChatGPT, in comprehending and assisting with the visual and complex tasks inherent to solidworks, are significant.

While the introduction of ChatGPT-4 has expanded capabilities to include image processing, preliminary trials indicate its limited effectiveness in interpreting engineering drawings or solidworks screenshots to offer relevant advice. This limitation likely stems from the scarcity of engineering design and modeling data within ChatGPT’s training dataset. Consequently, foundational models such as ChatGPT cannot fully assume the role of a designer in complex scenarios without further fine-tuning and specialization to address these specific challenges.

5 Limitations

While our study demonstrates the potential of using ChatGPT 3.5 for guiding mechanical engineering design, several limitations should be acknowledged.

Our work focuses on the impact of ChatGPT 3.5 on engineering design education, but the model’s internal architectures and training datasets remain unclear due to its proprietary nature. This lack of transparency makes it difficult to fully understand how the model generates its responses and to propose corresponding model customization suggestions. Additionally, our evaluation of the design suggestions generated by ChatGPT was qualitative, focusing on the relevance and creativity of the outputs rather than on quantitative performance metrics. More rigorous, quantitative evaluations are needed to assess the effectiveness and reliability of LLMs in design settings.

OpenAI has released multiple multimodal versions of ChatGPT that integrate text, images, and other data types. Some of the limitations identified in GPT-3.5, such as the model’s lack of reasoning capability through complex tasks or inability to handle non-textual inputs, have been overcome in these newer versions. However, due to time constraints, our study did not explore the capabilities of these multimodal models, and their impact on the design process remains an open area for future investigation.

While we focused on the generalized reasoning capabilities of ChatGPT—which should be able to generalize to other LLMs—we did not conduct tests on other LLMs in this work. The performance and applicability of other LLMs (e.g., LLaMA, Mistral) in engineering design have not been empirically tested. Further research is needed to understand any model-specific nuances that could affect the outcomes.

6 Conclusions

In this work, we explored the integration of ChatGPT into the design process of engineering students, focusing on concept generation, and detailed CAD modeling tasks. The findings reveal distinct patterns in how AI and non-AI groups approach design challenges, with AI assistance significantly influencing ideation, concept development, and the evaluation of design aspects beyond basic requirements.

The case study of concept generation revealed a clear distinction between AI-assisted and non-AI groups in terms of concept generation diversity and design preferences. Teams utilizing AI assistance demonstrated a wider range of design concepts, as evidenced by the higher MAD values, indicating a broader exploration of potential solutions. The AI group also showed a heightened focus on various design aspects such as realism and manufacturability, adopting a more critical perspective by acting as discerning clients rather than direct solution providers. Challenges for the AI group included translating textual suggestions into sketches and efficient time management. The comparison of final design products pointed to a divergence in final design selections between groups, with a notable shift toward non-hinge designs among AI-assisted teams. This trend might reflect AI’s influence in promoting more commercially viable or established design concepts, potentially limiting the exploration of more novel or unconventional ideas. Additionally, the observed rise in non-categorized designs indicates an inventive application of ChatGPT in blending diverse requirements with established technologies.

In a detailed CAD modeling study focusing on solidworks, analysis of the submitted CAD files revealed that the primary challenge for students lies not in identifying functions, but in their application, particularly for complex tasks such as merging curves. This highlights how AI tools such as ChatGPT could help students with CAD modeling by suggesting possible functions they might use, instead of giving detailed step-by-step guides. Students appreciated ChatGPT for its clarity, easy-to-understand instructions, and quick responses. However, they highlighted that its usefulness was more limited in detailed CAD modeling tasks compared to its effectiveness during the concept generation phase. The reliance on text-based instruction posed challenges, especially when compared to more intuitive visual resources like tutorial videos. Additionally, the occurrence of unreliable responses emphasized the limitations of relying on ChatGPT as a tool for 3D modeling assistance, particularly for learners. This feedback indicates that while ChatGPT can provide significant support in some aspects of the design process, it cannot fully substitute for traditional learning materials or the depth of understanding cultivated through direct, hands-on practice. However, with more customized LLMs—both in terms of model architecture and training data–it becomes possible to provide more reliable feedback, or even assign a confidence score to each suggestion during the learning process. Such enhancements could increase the reliability of AI tools in CAD modeling, offering students more dependable guidance and reinforcing their confidence in the tool’s suggestions.

In conclusion, integrating ChatGPT into engineering design education presents both opportunities and challenges. Our study illustrates that while AI assistance enriches the ideation and concept generation phases with a wider array of design concepts, it also introduces bias for existing popular designs. Particularly in the scenario of CAD modeling, the limitations of current AI tools become evident, emphasizing the necessity of supplementary traditional learning resources and hands-on practice. Moving forward, the goal should be to leverage AI not as a replacement but as an enhancement to the conventional design process, ensuring that students are equipped with both the innovative mindset and the practical skills required to navigate the evolving landscape of engineering design.

Acknowledgment

This research is supported by the McGill Engineering Doctoral Awards (MEDA).

Author Contribution Statement

Chonghui Zhang: conceptualization, methodology, visualization, writing — original draft; Yaoyao Fiona Zhao: supervision, project administration, funding acquisition, writing — review, and editing.

Conflict of Interest

There are no conflicts of interest.

Data Availability Statement

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.

References

1.
Viswanathan
,
V.
,
Tomko
,
M.
, and
Linsey
,
J.
,
2016
, “
A Study on the Effects of Example Familiarity and Modality on Design Fixation
,”
AI EDAM
,
30
(
2
), pp.
171
184
.
2.
Nishimoto
,
K.
,
Sumi
,
Y.
, and
Mase
,
K.
,
1996
, “
Toward an Outsider Agent for Supporting a Brainstorming Session–An Information Retrieval Method From a Different Viewpoint
,”
Knowl.-Based Syst.
,
9
(
6
), pp.
377
384
.
3.
Buzan
,
T.
, and
Buzan
,
B.
,
1996
,
How to Use Radiant Thinking to Maximize Your Brain’s Untapped Potential
,
Dutton
,
New York, NY
.
4.
Altshuller
,
G.
,
Shulyak
,
L.
,
Rodman
,
S.
, and
Fedoseev
,
U.
,
2002
, “40 Principles: Triz Keys to Innovation,” Technical Innovation Center.
5.
Jin
,
X.
, and
Dong
,
H.
,
2020
, “
New Design Heuristics in the Digital Era
,”
Proceedings of the Design Society: DESIGN Conference
,
Cavtat, Croatia
,
Oct. 26–29
, Cambridge University Press, pp.
607
616
.
6.
Sarica
,
S.
,
Luo
,
J.
, and
Wood
,
K. L.
,
2020
, “
Technet: Technology Semantic Network Based on Patent Data
,”
Exp. Syst. Appl.
,
142
(
1
), p.
112995
.
7.
Sarica
,
S.
,
Song
,
B.
,
Luo
,
J.
, and
Wood
,
K. L.
,
2021
, “
Idea Generation With Technology Semantic Network
,”
AI EDAM
,
35
(
3
), pp.
265
283
.
8.
Sangelkar
,
S.
, and
McAdams
,
D. A.
,
2017
, “
Automated Graph Grammar Generation for Engineering Design With Frequent Pattern Mining
,” In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol.
58127
,
American Society of Mechanical Engineers
, p.
V02AT03A006
.
9.
Arslan
,
Y.
,
Allix
,
K.
,
Veiber
,
L.
,
Lothritz
,
C.
,
Bissyandé
,
T. F.
,
Klein
,
J.
, and
Goujon
,
A.
,
2021
, “
A Comparison of Pre-Trained Language Models for Multi-Class Text Classification in the Financial Domain
,”
The Web Conference 2021
,
Ljubljana, Slovenia
,
Apr. 19–23
.
10.
Kublik
,
S.
, and
Saboo
,
S.
,
2022
,
GPT-3
,
O’Reilly Media, Inc
,
Sebastopol, CA
.
11.
Ali
,
S. R.
,
Dobbs
,
T. D.
,
Hutchings
,
H. A.
, and
Whitaker
,
I. S.
,
2023
, “
Using ChatGPT to Write Patient Clinic Letters
,”
Lancet Digit. Health
,
5
(
4
), pp.
e179
e181
.
12.
Wong
,
I. A.
,
Lian
,
Q. L.
, and
Sun
,
D.
,
2023
, “
Autonomous Travel Decision-Making: An Early Glimpse Into ChatGPT and Generative AI
,”
J. Hospit. Tourism Manage.
,
56
(
1
), pp.
253
263
.
13.
Dai
,
Y.
,
Liu
,
A.
, and
Lim
,
C. P.
,
2023
, “
Reconceptualizing ChatGPT and Generative AI as a Student-Driven Innovation in Higher Education
,”
Proc. CIRP
,
119
(
1
), pp.
84
90
.
14.
Amaro
,
I.
,
Della Greca
,
A.
,
Francese
,
R.
,
Tortora
,
G.
, and
Tucci
,
C.
,
2023
, “
AI Unreliable Answers: A Case Study on ChatGPT
,”
Artificial Intelligence in HCI
,
Copenhagen, Denmark
, Springer, pp.
23
40
.
15.
Pavlik
,
J. V.
,
2023
, “
Collaborating With ChatGPT: Considering the Implications of Generative Artificial Intelligence for Journalism and Media Education
,”
J. Mass Commun. Educ.
,
78
(
1
), pp.
84
93
.
16.
Kung
,
T. H.
,
Cheatham
,
M.
,
Medenilla
,
A.
,
Sillos
,
C.
,
De Leon
,
L.
,
Elepaño
,
C.
,
Madriaga
,
M.
, et al.,
2023
, “
Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models
,”
PLoS Digit. Health
,
2
(
2
), p.
e0000198
.
17.
Baidoo-Anu
,
D.
, and
Ansah
,
L. O.
,
2023
, “
Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning
,”
J. AI
,
7
(
1
), pp.
52
62
.
18.
Bernabei
,
M.
,
Colabianchi
,
S.
,
Falegnami
,
A.
, and
Costantino
,
F.
,
2023
, “
Students’ Use of Large Language Models in Engineering Education: A Case Study on Technology Acceptance, Perceptions, Efficacy, and Detection Chances
,”
Comput. Edu.: Artif. Intell.
,
5
(
1
), p.
100172
.
19.
Tsai
,
M.-L.
,
Ong
,
C. W.
, and
Chen
,
C.-L.
,
2023
, “
Exploring the Use of Large Language Models (LLMs) in Chemical Engineering Education: Building Core Course Problem Models With Chat-GPT
,”
Edu. Chem. Eng.
,
44
(
1
), pp.
71
95
.
20.
Liu
,
R.
,
Zenke
,
C.
,
Liu
,
C.
,
Holmes
,
A.
,
Thornton
,
P.
, and
Malan
,
D. J.
,
2024
, “
Teaching CS50 With AI: Leveraging Generative Artificial Intelligence in Computer Science Education
,”
SIGCSE 2024: The 55th ACM Technical Symposium on Computer Science Education
,
Portland, OR
, Vol. 1, pp.
750
756
.
21.
Cotton
,
D. R.
,
Cotton
,
P. A.
, and
Shipway
,
J. R.
,
2024
, “
Chatting and Cheating: Ensuring Academic Integrity in the Era of Chat-GPT
,”
Innovat. Edu. Teach. Int.
,
61
(
2
), pp.
228
239
.
22.
Kim
,
D.
,
Kim
,
T.
,
Kim
,
Y.
,
Byun
,
Y.-H.
, and
Yun
,
T. S.
,
2024
, “
A ChatGPT-MATLAB Framework for Numerical Modeling in Geotechnical Engineering Applications
,”
Comput. Geotechn.
,
169
(
1
), p.
106237
.
23.
Tian
,
J.
,
Hou
,
J.
,
Wu
,
Z.
,
Shu
,
P.
,
Liu
,
Z.
,
Xiang
,
Y.
,
Gu
,
B.
,
Filla
,
N.
,
Li
,
Y.
,
Liu
,
N.
, and
Chen
,
X.
,
2024
, “Assessing Large Language Models in Mechanical Engineering Education: A Study on Mechanics-Focused Conceptual Understanding,”
arXiv preprint
. https://arxiv.org/abs/2401.12983
24.
Zheng
,
C.
,
Liu
,
Z.
,
Xie
,
E.
,
Li
,
Z.
, and
Li
,
Y.
,
2023
, “Progressive-Hint Prompting Improves Reasoning in Large Language Models,”
arXiv preprint
. https://arxiv.org/abs/2304.09797
25.
Wei
,
J.
,
Wang
,
X.
,
Schuurmans
,
D.
,
Bosma
,
M.
,
Xia
,
F.
,
Chi
,
E.
,
Le
,
Q. V.
, and
Zhou
,
D.
,
2022
, “
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
,”
Adv. Neur. Inf. Process. Syst.
,
35
(
1
), pp.
24824
24837
.
26.
Pahl
,
G.
,
Beitz
,
W.
,
Feldhusen
,
J.
, and
Grote
,
K.-H.
,
1996
,
Engineering Design: A Systematic Approach
, Vol.
3
,
Springer
,
London UK
.
27.
ElMaraghy
,
W.
,
ElMaraghy
,
H.
,
Tomiyama
,
T.
, and
Monostori
,
L.
,
2012
, “
Complexity in Engineering Design and Manufacturing
,”
CIRP Ann.
,
61
(
2
), pp.
793
814
.
28.
Rattray
,
J.
, and
Jones
,
M. C.
,
2007
, “
Essential Elements of Questionnaire Design and Development
,”
J. Clin. Nurs.
,
16
(
2
), pp.
234
243
.
29.
Toh
,
C. A.
, and
Miller
,
S. R.
,
2016
, “
The Preferences for Creativity Scale (PCS): Identifying the Underlying Constructs of Creative Concept Selection
,” International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol.
50190
,
American Society of Mechanical Engineers
, p.
V007T06A013
.
30.
Pham-Gia
,
T.
, and
Hung
,
T. L.
,
2001
, “
The Mean and Median Absolute Deviations
,”
Math. Comput. Modell.
,
34
(
7–8
), pp.
921
936
.
31.
Arkes
,
H. R.
, and
Blumer
,
C.
,
1985
, “
The Psychology of Sunk Cost
,”
Organ. Behav. Human Decis. Process.
,
35
(
1
), pp.
124
140
.
32.
Makatura
,
L.
,
Foshey
,
M.
,
Wang
,
B.
,
HähnLein
,
F.
,
Ma
,
P.
,
Deng
,
B.
,
Tjandrasuwita
,
M.
,
Spielberg
,
A.
,
Owens
,
C. E.
,
Chen
,
P. Y.
, and
Zhao
,
A
,
2023
, “
How Can Large Language Models Help Humans in Design and Manufacturing?
,”
arXiv preprint arXiv:2307.14377
. https://arxiv.org/abs/2307.14377