Roles: 

Interaction Developer, Gameplay Developer, Experience Designer

​

​

Platform: HTC VIVE & PC |  Engine: Unity 3D 

   Team: 6 person |  Language: C# JSON  |  Time: 14 Week

​

Brief Intro:

        In the Semester-long project in the spring of 2019, I worked with five other teammates to create a cross_platform tutorial prototype. The project's goal is to apply immersive technology to the lab to create an efficient, safe and immersive teaching experience.

        Our project is based on EOS290, a metal 3D printer. The machine has high requirements on the surrounding environment, the setup process is complicated, and there are serious safety hazards due to improper operation.

        The target user group are the machine users who completed the initial training course in the lab, and our project will reinforce the knowledge they acquired on traditional media tutorials and ensure they can operate the machine individually after completing our training. So we developed the linear progress starts with the safety instructions and leads the user through the entire set up process until the machine enters the print mode.

        The PC version is based on WebGL, focuses on safety instructions and a general overview of the machine through text guides and quizzes.

        Based on the Uniy3D engine and Steam VR, the VR version focuses on getting the player familiar with the machine's operating procedures and machine parts functions.

​

Personal Contribution:
        Although the project is divided into two platforms, VR and PC, they share the same development structure. The entire system can be divided into three parts: the back-end data system, the interactive system, and the level implementation. I am mainly responsible for implementing the overall level structured. I also evolved in developing interaction and visualization systems.

        In addition, I also participated in interactive design and visual design. From a programmer's perspective, participating in process design allows programmers to develop results that are more related to the design pattern, and suggestions from developer side for designers can also help them better polish and iterate design maps.

​

​

Project Go through:

​

~BrainStorming

    After communicating with the client at the beginning of the project, we first entered Lab to learn about the 3D printer and the printing process.

​

    After observing the explanation of the machine flow in the lab, we have raised the following questions worth considering:

1.     In the operation of the machine, what are the angles in the traditional media tutorial that cannot be covered?

2.    What difficulties did the professor encounter in guiding the use of the machine?

3.    Problems in these processes. If Immersive technology is applied, how can we provide a more reasonable solution?

4.    What traditional lessons can be applied to immersive technology applications in traditional media teaching?

​

~Design

        Based on the above questions, we have extracted the following user pain points (priority with severity):

1.     The operating environment of the machine and its precision, so the environment for teaching is very strict, and the teaching experience in the lab has many inconveniences.

2.     Minor operational errors can cause serious security incidents, so the teaching process lacks a repetitive virtual experience.

3.    Due to the limitations of time and traditional media, it is difficult for students to associate the terminology of the parts in the machine with the actual model, so that the fusion will occur when the student is learning further.

4.    The teaching process lacks inspection and inspection links.

5.    The course must be completed in the lab and the number of hours per week is extremely limited

6.    Users can't match the operation in the tutorial with the actual operation when reading the paper media and watching the instructional video.

​

        Therefore, we decided to use Reality Immitation platforms to develop a rigorous, rigorous and fully functional educational application, including overview, practice, inspection and unsupervised operations.

        We want the simulation lab to create a repeatable, secure and relatively free application environment for our users, so we use VR heads and controllers for prototyping.

​

~Proposal review

        After identifying the VR platform and the plans developed during the semester, we and the Client gave feedback. After considering the time scope, we decided to implement the overview and inspection on the PC platform, and maximize the immersive operation experience on the VR platform to present a satisfactory and complete machine operation process.

        In addition, we also got quite a lot of valuable reference materials from the client. Although the high-precision model of the machine is not provided, we have obtained a large number of paper media, video and reference photos that were previously used for teaching; and we have obtained many useful materials and professors' verbal details by entering the lab many times.

​

~Technical research

       Since I am mainly responsible for the development of the VR platform, I started to refer to other VR tutorial in the industry. Because the experience of reading large texts in the VR experience is not satisfactory, I decided to use the handheld tablet to display the short step title and summary in VR. The separate content of each step is provided to the student on the PC side. 

        To compensate for the lack of textual information, I consider using a more intuitive visualization approach to provide instruction. We try to use the outline and highlight to mark the interactable objects and use the floating tag to label the name of each component.

        After consulting the data, we think that the setup process can be divided into several large steps and dozens of small steps. We want to use a stable back-end system to store the entire process and easy to maintain. The other programmer in the group proposes to use JSON file as the backend file, store the information in the scene, and store the information of each step according to the following structure.

        In addition, we looked at the latest APIs from Steam VR and tried out the new Interaction System. This series of scripts allows developers to set up clicks, grips, etc. through the handles more quickly and freely. Based on this series of interaction templates, we have written how to interact with different parts of the printer.

 

~Low-Fi Prototyping

        Based on a different part of the experience, the three of us started the low-fi prototype. They tried to implement functions such as file reading on the back end and object lookup in the scene, and I started to implement the interactive content of the machine. Due to the complex internal structure of the machine, I divided the overall interactive content into the following categories:

        After implementing the interaction of the main structures in the machine, we selected a more representative step in the overall process: blade installation was implemented. We store the contents of step 5 in the backend file and show the linear structural process on the placeholder model of the chamber. We hope that the low-fi prototype can give our client and advisor a brief draft about the system we developed.

        After getting the feedback, we modified our system according to the feedback, and follow the steps to develop and iterate the interaction, backend, and scene.

~Consecutive playtests

        At the weekly client meeting and faculty meeting, we will show the progress of this week's development to ensure that our products do not deviate from their requirements. We also invite students and faculties to stop by to test our prototype. This process provides us with a lot of valuable naive user feedback.

        Since our product target user is a user with a certain knowledge of the printer, we will get targeted feedback based on the user's identity when we perform user test. If the user does not have printer relative knowledge, we would like them to give feedback on whether the interaction system is easy to operate, and whether the construction system is intuitive. If the user is the future user of the printer, we want them to provide feedback on the process itself to ensure that future users can indeed gain knowledge and training from the process.

~Hi-Fi prototyping

        In the 14-week development process, we continually iterate over the user experience of the tutorial and the entire printer setup process. In addition to installing the components to the machine, the press button, and debugging the instrument, we also simulated the operational details of some steps, such as Rubbing the lens and tamping the powder, we hope that this kind of realistic interaction allows the user to form a muscle memory that is practical in the lab.
In addition, we have redesigned the VISual Instruction system. We added arrow animation instructions for some complex operations and added highlights and outlines for the interactive components of each step.

        We also added interactive feedback for handheld screens, buttons and displays on the machine. In the VR experience, the use of visual and audio feedback can effectively improve the player's interaction efficiency, so we set the different state relationships for these components to ensure that users have instant feedback in the hover, near, press, hold state.

~Hi-Fi User test

        With the continuous improvement of our prototype, our test target is not limited to our advisor, client and etc classmates. We started to try more professional engineers to test our prototype. The client has provided us with several valuable testers, all of whom are Ph.D. students who are familiar with the machine and have participated in the printer course. They used their knowledge of the machine to complete the setup process simulation very well and were very satisfied with our interactive system.

        In addition, because the client is very satisfied with our results, we have the opportunity to participate in the manufacturing futures initiative forum on the main campus to showcase our work. Nearly one hundred industry insiders including CMU Dean came to participate in the day. There are also quite a few companies interested in Immersive technology application in education. They are very interested in our concepts and prototypes, and also presented valuable feedback for our process experience. 

~Presentation

        As a student of ETC, we also need to show our cooperation results in one semester through the final report. Therefore, we also carefully prepared the presentation material and ensure that the presentation can show our semester output and knowledge acquired within the specified time. Fortunately, our client also came to the presentation site and provided support and positive feedback for our presentation, so ETC's faculties are generally satisfied with our project.

​

~Further Insights

        Objectively, our project completed most of the Client's requirements. But due to time constraints, we don't have time to go deeper with evaluation and Freeplay experience development. In addition, although we collect a lot of valuable user data, we still need to analyze, visualize and further feedback them on the project iteration.

        As a tutorial, I think we don't have enough energy in the test and review part, so we should spend more time in the Linear experience to investigate and repeat the emphasis on complex operations to ensure that users have a deeper memory of these links.

~Challenges and Lessons

        This is my first time at ETC to participate in the semester-long project. There is no doubt that we have encountered friction and confusion in the project process since balancing the requirements of the client and our vision is indeed a matter that takes time to master. But from the technical or research perspective, I have gained a lot of thoughts and attempts in the project to implement interactions in the immersive environment. The 14-week development cycle has also made my understanding of Unity and SteamVR deeper. 

        ETC faculties often remind us that the subtle teamworking skills, scheduling skills, and communication skills are more beneficial lessons learned in the project. After a semester of teamwork, I enjoyed this process of collaboration and innovation within the team. I also found my career target in the semester: VR/AR/MR experience developer, I think this project provided me with irreplaceable experience and guidance.
 

​