Learning new languages can be a hectic and time-consuming process. It involves a lot of study and requires putting in an ample amount of time and effort. Often, the process can be boring as foreign words are unrelatable and need repeated practice to memorize. To make the learning process more intuitive, Rensselaer Polytechnic Institute (RPI) situated in Troy, New York has developed a classroom that uses AI and VR to teach Mandarin Chinese.
The AI & VR powered classroom is made in collaboration with IBM Research. The classroom has a 360° virtual environment based on a virtual Beijing street. The learning process is incorporated by making the students order food and interacting with street vendors inside the environment.
The people behind this project believe that the easiest way to learn a language is to get into an environment where the target language is being spoken natively. Since moving into a new country just to learn a new language is obviously not feasible, this classroom provides a similar experience.
Responses to the student queries are provided in real-time, thanks to the AI. The idea for the project originated from two faculty members from the institute who used role-playing games for the students to get a better understanding of the conversation.
The classroom makes use of several sensors to adapt to the behavior of students to provide a better learning experience based on their grasping abilities. The image below shows a student interacting with the system using the mic for voice input and camera for tracking the body movements.
Interacting with the environment is as simple as asking a question. To know more information about an object, the student just needs to point to it and ask a question like “What is this?“ and the AI will provide relevant and satisfying answers.
The platform does not miss-out on improving the pronunciation of students. It provides audio and visual feedback to the students to guide them to pronounce it correctly. The algorithms incorporated in the environment also compares the student’s speech with native speakers and shows how the pronunciation of the student differs from the native speakers.
The classroom is highly experimental at this moment and will be implemented for the first time this year as a six-week course at RPI. This year, the course will use the virtual environment for half the duration while traditional classes will be provided for the rest of the time. However, this is subject to changes if the virtual environment turns out to be efficient and effective to the students. So, what do you think of these next-gen immersive classrooms? Let us know in the comments.