Bringing Mixed Reality Generative AI To Frontline Workers
At the recent Microsoft Ignite 2023 conference Copilot in Microsoft Dynamics 365 Guides was announced which combines the power of generative AI with mixed reality to help frontline workers complete complex tasks and resolve issues faster with less disruption to the flow of work.
Copilot in Dynamics 365 Guides assists workers in industrial settings who deal with complex equipment and processes. New technicians need to ramp up quickly while learning how to operate and service that equipment. When equipment stops functioning, workers must act quickly to diagnose the problem, implement a fix, and then document their work. Instead of workers needing to search through volumes of digital documents or paper manuals, Copilot enables workers to ask for relevant information using natural language and human gestures, such as looking and pointing. Copilot uses generative AI to search for information from technical documentation, service records, training content, and any other data sources that customers curate. Copilot then summarizes the information to provide interactive guidance through content and holograms overlaid on the equipment in operation or maintenance.
Mixed reality together with AI presents a human-first interface that promises to transform real-world operations. Copilot can assist industrial workers in real-time with speech and holograms.
Point and Ask Guidance
Humans have always learned from and interacted with their environment in a hands-on manner, asking questions of others as they work. Copilot in Dynamics 365 Guides builds on that natural way of interacting to help workers better understand the world around them. Workers can point to a component on an asset and ask Copilot questions, such as “What is the torque limit for these bolts?” New workers can ask general questions, such as “What is the function of this component?” without needing to interrupt a more experienced coworker. Copilot recognizes what the user is pointing at and provides answers. 3D models and spatial maps provide the foundation for the technology to identify what the worker is looking at and then project a hologram to show instructions. Workers can ask more complex questions, such as “Which direction does this part fit?” or “Show me where the sensor module is on this machine,” and Copilot will display holograms to show where the part belongs.
Spatial Content Generation
Copilot in Dynamics 365 Guides accelerates the out-of-the-box value of mixed reality by significantly reducing the upfront investment needed to create spatial content. Copilot simplifies the creation, management, and delivery of mixed reality content using customer-defined data sources, enabling companies to adopt mixed reality much faster.
Powered by Microsoft Azure OpenAI Service, large language models allow Copilot in Dynamics 365 Guides to do far more than answer one-off questions. Workers can have progressive conversations, such as “Walk me through the steps to disassemble this filtration unit,” or “I disconnected the motor, what’s next?” Copilot follows along with the process and conversation using customer’s curated knowledge base to articulate answers with speech, text, or 3D holograms, eliminating ambiguity and reducing the need for rework.
On Demand Information
With connections to the Internet of Things (IoT) sensors, operational data, and service records, Copilot in Dynamics 365 Guides can answer questions, such as “What is the current humidity rating?” or “When was this last inspected, and were there any concerns?” HoloLens 2 allows Copilot to pinpoint specific assets, identify them, and access relevant information sources in real time. Frontline workers can access rich information to troubleshoot or repair with AI assistance in the context of their work.
Expert Knowledge Integration
Often only a more experienced worker has the knowledge required to complete a complex task or repair a vital piece of equipment. When that worker is unavailable or retires, that knowledge is lost. Copilot in Dynamics 365 Guides can leverage notes or call transcripts from previous expert activity to guide new workers through the same process. This makes it easier and faster for workers to learn from each other and automate internal knowledge transfer—bolstering workforce productivity and business agility.
Streamlined Documentation
Workers will be able to use Copilot in Dynamics 365 Field Service mobile to speed up the process of completing and documenting their work. Instead of filling out paperwork, workers can simply describe their progress to Copilot using their voice. Copilot will provide suggestions to efficiently check off service tasks, add notes, and update product quantities as well as statuses, accelerating data entry.
Copilot in Dynamics 365 Guides transforms the real world into a prompt and an anchor for AI. Keeping human agency at the center of the product, this tool radically transforms how computing technology can help frontline workers take action in the real world. Copilot in Dynamics 365 Guides holds significant potential for businesses seeking greater efficiency and effectiveness in their operations.
For more information: www.microsoft.com
