Build low-code mixed-reality apps in Power Apps, AI Builder and HoloLens

hololens2building-model.jpg

Picture: Microsoft

Microsoft’s low-code Power Platform is a key piece of its business software suite. By sitting between Microsoft 365’s productivity tools and the Dynamics 365 line-of-business applications, it’s a way to build out custom workflows and add your own user experiences. As an added bonus you get access to Azure’s Cognitive Services for AI as a service and connectors that link to third-party applications.

The result is a quick way to roll out code as it’s needed, filling what’s often referred to as the “enterprise app gap.” The Power Platform’s tooling is ready for use by both users who need an app, but there’s nothing there to fill that need, and professional developers who need to roll out a solution quickly. Technologies like the PowerFX language and Visual Studio Code support make it easy to bring the Power Platform into traditional development toolchains, supporting the development of cross-disciplinary and cross-functional fusion teams to design, build and deploy apps.

EES: Windows, Linux, and Mac commands everyone needs to know (free PDF) (TechRepublic)

If you’ve built an app in Excel or in Access, you can build one in Power Platform, especially using its Power Apps tools. There are three ways to build Power Apps apps: using Canvas apps to drag and drop controls onto a design surface, attaching code to them, or using model-driven apps to go straight from data to code. Canvas apps give you the most flexibility, as they can be used to build a wide range of different types of app using many different types of data. They’re also how you can host mixed reality in your low-code app.

Power Platform in the metaverse

One technology that’s getting a lot of interest at the moment is mixed reality or, as it’s becoming known, the metaverse. Microsoft has a lot of experience in using 3D content in virtual and augmented reality (VR/AR), with tooling built into Windows and Azure, as well as mixed reality content support for Android and iOS devices using their augmented reality SDKs (software development kits ).

Microsoft’s initial experiments with enterprise mixed reality were support tools, offering first-line workers access to real-time support with fellow workers looking through their eyes (or at least their HoloLens camera) and annotating the world for them or providing life-size examples of equipment for training and planning.

Those concepts are very much part of how Microsoft approaches mixed reality in Power Apps—as a tool rather than a gimmick. That’s not to say they can’t be used as part of a marketing app, but there’s a lot more to how Microsoft sees the technology.

The benefit, of course, is that using Power Apps reduces the expertise needed to build a mixed reality application. As a result, instead of having to learn Unity or a similar 3D development environment, all you need is a Canvas app and the right controls. That makes building a prototype a matter of minutes, no matter what devices you’re targeting.

Adding mixed reality to a Power Apps application

The Power Apps mixed reality controls build on work from the Babylon.js team. A tool for building 3D images on the web using JavaScript, it’s now been integrated with React Native for use on mobile devices outside the browser. Microsoft was already using the React Native framework to deliver the Power Apps Mobile controls, so using it to deliver mixed reality content made a lot of sense.

Work had to be done to integrate Babylon’s Native release with React Native as well as with iOS’s ARKit and Android’s ARCore. For PCs using the Power Apps web tools, there’s integration with the WebXR toolkit.

While you don’t need to understand the back end to build Power Apps mixed reality, the resulting controls support key features your apps need, including providing anchors for views as well as hit tests and plane detection. This now means you can test 3D mixed reality content using Babylon on PCs before moving it into Power Apps and using it on devices.

EES: Hiring kit: Backend Developer (TechRepublic Premium)

Using the controls is relatively simple. Content is stored in your application back end and loaded as needed, where it is displayed using the View in MR Power Apps control using the dimensions you’ve set. Your device becomes a viewport into a mixed reality world, placing objects on a floor or on a wall for you to move around.

You don’t want to use this tool all the time; it’ll most likely be a way of taking an item from a catalog and showing it where it might be used. Maybe it’s a piece of furniture in a home, or maybe it’s industrial equipment on a factory floor. Users click a button in your app to launch the mixed reality view, replacing the view they were using.

Bringing Power Apps, device sensors and AI together

As well as using Power Apps to add items to augmented reality, you can use mixed reality tools to measure the environment around them. This can be useful if you’re building an app to help calculate building costs.

Adding the Measuring Camera control allows users to pick a start point and then pan to measure a space, with an overlay showing the current measurements. This uses the built-in depth sensors in most modern phones; you’ll get the best results if a phone has LIDAR or a similar accurate sensor. Mixing the measurement tools with the view control can help with planning equipment installation, showing where something can fit safely.

Things get even more interesting when you mix these tools with Power Apps AI Builder. Here, you get low-code access to Azure’s Cognitive Services, mixing prebuilt models with custom capabilities.

You can use the Power Apps camera controls and its object recognition features to, for example, detect which version of a piece of equipment needs maintenance, automatically loading the right mixed reality model for comparison. The app can then launch the appropriate Dynamics 365 Guides to aid in maintenance, using an AR phone, or more usefully a hands-free HoloLens. One sample from Azure’s developer advocacy team shows how you can use the tools to identify different hardware.

Building the future of application development with Power Apps

Building complex mixed-reality apps like this used to be hard, requiring a lot of developer resources. By using Power Apps, businesses can take advantage of its ability to quickly connect together different applications, using device cameras and cloud services along with simple workflows.

The result might not be as pretty or as fully featured as a specially designed app, but it’s going to do the job and do it more than well enough. And what’s more, you can have something up and running in a matter of days as a proof of concept or a way of quickly solving problems.

LEAVE A REPLY

Please enter your comment!
Please enter your name here