360 Capture Innovations: Remotely Piloted Rovers for Campus Capture & Industry Applications

Safety and security are critical to every campus environment. By using new technologies in 360 photogrammetry and developing innovative remote capture methods, Mohawk College’s School of Building and Construction Sciences applied research team is building a 360° virtual walkthrough of their entire campus to support safety, security and facilities management efforts.

The applied research team, composed of students and faculty led by Professor Richard Borger and faculty member Matt Shelley, is designing remotely piloted robotic vehicles that leverage Visual Plan’s capture and processing platform to generate a comprehensive and accurate virtual 360° walkthrough of the college’s campus.  “There are always more parts and pieces being added to our buildings and facilities: having this 360 photo capture allows them to more closely monitor progress and be able to inventory things, places and spaces across the campus,” said Richard Borger of the project.

As the Mohawk College campus capture project nears completion, Borger, Shelley and their students are having conversations with facilities management and security teams about how to further leverage the visual model as a resource to share with local first responders. The faculty and students are also utilizing their innovations when working with industry partners looking for virtual workspaces to showcase an environment for various applications, training or safety initiatives. The applied research team supports those partners as they begin implementing 360 capture technology, testing functionality, and helping walk them through the project from start to finish.

“Our main role as faculty is educating students and supporting industry partners with the adoption of new technology. We’ve focused on drone and remotely piloted technology, as well as imagery technology like photogrammetry and Lidar – Visual Plan is a technology where those two worlds meet.” – Richard Borger, Professor at Mohawk College

Challenges: Campus Access Under COVID-19 Protocols & Remote Piloting For Optical Height

Initial challenges for the campus capture project concerned access and inaccurate facility staging: When the team gained access to the campus in June 2020, all of the College’s COVID protocols were already in place – water fountains were covered in plastic bags, COVID signage was posted and common areas were without seating, resulting in an inaccurate visual representation of the normal campus. 

In order to produce a capture that was visually consistent with normal campus function, the applied research team has spent a significant amount of time meticulously staging each environment before 360° photos were taken. As explained by Borger, “We’re planning to use the digital twin for building security, so we want the 360 visualization to be representative of what first responders will actually see during an active situation. And secondly, that we aren’t blocking major facility infrastructure: for example, temporarily stacked chairs in front of a fuse panel, white boards or projector screens hiding a major piece of infrastructure. We are trying to expose as much of the facility as possible so that when we deliver it to our safety and facilities management teams, they have something that is functionally accurate.”

For their final deliverables, the end user experience of the virtual walkthrough was paramount to maximize its future utility. Initial captures using a hardhat mount resulted in a disorienting user experience with the photogrammetry having an optical height higher than the average person’s eye-line: the Ricoh Theta Z1 camera center point was about 7 feet off the ground. The applied research team needed both a versatile capture technology and a more agile delivery mechanism to facilitate and expedite the capture process, generating the campus’ virtual walkthrough experience.

Solution: Design A Remote-Controlled Delivery Mechanism Leveraging Visual Plan’s 360° Capture and Processing Power

With their main goal to have the camera at a consistent optical height without having to use a manually positioned tripod for each capture, Borger, Shelley and their students initially discussed ideas about robot trucks to attain consistent center point focus during the 360 capture process. Working with a local hobby shop, they developed a working prototype of a remotely piloted robotic vehicle featuring a wide wheel base and a surveying pole mounted on top. By collaborating with Visual Plan on the capture options and utilizing it as the capture platform, the team was able to leverage the platform’s built-in video capture mode while operating the moving vehicle mount for faster remote 360° photo capture at regular intervals. 

With Visual Plan, the remote vehicle capture method has worked seamlessly with the platform’s AI-driven pano placement and spatial alignment of the captured 360 images. “The workflow has been outstanding. When we’re in lecture theaters – normally we’d walk between the rows for manual capture. Now, we put the Rover at the start of one row and drive, and the metadata uploads and puts the panos in that right room. We just drive the Rover in a series of rows down the classroom with one crossing, and the whole room is done. It’s been really fantastic to see the workflow so smooth and intuitive with data placement as well,” said Borger of the Visual Plan experience.

The combination of video 360 capture and the remote-controlled vehicle speeds up the data collection process, allows for consistent data capture and opens the door for future capture capabilities and opportunities in toxic or dangerous environments.

Remote Capture: Designed for Extended Industry Applications & Data Integration

When industries evaluate methods for facility visualization, the technology spans from 360° photogrammetry to lidar to aerial data, all aiming to build a detailed and comprehensive picture of an asset, facility or environment. Stakeholders can eliminate data silos via the Visual Plan interface to use 360° photos and virtual walkthroughs to navigate to a specific location in a tour, and then access the BIM, lidar or other data type for that specific area as needed, all from one platform.

 “When we are actually interacting with the data, most people prefer to use the photogrammetry data. When we interact with the 360 photos, it’s easier to experience and see the digital representation of the physical environment. The lidar is more effective for precise measurements, for example in-depth mechanical plumbing or electrical systems where robust measurement is needed. Platforms like Visual Plan can use the photogrammetry to place and position an individual, and then transition to the lidar for precise needs,” said Borger of his industry experience.

Drone imagery and orthomosaic data can also be leveraged and merged with remotely piloted 360° Rover or manually captured terrestrial data, providing both vertical and horizontal perspectives to mitigate line-of-sight limitations, shadowing, or other visual obstructions. 

For the students participating in the Rover development, campus capture and industry applications, they are gaining critical field experience in merging data sources to produce comprehensive walkthroughs. According to student James Kertz, “Each project has a varying level of detail for what’s necessary to capture. We optimized the LOD (level of detail) with the high resolution you get from a Lidar scanner at the most important areas – intersections between buildings or entryways where we need accurate measurements. The 360° images are going to be effective for visualizing what’s necessary from every angle, and then you can supplement that with the lidar scanner for more detail at key locations.”

“I think that Visual Plan is a great addition to IOT technologies – the internet of things. It’s a great asset for maintenance, risk assessments or quality inspections. Particularly with big warehouses where there are a lot of pieces of equipment – these assets could be located with sensors, to be visually connected as a list of all assets and their maintenance needs. The ‘internet of things’ is a great compliment for Visual Plan.” – Katherine, Mohawk College Student

Critical Features for Success

For a 360° capture project of this size, it was critical for the Mohawk team to be as organized as possible when uploading and processing the photogrammetry. The Visual Plan platform is optimized for the large square footage, using metadata to automatically place images into the uploaded floor plan for efficient creation and management.

With an intuitive interface and AI-enabled pano processing and placement, the applied research team students were able to learn, operate and leverage the Visual Plan’s capabilities on the Mohawk campus and as part of their industry-partners program, gaining real-world experience creating Digital Twins across industry and applications.

Critical Benefits:

• Barrier to entry was low with regard to the capital investment
• Easy import, scale and alignment of floor plans to GIS via Import Floorplan Wizard
• Simple auto and manual orientation of 360° imagery to CAD to enhance understanding
• Intuitive organization of 360 photos across levels, areas and spaces (accelerated via the Visual Plan Capture App)
• Intuitive interface and functionality

“The critical feature for us on this large project has been uploading the floorplan. Using the app on the phone has saved us a lot of time as well; using the Rover saves us more time, allows us to mark the initial position in the app, as well as the ending position, to expedite the capture processing and pano alignment process when we upload the pictures.“

– Katherine Anzola Bohorquez, Mohawk College Student

Driving Industry & Application through Innovation

Working with development partners, Visual Plan and their local hobby shop, the applied research team of faculty and students at Mohawk College have designed and built industry-changing, remotely piloted 360° Rovers for capturing 360° photogrammetry on their campus and beyond. Their current application of campus capture for safety and facilities management continues to be successful, relying on Visual Plan to capture and build reliable and accurate digital twins to assist future facility development and first responders. As their remote capture vehicle designs become more capable and versatile, so does the future potential of capturing and leveraging digital twins across various environments, industries and applications.

“The benefit is for Mohawk’s students to be able to work on industry-specific projects. We have students that are now given experiential learning opportunities, and we have industry that is now able to tap into a resource that never existed before, and it’s all through the support of companies like Visual Plan or Hot Rod Hobbies, where they are actually providing support for us as an educational institution, but then we can take that directly into industry and start integrating it into their workflow.”

– Richard Borger, Mohawk College

Meet the 360 Rovers

The Rock Crawler

• Gentle startup
• Wide turn radius
• Very stable on bumpy ground, long and wide wheelbase for more rugged environments with room to maneuver 
• Effective operation on inclined terrain

Rover 2

• High powered, fast startup
• Tighter turn radius