Combining thermal imaging and augmented reality (AR) enables firefighters see through smoke, in effect enhancing their vision in the life-threatening environment of a fire. AR capabilities can be deployed in a visor attached to a helmet, and an affixed thermal camera captures the images.
The most recent prototype of such a product is a robust helmet design that withstands rough treatment. The system also includes software processing that augments thermal images to enable firefighters to see the outline of objects more clearly, thus enabling their detection in the field.
augmented reality
Firefighters can see through the visor/screen, or they can view the thermal image video live overlaid onto their vision. A switch on the side of the helmet allows them to switch among various modes – gray scale, RGB mode, fusion mode, etc. National Fire Protection Association (NFPA) standards specify what information appears on an AR screen. In general, mostly emergency alerts appear so as to simplify the view.
The technology began as a project at McMaster University in Hamilton, Ontario, Canada, to combine augmented reality (AR) with thermal imaging for use by police and fire departments. The project has evolved into the startup company Longan Vision.
Live fire situations
Funding has been provided by Defense Research and Development of Canada (DRDC) and private sources
Development began in 2018 and has continued for two years through six generations of prototypes. The goal throughout the iterations has been to make sure the product is sufficiently robust for use in the field and will hold up to rough usage by firefighters, according to Enzo Jia, CEO and founder of Longan Vision. Funding has been provided by Defense Research and Development of Canada (DRDC) and private sources, and the project has progressed in partnership with industry manufacturers including helmet and apparatus companies.
After two years of development, the company is on the verge of testing the product in live fire situations. Pilot testing will begin in January 2021. The device will begin pursuing NFPA certification next March, and hopefully the company will be shipping products by June or July 2021.
Local fire departments
Longan Vision has depended on feedback from fire departments throughout development of the product. They currently have partnerships with more than eight local fire departments, three in the United States. They meet with fire chiefs and firefighters and ask for their honest opinions on how the device operates, whittling down the good and the bad.
Feedback from the departments included the need for hands-free thermal imaging and a system that displays all needed information in their view. The devices have also been tested by fire departments in Tokyo, which has close relationships with Longan Vision’s fire apparatus partners there. “We have tried to work very closely with firefighters,” says Alex Shortt, Longan Vision’s CTO. “We are a group of engineers, but we need firefighters to help us develop a good product. We keep a firefighter focus in all the work we do.”
Virtual reality visor
The AR device can connect with smart phones or other Internet of Things (IoT) devices to add additional information
There are three possibilities for taking the technology to market. One is to work with helmet manufacturers to incorporate the AR technology into helmets. A second possibility is to work with fire apparatus companies to provide the technology as part of a broader ‘system’ of capabilities surrounding modernized apparatus.
The third option is to equip the technology with a ‘universal mounting system’ – a series of bands and/or straps that allow the camera and attached virtual reality visor to be installed onto an existing helmet. The AR device can connect with smart phones or other Internet of Things (IoT) devices to add additional information to the system and visor display screen, such as the heart rate of the firefighter, for example.
Incident Command Center
Firefighters can communicate among themselves using a Mesh network, or alternatively, a 5G network if available. Each firefighter’s unit can also communicate with an incident Command Center. Incorporating VR into a robust helmet is preferable in the fire industry to using an off-the-shelf AR device such as Google Glass or Microsoft HoloLens, says Jia.
The AR view can be recorded as video to an SD card in the helmet, which can hold two hours or so of footage
These devices are not designed for the rigors of the firefighting environment and therefore must be worn inside a self-contained breathing apparatus (SCBA) mask. However, in that scenario, if there is a device failure, removing the device is more complicated, as is the process of handing the device over to another firefighter if someone ‘steps in’ to an emergency situation. The AR view can be recorded as video to an SD card in the helmet, which can hold two hours or so of footage. However, the demand for such a feature is mixed among fire departments.
Early detection method
A fire chief might want the recording to use for training purposes, but a firefighter might not want the video to be used to judge the potential liability of his every action, which could be open to legal scrutiny or even lawsuits. The COVID-19 pandemic has had an impact on Longan Vision as they look to launch their new product, including a tightening in the procurement market.
On the positive side, the company has adapted the technology into another product, Gatekeeper, that uses thermal imaging to measure skin temperature as an early detection method for elevated temperatures. The system has an accuracy of plus-or-minus 0.3 degrees Celsius and can measure a maximum of five people at a time. The wall-mounted, non-contact unit captures a face image of people entering the video frame and displays each face alongside their skin temperature.
