Intelligent Machine Learning for Real Time Processing of Hyperspectral Video Streams

Signature Detection & Classification

Signature Detection and Classification centers on identifying and categorizing specific features within data. Our research has yielded two major contributions: the application of neuromorphic principles and unsupervised hyperspectral image segmentation, addressing challenges from deblurring to image segmentation. These advancements not only push the boundaries of Signature Detection but extend to various data analysis domains. In the field of Integration of IR/EO Video Streams, we've laid milestones in computer vision through methods like consistent shape matching and image intensity recovery. By synthesizing data-driven modeling, parametric mesh generation, and event-enhanced algorithms, we've created building blocks for future video processing systems. These contributions are outlined as follows:

DeblurSR: Event-Based Motion Deblurring Under the Spiking Representation

DeblurSR presents a novel approach to motion deblurring, converting a blurry image into a sharp video (Song et al., 2023). Leveraging event data and the spiking representation (SR), DeblurSR compensates for motion ambiguities and parameterizes the output video as a time-to-intensity mapping. Inspired by the biological principles governing neuron communication in living organisms, the SR explains the representation of sharp edges and interprets spiking parameters from a neuromorphic perspective.

The advantages of DeblurSR include superior output quality and reduced computational resources compared to contemporary event-based motion deblurring methods. Notably, our approach is easily extendable to video super-resolution, particularly when integrated with the latest developments in implicit neural representation. This contribution to Signature Detection accentuates the fusion of neuromorphic principles with imaging technology, forging new pathways in image deblurring. The architecture is shown in Figure 1.

Flowchart showing process of deblurring an image
Figure 1: DeblurSR Pipeline.

Given a blurry image and its associated events in the exposure interval, we apply a Convolutional Neural Network (CNN) to extract an image embedding with the same spatial resolution as the input. For each pixel (x,y), we fuse the image embedding with the coordinate embedding using the addition operation. A group of fully-connected layers take the resulting per-pixel feature vector as input and regress the spiking parameters for each pixel as output. At time tr, we assemble a spatially varying kernel from the predicted spiking parameters. The convolution of this kernel with the input blurry image gives the output sharp frame at time tr. By changing the timestamps, the spiking representation allows DeblurSR to render a sharp video with an arbitrarily high frame-rate.

Code: The code for this project is available on the SharePoint. We experimented DeblurSR on the REDS dataset and the HDF dataset. Please refer to REDS_Dataset.md and HQF_Dataset.md for instructions on how to set up these two datasets.

A Distribution-dependent Mumford-Shah Model for Unsupervised Hyperspectral Image Segmentation

The challenge of unsupervised hyperspectral image segmentation is addressed in our work on a distribution-dependent Mumford-Shah (MS) model (Cohrs et al., 2022). Hyperspectral images, encapsulating detailed spectral data for each pixel, demand intricate segmentation into different classes, a task compounded by spectral variability and noise.

Our framework commences with denoising and dimensionality reduction using the well-established Minimum Noise Fraction (MNF) transform, followed by the application of the MS segmentation functional. Enhanced with a robust distribution-dependent indicator function, the MS functional is tailored to the unique challenges of hyperspectral data. An efficient fixed-point iteration scheme optimizes the objective function, leading to competitive results that substantially outperform several state-of-the-art methods on benchmark datasets.

This contribution to Signature Detection highlights the integration of unsupervised learning with hyperspectral imaging, providing a nuanced approach to classifying complex spectral data. Figure 2 compares segmentation methods on the Pavia University dataset, including the ground truth, k-means, GMM, BGM, 3D-CAE, MS-2, and our proposed method. The Pavia University dataset presented unique challenges, and the figure illustrates the comparative performance of various approaches, demonstrating the effectiveness of our method.

Comparison of hyperspectral image segmentation methods
Figure 2: Hyperspectral Image Segmentation Comparison.
Comparison of hyperspectral image segmentation methods on the Pavia University dataset. Methods compared include (a) Ground truth, (b) k-means (0.534), (c) GMM (0.517), (d) BGM (0.444), (e) 3D-CAE (0.535), (f) MS-2 (0.564), (g) Ours (0.562).

References

  • Cohrs, J.-C., Bajaj, C., & Berkels, B. (2022). A Distribution-Dependent Mumford–Shah Model for Unsupervised Hyperspectral Image Segmentation. IEEE Transactions on Geoscience and Remote Sensing, 60, 1–21.

  • Song, C., Bajaj, C., & Huang, Q. (2023). DeblurSR: Event-Based Motion Deblurring Under the Spiking Representation. ArXiv Preprint ArXiv:2303.08977.