PHYS/ASTR Colloquium: "Towards Physically-Contextualized Computer Vision through Studies in the Cryosphere" - Ellianna Abrahams (PhD Candidate, U.C. Berkeley)

San Francisco State University

Physics & Astronomy Colloquium Series

Monday, February 12, 2024

Thornton Hall 411, 3:30 PM

Towards Physically-Contextualized Computer Vision through Studies in the Cryosphere

Ellianna Abrahams (PhD Candidate, U.C. Berkeley)

Remote sensing datasets have become a key facet in the observation and analysis of evolving processes across many sectors of applied physics, like astronomy, geophysics, and planetary science. These large, rich datasets are highly dimensional, and provide some of the only near-continuous time series of observed features for remote locations on Earth, for other planets in the solar system, and for stars and galaxies beyond. Off-the- shelf computer vision methods are traditionally developed for highly-labeled or synthesized data, and do not always easily parse complex, unlabeled, “real-world” scenes from satellite missions. In contrast, a human domain expert can provide intuition from their scientific discipline that allows them to parse a scene by hand, identifying even rare classes from contextual information and selecting features that have previously known scientific relationships. However with >100TB of data being collected daily, the need for robust automation through machine learning to aid in inference and discovery is evident. As a solution to these challenges, I present a case study at the ice-ocean-atmosphere interface in Antarctica that explicitly models scientific context at different stages of the deep learning pipeline, from data preprocessing through model architecture and optimization. I demonstrate the potential of geophysics-contextualized neural networks to robustly automate complex classification tasks, such as pixel-based labeling of remote sensing imagery, that are a necessary stepping stone to understanding the uncertainties in forecasting future sea level rise.