New AI tool generates realistic satellite images of future flooding
Visualizing the potential impacts of a hurricane on people’s homes before it hits can help residents prepare and decide whether to evacuate. MIT scientists have developed a method that generates satellite imagery from the future to depict how a region would look after a potential flooding event. The method combines a generative artificial intelligence model with a physics-based flood model to create realistic, birds-eye-view images of a region, showing where flooding is likely to occur given the strength of an oncoming storm. As a test case, the team applied the method to Houston and generated satellite images depicting what certain locations around the city would look like after a storm comparable to Hurricane Harvey, which hit the region in 2017. The team compared these generated images with actual satellite images taken of the same regions after Harvey hit. They also compared AI-generated images that did not include a physics-based flood model. The team’s physics-reinforced method generated satellite images of future flooding that were more realistic and accurate. The AI-only method, in contrast, generated images of flooding in places where flooding is not physically possible. The team’s method is a proof-of-concept, meant to demonstrate a case in which generative AI models can generate realistic, trustworthy content when paired with a physics-based model. In order to apply the method to other regions to depict flooding from future storms, it will need to be trained on many more satellite images to learn how flooding would look in other regions. “The idea is: One day, we could use this before a hurricane, where it provides an additional visualization layer for the public,” says Björn Lütjens, a postdoc in MIT’s Department of Earth, Atmospheric and Planetary Sciences, who led the research while he was a doctoral student in MIT’s Department of Aeronautics and Astronautics (AeroAstro). “One of the biggest challenges is encouraging people to evacuate when they are at risk. Maybe this could be another visualization to help increase that readiness.” To illustrate the potential of the new method, which they have dubbed the “Earth Intelligence Engine,” the team has made it available as an online resource for others to try. The researchers report their results today in the journal IEEE Transactions on Geoscience and Remote Sensing. The study’s MIT co-authors include Brandon Leshchinskiy; Aruna Sankaranarayanan; and Dava Newman, professor of AeroAstro and director of the MIT Media Lab; along with collaborators from multiple institutions. Generative adversarial images The new study is an extension of the team’s efforts to apply generative AI tools to visualize future climate scenarios. “Providing a hyper-local perspective of climate seems to be the most effective way to communicate our scientific results,” says Newman, the study’s senior author. “People relate to their own zip code, their local environment where their family and friends live. Providing local climate simulations becomes intuitive, personal, and relatable.” For this study, the authors use a conditional generative adversarial network, or GAN, a type of machine learning method that can generate realistic images using two competing, or “adversarial,” neural networks. The first “generator” network is trained on pairs of real data, such as satellite images before and after a hurricane. The second “discriminator” network is then trained to distinguish between the real satellite imagery and the one synthesized by the first network. Each network automatically improves its performance based on feedback from the other network. The idea, then, is that such an adversarial push and pull should ultimately produce synthetic images that are indistinguishable from the real thing. Nevertheless, GANs can still produce “hallucinations,” or factually incorrect features in an otherwise realistic image that shouldn’t be there. “Hallucinations can mislead viewers,” says Lütjens, who began to wonder whether such hallucinations could be avoided, such that generative AI tools can be trusted to help inform people, particularly in risk-sensitive scenarios. “We were thinking: How can we use these generative AI models in a climate-impact setting, where having trusted data sources is so important?” Flood hallucinations In their new work, the researchers considered a risk-sensitive scenario in which generative AI is tasked with creating satellite images of future flooding that could be trustworthy enough to inform decisions of how to prepare and potentially evacuate people out of harm’s way. Typically, policymakers can get an idea of where flooding might occur based on visualizations in the form of color-coded maps. These maps are the final product of a pipeline of physical models that usually begins with a hurricane track model, which then feeds into a wind model that simulates the pattern and strength of winds over a local region. This is combined with a flood or storm surge model that forecasts how wind might push any nearby body of water onto land. A hydraulic model then maps out where flooding will occur based on the local flood infrastructure and generates a visual, color-coded map of flood elevations over a particular region. “The question is: Can visualizations of satellite imagery add another level to this, that is a bit more tangible and emotionally engaging than a color-coded map of reds, yellows, and blues, while still being trustworthy?” Lütjens says. The team first tested how generative AI alone would produce satellite images of future flooding. They trained a GAN on actual satellite images taken by satellites as they passed over Houston before and after Hurricane Harvey. When they tasked the generator to produce new flood images of the same regions, they found that the images resembled typical satellite imagery, but a closer look revealed hallucinations in some images, in the form of floods where flooding should not be possible (for instance, in locations at higher elevation). To reduce hallucinations and increase the trustworthiness of the AI-generated images, the team paired the GAN with a physics-based flood model that incorporates real, physical parameters and phenomena, such as an approaching hurricane’s trajectory, storm surge, and flood patterns. With this physics-reinforced method, the team generated satellite images around Houston that depict the same flood extent, pixel by pixel, as forecasted by the flood model. “We show a tangible way to combine
Improving health, one machine learning system at a time
Captivated as a child by video games and puzzles, Marzyeh Ghassemi was also fascinated at an early age in health. Luckily, she found a path where she could combine the two interests. “Although I had considered a career in health care, the pull of computer science and engineering was stronger,” says Ghassemi, an associate professor in MIT’s Department of Electrical Engineering and Computer Science and the Institute for Medical Engineering and Science (IMES) and principal investigator at the Laboratory for Information and Decision Systems (LIDS). “When I found that computer science broadly, and AI/ML specifically, could be applied to health care, it was a convergence of interests.” Today, Ghassemi and her Healthy ML research group at LIDS work on the deep study of how machine learning (ML) can be made more robust, and be subsequently applied to improve safety and equity in health. Growing up in Texas and New Mexico in an engineering-oriented Iranian-American family, Ghassemi had role models to follow into a STEM career. While she loved puzzle-based video games — “Solving puzzles to unlock other levels or progress further was a very attractive challenge” — her mother also engaged her in more advanced math early on, enticing her toward seeing math as more than arithmetic. “Adding or multiplying are basic skills emphasized for good reason, but the focus can obscure the idea that much of higher-level math and science are more about logic and puzzles,” Ghassemi says. “Because of my mom’s encouragement, I knew there were fun things ahead.” Ghassemi says that in addition to her mother, many others supported her intellectual development. As she earned her undergraduate degree at New Mexico State University, the director of the Honors College and a former Marshall Scholar — Jason Ackelson, now a senior advisor to the U.S. Department of Homeland Security — helped her to apply for a Marshall Scholarship that took her to Oxford University, where she earned a master’s degree in 2011 and first became interested in the new and rapidly evolving field of machine learning. During her PhD work at MIT, Ghassemi says she received support “from professors and peers alike,” adding, “That environment of openness and acceptance is something I try to replicate for my students.” While working on her PhD, Ghassemi also encountered her first clue that biases in health data can hide in machine learning models. She had trained models to predict outcomes using health data, “and the mindset at the time was to use all available data. In neural networks for images, we had seen that the right features would be learned for good performance, eliminating the need to hand-engineer specific features.” During a meeting with Leo Celi, principal research scientist at the MIT Laboratory for Computational Physiology and IMES and a member of Ghassemi’s thesis committee, Celi asked if Ghassemi had checked how well the models performed on patients of different genders, insurance types, and self-reported races. Ghassemi did check, and there were gaps. “We now have almost a decade of work showing that these model gaps are hard to address — they stem from existing biases in health data and default technical practices. Unless you think carefully about them, models will naively reproduce and extend biases,” she says. Ghassemi has been exploring such issues ever since. Her favorite breakthrough in the work she has done came about in several parts. First, she and her research group showed that learning models could recognize a patient’s race from medical images like chest X-rays, which radiologists are unable to do. The group then found that models optimized to perform well “on average” did not perform as well for women and minorities. This past summer, her group combined these findings to show that the more a model learned to predict a patient’s race or gender from a medical image, the worse its performance gap would be for subgroups in those demographics. Ghassemi and her team found that the problem could be mitigated if a model was trained to account for demographic differences, instead of being focused on overall average performance — but this process has to be performed at every site where a model is deployed. “We are emphasizing that models trained to optimize performance (balancing overall performance with lowest fairness gap) in one hospital setting are not optimal in other settings. This has an important impact on how models are developed for human use,” Ghassemi says. “One hospital might have the resources to train a model, and then be able to demonstrate that it performs well, possibly even with specific fairness constraints. However, our research shows that these performance guarantees do not hold in new settings. A model that is well-balanced in one site may not function effectively in a different environment. This impacts the utility of models in practice, and it’s essential that we work to address this issue for those who develop and deploy models.” Ghassemi’s work is informed by her identity. “I am a visibly Muslim woman and a mother — both have helped to shape how I see the world, which informs my research interests,” she says. “I work on the robustness of machine learning models, and how a lack of robustness can combine with existing biases. That interest is not a coincidence.” Regarding her thought process, Ghassemi says inspiration often strikes when she is outdoors — bike-riding in New Mexico as an undergraduate, rowing at Oxford, running as a PhD student at MIT, and these days walking by the Cambridge Esplanade. She also says she has found it helpful when approaching a complicated problem to think about the parts of the larger problem and try to understand how her assumptions about each part might be incorrect. “In my experience, the most limiting factor for new solutions is what you think you know,” she says. “Sometimes it’s hard to get past your own (partial) knowledge about something until you dig really deeply into a model, system, etc., and realize that you didn’t understand a subpart correctly or fully.” As passionate as Ghassemi is about her work, she intentionally keeps track of life’s bigger picture. “When you love your research, it can be hard to stop that from
