Teaching AI to communicate sounds like humans do
Whether you’re describing the sound of your faulty car engine or meowing like your neighbor’s cat, imitating sounds with your voice can be a helpful way to relay a concept when words don’t do the trick. Vocal imitation is the sonic equivalent of doodling a quick picture to communicate something you saw — except that instead of using a pencil to illustrate an image, you use your vocal tract to express a sound. This might seem difficult, but it’s something we all do intuitively: To experience it for yourself, try using your voice to mirror the sound of an ambulance siren, a crow, or a bell being struck. Inspired by the cognitive science of how we communicate, MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) researchers have developed an AI system that can produce human-like vocal imitations with no training, and without ever having “heard” a human vocal impression before. To achieve this, the researchers engineered their system to produce and interpret sounds much like we do. They started by building a model of the human vocal tract that simulates how vibrations from the voice box are shaped by the throat, tongue, and lips. Then, they used a cognitively-inspired AI algorithm to control this vocal tract model and make it produce imitations, taking into consideration the context-specific ways that humans choose to communicate sound. The model can effectively take many sounds from the world and generate a human-like imitation of them — including noises like leaves rustling, a snake’s hiss, and an approaching ambulance siren. Their model can also be run in reverse to guess real-world sounds from human vocal imitations, similar to how some computer vision systems can retrieve high-quality images based on sketches. For instance, the model can correctly distinguish the sound of a human imitating a cat’s “meow” versus its “hiss.” In the future, this model could potentially lead to more intuitive “imitation-based” interfaces for sound designers, more human-like AI characters in virtual reality, and even methods to help students learn new languages. The co-lead authors — MIT CSAIL PhD students Kartik Chandra SM ’23 and Karima Ma, and undergraduate researcher Matthew Caren — note that computer graphics researchers have long recognized that realism is rarely the ultimate goal of visual expression. For example, an abstract painting or a child’s crayon doodle can be just as expressive as a photograph. “Over the past few decades, advances in sketching algorithms have led to new tools for artists, advances in AI and computer vision, and even a deeper understanding of human cognition,” notes Chandra. “In the same way that a sketch is an abstract, non-photorealistic representation of an image, our method captures the abstract, non-phono–realistic ways humans express the sounds they hear. This teaches us about the process of auditory abstraction.” The art of imitation, in three parts The team developed three increasingly nuanced versions of the model to compare to human vocal imitations. First, they created a baseline model that simply aimed to generate imitations that were as similar to real-world sounds as possible — but this model didn’t match human behavior very well. The researchers then designed a second “communicative” model. According to Caren, this model considers what’s distinctive about a sound to a listener. For instance, you’d likely imitate the sound of a motorboat by mimicking the rumble of its engine, since that’s its most distinctive auditory feature, even if it’s not the loudest aspect of the sound (compared to, say, the water splashing). This second model created imitations that were better than the baseline, but the team wanted to improve it even more. To take their method a step further, the researchers added a final layer of reasoning to the model. “Vocal imitations can sound different based on the amount of effort you put into them. It costs time and energy to produce sounds that are perfectly accurate,” says Chandra. The researchers’ full model accounts for this by trying to avoid utterances that are very rapid, loud, or high- or low-pitched, which people are less likely to use in a conversation. The result: more human-like imitations that closely match many of the decisions that humans make when imitating the same sounds. After building this model, the team conducted a behavioral experiment to see whether the AI- or human-generated vocal imitations were perceived as better by human judges. Notably, participants in the experiment favored the AI model 25 percent of the time in general, and as much as 75 percent for an imitation of a motorboat and 50 percent for an imitation of a gunshot. Toward more expressive sound technology Passionate about technology for music and art, Caren envisions that this model could help artists better communicate sounds to computational systems and assist filmmakers and other content creators with generating AI sounds that are more nuanced to a specific context. It could also enable a musician to rapidly search a sound database by imitating a noise that is difficult to describe in, say, a text prompt. In the meantime, Caren, Chandra, and Ma are looking at the implications of their model in other domains, including the development of language, how infants learn to talk, and even imitation behaviors in birds like parrots and songbirds. The team still has work to do with the current iteration of their model: It struggles with some consonants, like “z,” which led to inaccurate impressions of some sounds, like bees buzzing. They also can’t yet replicate how humans imitate speech, music, or sounds that are imitated differently across different languages, like a heartbeat. Stanford University linguistics professor Robert Hawkins says that language is full of onomatopoeia and words that mimic but don’t fully replicate the things they describe, like the “meow” sound that very inexactly approximates the sound that cats make. “The processes that get us from the sound of a real cat to a word like ‘meow’ reveal a lot about the intricate interplay between physiology, social reasoning, and communication in the evolution of language,” says Hawkins, who wasn’t
New AI tool generates realistic satellite images of future flooding
Visualizing the potential impacts of a hurricane on people’s homes before it hits can help residents prepare and decide whether to evacuate. MIT scientists have developed a method that generates satellite imagery from the future to depict how a region would look after a potential flooding event. The method combines a generative artificial intelligence model with a physics-based flood model to create realistic, birds-eye-view images of a region, showing where flooding is likely to occur given the strength of an oncoming storm. As a test case, the team applied the method to Houston and generated satellite images depicting what certain locations around the city would look like after a storm comparable to Hurricane Harvey, which hit the region in 2017. The team compared these generated images with actual satellite images taken of the same regions after Harvey hit. They also compared AI-generated images that did not include a physics-based flood model. The team’s physics-reinforced method generated satellite images of future flooding that were more realistic and accurate. The AI-only method, in contrast, generated images of flooding in places where flooding is not physically possible. The team’s method is a proof-of-concept, meant to demonstrate a case in which generative AI models can generate realistic, trustworthy content when paired with a physics-based model. In order to apply the method to other regions to depict flooding from future storms, it will need to be trained on many more satellite images to learn how flooding would look in other regions. “The idea is: One day, we could use this before a hurricane, where it provides an additional visualization layer for the public,” says Björn Lütjens, a postdoc in MIT’s Department of Earth, Atmospheric and Planetary Sciences, who led the research while he was a doctoral student in MIT’s Department of Aeronautics and Astronautics (AeroAstro). “One of the biggest challenges is encouraging people to evacuate when they are at risk. Maybe this could be another visualization to help increase that readiness.” To illustrate the potential of the new method, which they have dubbed the “Earth Intelligence Engine,” the team has made it available as an online resource for others to try. The researchers report their results today in the journal IEEE Transactions on Geoscience and Remote Sensing. The study’s MIT co-authors include Brandon Leshchinskiy; Aruna Sankaranarayanan; and Dava Newman, professor of AeroAstro and director of the MIT Media Lab; along with collaborators from multiple institutions. Generative adversarial images The new study is an extension of the team’s efforts to apply generative AI tools to visualize future climate scenarios. “Providing a hyper-local perspective of climate seems to be the most effective way to communicate our scientific results,” says Newman, the study’s senior author. “People relate to their own zip code, their local environment where their family and friends live. Providing local climate simulations becomes intuitive, personal, and relatable.” For this study, the authors use a conditional generative adversarial network, or GAN, a type of machine learning method that can generate realistic images using two competing, or “adversarial,” neural networks. The first “generator” network is trained on pairs of real data, such as satellite images before and after a hurricane. The second “discriminator” network is then trained to distinguish between the real satellite imagery and the one synthesized by the first network. Each network automatically improves its performance based on feedback from the other network. The idea, then, is that such an adversarial push and pull should ultimately produce synthetic images that are indistinguishable from the real thing. Nevertheless, GANs can still produce “hallucinations,” or factually incorrect features in an otherwise realistic image that shouldn’t be there. “Hallucinations can mislead viewers,” says Lütjens, who began to wonder whether such hallucinations could be avoided, such that generative AI tools can be trusted to help inform people, particularly in risk-sensitive scenarios. “We were thinking: How can we use these generative AI models in a climate-impact setting, where having trusted data sources is so important?” Flood hallucinations In their new work, the researchers considered a risk-sensitive scenario in which generative AI is tasked with creating satellite images of future flooding that could be trustworthy enough to inform decisions of how to prepare and potentially evacuate people out of harm’s way. Typically, policymakers can get an idea of where flooding might occur based on visualizations in the form of color-coded maps. These maps are the final product of a pipeline of physical models that usually begins with a hurricane track model, which then feeds into a wind model that simulates the pattern and strength of winds over a local region. This is combined with a flood or storm surge model that forecasts how wind might push any nearby body of water onto land. A hydraulic model then maps out where flooding will occur based on the local flood infrastructure and generates a visual, color-coded map of flood elevations over a particular region. “The question is: Can visualizations of satellite imagery add another level to this, that is a bit more tangible and emotionally engaging than a color-coded map of reds, yellows, and blues, while still being trustworthy?” Lütjens says. The team first tested how generative AI alone would produce satellite images of future flooding. They trained a GAN on actual satellite images taken by satellites as they passed over Houston before and after Hurricane Harvey. When they tasked the generator to produce new flood images of the same regions, they found that the images resembled typical satellite imagery, but a closer look revealed hallucinations in some images, in the form of floods where flooding should not be possible (for instance, in locations at higher elevation). To reduce hallucinations and increase the trustworthiness of the AI-generated images, the team paired the GAN with a physics-based flood model that incorporates real, physical parameters and phenomena, such as an approaching hurricane’s trajectory, storm surge, and flood patterns. With this physics-reinforced method, the team generated satellite images around Houston that depict the same flood extent, pixel by pixel, as forecasted by the flood model. “We show a tangible way to combine
Donald Trump Taps Dr. Oz to Head U.S. Medicaid & Medicare
President-elect Donald Trump has nominated Dr. Mehmet Oz, a well-known television personality and surgeon, to lead the Centers for Medicare and Medicaid Services (CMS). Oz, who will oversee programs impacting millions of Americans, is known for his media presence and health advocacy. However, some have criticized his promotion of treatments lacking scientific support. Trump’s selection of Oz highlights his unconventional approach to leadership, blending celebrity influence with healthcare reform goals. Related Podcast Trump Taps Dr. Oz for Medicaid & Medicare President-elect Donald Trump has nominated Dr. Mehmet Oz, a well-known television personality and surgeon, to lead the Centers for Medicare and Medicaid Services (CMS).
Whoopi Goldberg launches All Women’s Sports Network (AWSN)
Trailblazing actress and activist Whoopi Goldberg is breaking new ground in sports media with the launch of the All Women’s Sports Network (AWSN), a groundbreaking television channel dedicated exclusively to women’s sports. In collaboration with Jungo TV, AWSN aims to provide a platform for female athletes to shine, offering live coverage of major leagues such as UEFA, FIBA, WTA, and WNBL. This bold initiative is already available in 65 countries, reaching a potential audience of over 2 billion people, and has successfully debuted in parts of Asia and the Middle East. The launch of AWSN comes at a critical time, as women’s sports continue to gain momentum but still receive only a fraction of the media coverage given to men’s sports. Goldberg’s vision, which she has nurtured for 16 years, seeks to address this disparity by celebrating the talent, dedication, and athleticism of women across a wide range of sports, including soccer, basketball, tennis, cricket, and even curling. By amplifying these stories, AWSN hopes to inspire future generations of athletes and create a more equitable playing field. Related Podcast Whoopi launches women’s sports network Whoopi Goldberg is breaking new ground in sports media with the launch of the All Women’s Sports Network (AWSN) – the first global television channel dedicated exclusively to women’s sports. In collaboration with Jungo TV, AWSN is now available in 65 countries, providing live coverage of major women’s sports leagues. Goldberg’s initiative is not only a cultural milestone but also a timely response to the growing interest in women’s sports. With record-breaking viewership for events like the FIFA Women’s World Cup and increased investments from brands and organizations, AWSN is poised to meet the demand for more diverse and inclusive sports programming. It also serves as a powerful reminder of the need for consistent and widespread media representation for female athletes, who often perform at the highest levels with minimal recognition. As AWSN expands globally, Goldberg’s mission to champion women in sports sends a resounding message: athletic excellence knows no gender. The All Women’s Sports Network stands as a testament to the power of visibility and the importance of creating platforms that recognize and celebrate the achievements of women, on and off the field. Through AWSN, Goldberg is not only transforming sports media but also paving the way for a more inclusive future.
Trump poised for unrestricted leadership
Donald Trump has made history once again. Nearly eight years after his surprising win over Hillary Clinton, and four years after Joe Biden’s term removed him from office, Trump is ready to return to the White House. With a strong showing in key early voting states and improved support across the country, Trump declared he has a “powerful mandate” to lead. “This will truly be America’s golden age,” he told a cheering crowd in West Palm Beach, Florida. A Stronger Conservative Movement Trump’s victory signals a continued shift in U.S. politics toward conservative populism—a movement that began with his 2016 election and appeared at risk after his 2020 defeat. Now, this movement looks stronger than ever. Trump now has the chance to build a new administration and act on the promises he’s made for a brighter American future. This administration will also be supported by a Republican-controlled Senate, which makes confirming Trump’s political appointees easier, including his Cabinet and judges. Although results for the House of Representatives aren’t final, Trump predicted a Republican win there too. Having a Republican-led Congress will help him advance his plans, including a complete federal overhaul by appointing loyalists to key positions across the government. Joining Trump in his administration are big names like billionaire Elon Musk, vaccine skeptic Robert F. Kennedy Jr., former Democrat Tulsi Gabbard, entrepreneur Vivek Ramaswamy, and others in his unique coalition. Four Years to Fulfill Promises Trump has also pledged new tariffs to protect U.S. industries, targeted tax breaks, and large-scale deportation of undocumented immigrants. On foreign policy, he promises to quickly end conflicts in Ukraine and Gaza, focusing on America’s interests. By January, his administration will handle these global challenges. Vice President Kamala Harris, Democrats, and some of Trump’s former advisors warn that his policies may disrupt the economy and society, possibly even affecting global stability. They worry that a second Trump term could be even less restrained than his first. Trump himself admitted his second term might “be rough at times,” but he promised good results in the end. A majority of voters, it seems, agreed. With a Republican Congress, Trump could reverse many of the policies from the last four years and pass conservative laws on taxes, spending, trade, and immigration, aiming to leave a lasting mark on government. A Remarkable Comeback Trump’s victory marks a surprising comeback for someone who left office after the January 6 incident, seemingly with his reputation in trouble. After intense criticism from both Democrats and some Republicans, Trump spent four years working his way back to the top. During that time, he faced legal challenges, including criminal indictments, felony convictions, civil judgments related to assault claims, and major fines for his business. Yet, he pushed forward, securing the Republican nomination and focusing on his campaign. Though his rallies could sometimes be unfocused, Trump built a skilled team. Polls showed voters trusted him on immigration and the economy, and his campaign stayed on these issues. Being aligned with voter concerns proved crucial, as many Americans—and people in democracies worldwide—were increasingly anti-incumbent. Trump’s campaign mobilized rural voters and cut into Democratic strongholds in urban areas. Preliminary exit polls show he even gained ground with younger, Hispanic, and Black voters, groups that usually vote Democrat. Although Trump’s team initially struggled when Biden left the race, leaving him to face Kamala Harris, he soon found his stride, riding the wave of anti-incumbent sentiment back to the White House. Now, with four more years ahead, Trump has a more organized political team ready to turn his promises into lasting policies. We’ll be following his presidency closely, and reporting on his progress here at Readovia. Related Podcast Trump Reclaims White House Throne Donald Trump has defeated Kamala Harris and will return to the White House in January 2025. Sources attribute Trump’s victory to key wins in swing states and suggest that inflation may have been a contributing factor. In this discussion, we highlight the significance of Trump’s victory, including his position as the first convicted felon to win the presidency.