From a Whale’s Song to an Ocean’s Symphony: How AI Decodes Underwater Sound

^{By Zoraida Díaz}

At the peak of the pandemic in late 2020, the University of South Florida’s (USF) College of Marine Science hosted the virtual conference Acoustic Anthropogenic-Biological Indicators Workshop. The event gathered a multidisciplinary, highly specialized group of oceanographers, bioacoustic engineers, and data scientists to examine a looming crisis in ocean observation: how to translate terabytes of passive acoustic monitoring (PAM) raw audio into automated, manageable metrics for measuring ecosystem health.

The scientists had access to a massive amount of acoustic data from NOAA and Ocean Observatories Initiative hydrophone arrays in the Gulf of America and the Pacific Northwest. The raw audio had been captured using different sound recording systems with inconsistent sampling rates. The researchers set a deceptively simple objective: develop an algorithm capable of automatically isolating boat noise across thousands of hours of underwater recordings originally captured to monitor marine life.

Instead of building a Machine Learning (ML) model from scratch, the research team built a boat engine detection workflow on Google DeepMind’s Perch 1.0, a bioacoustic foundational framework trained primarily on bird sounds. Researchers leveraged transfer learning by feeding the model cross-domain, reef, and unrelated audio to build a highly adaptable, localized classifier. When the framework flags ambiguous or unrecognized audio segments, it requests a human to manually verify the segment as “Boat” or “No Boat”. This process quickly trains the customized classifier to detect vessel noise, enabling it to process thousands of hours of recordings with 88-90% accuracy. And most importantly, it eliminates the traditionally exhaustive manual labeling of audio files.

This early predictive model laid the foundation for an ambitious SECOORA-funded project, Augmenting Ocean Observing through Artificial Intelligence: Annotation, Data Standards, and Applications. This innovative blend of marine ecology, bioacoustics, and computational modeling aimed to “standardize regional passive acoustic monitoring (PAM) databases into a benchmark dataset; train machine learning (ML) models for automatic biological and anthropogenic sound detection; deploy a real-time edge application; and create standards and cloud resources for future PAM databases.” 

Luke McEachron, PhD, the project’s principal investigator and then-Research Administrator for the Florida Fish & Wildlife Conservation Commission (FWC), spearheaded the regional effort. Alongside co-investigators from USF’s Institute for Marine Remote Sensing, the University of South Carolina Beaufort’s Marine Sensory and Neurobiology Lab, and the University of Colorado Boulder’s Cooperative Institute for Research in Environmental Science, the team established a standardized framework to help marine scientists and other stakeholders train AI and deploy ML models effectively.

“To start from scratch and train a model like Perch, you need massive resources,” explained Dr. McEachron. “Instead, the existing model can be refined to your needs.” Google Perch was trained on 1.5 million source recordings covering nearly 15,000 species, including birds, mammals, amphibians, and insects. It relied on crowdsourced, curated data from platforms such as Xeno-Canto and iNaturalist, with labeling by thousands of field biologists and community experts.

“Perch is trained to look for patterns and build spectrograms,” says Dr. McEachron, “and this can work well for fish too—or boats—if you can identify them.” A spectrogram is the visual representation of an audio file. Ironically, AI isn’t “listening” to the song of a whale or the low-frequency vibrations of a boat passing. Instead, AI is “seeing” a cluster of pixels on a screen. 

“It’s a visualization of sound,” explains Dr. McEachron: “A picture of silence.” 

Passive Acoustic Monitoring: A History

Marine Passive Acoustic Monitoring (PAM) has come a long way. 

Earlier this year, researchers at the Woods Hole Oceanographic Institution (WHOI) discovered the earliest known recording of a humpback whale’s song within their archives. The audio was originally captured near Bermuda on March 7, 1949.

Researchers from WHOI and the U.S. Office of Naval Research were testing sonar systems and conducting acoustic experiments aboard the R/V Atlantis. The mysterious sounds were recorded with a crude hydrophone on an improvised Audograph office dictating machine that etched audio onto plastic discs instead of magnetic tape. The content in the well-preserved analog discs was not cataloged because the researchers could not identify the eerie sounds.

What researchers did identify was the military potential of passive acoustic underwater technology. In the 1950s, the U.S. Navy secretly developed the highly classified Sound Surveillance System (SOSUS), a massive network of underground microphones (hydrophones) anchored to the ocean floor across the Atlantic and Pacific Oceans.

During the Cold War, Navy operators listened for the low-frequency hum of Soviet submarines but kept running into what they believed was acoustic interference. The mysterious wailing sounds were later identified as the songs of baleen whales—blue, fin, and humpback. It wasn’t until 1992 that the U.S. Navy partially declassified the SOSUS network in the North Pacific, allowing NOAA access to the low-frequency seafloor hydrophone arrays. Scientists were then able to access archives with decades of recorded underwater data and make use of the extensive technology.

In the 70s and 80s, the field of Bioacoustics quickly developed. Early systems included towed hydrophone arrays from ships and stationary hydrophones placed in specific locations to track whale movement. These early systems were limited by short battery lives and large magnetic tape recorders. Roger Payne and Scott McVay’s landmark 1971 paper “Songs of Humpback Whales” revolutionized the field and gave birth to the global anti-whaling movement. In 1978, during an episode of the series Life on Earth, David Attenborough was filmed on a boat off the coast of Hawaii listening to the songs of humpback whales from a hydrophone lowered over the side of the boat as the reels on his analog device turned.

The 1990s saw the transition from analog tape recordings to digital storage. The smaller, more energy-efficient microprocessors paved the way for autonomous recorders that could remain on the seafloor for weeks at a time.

With the explosion of flash memory and lithium-ion batteries, PAM evolved from an experimental method to a standard, scalable scientific protocol. Moored autonomous recorders, such as the High-frequency Acoustic Recording Package (HARP) and the Ecological Acoustic Recorder (EAR), allow units to remain underwater for months, capturing data year-round through the harshest seasons.

Historically, analyzing PAM data required a researcher to listen to each audio file and match acoustic signatures to specific species. Just five years ago, at the University of South Carolina Beaufort’s Marine Sensory and Neurobiology Lab, students manually reviewed and annotated 43,800 sound files from their 10 listening stations every six months. This effort required roughly six months of processing time before spectrograms could even be evaluated to decode broader acoustic patterns.

Advances in recording technology have overwhelmed researchers with massive data repositories filled with sound files that now scale into petabytes—a digital storage unit equivalent to 1,024 terabytes. If manually reviewing 30-60-minute analog recordings was once manageable, sifting through streams of data points from high-resolution cameras, satellite tags, and hydrophones is no longer a viable strategy.

It was there, amid a ravaging pandemic, that scientists met to problem-solve.

SECOORA AI Gateway

The five-year Augmenting Ocean Observing through Artificial Intelligence initiative achieved ground-breaking results in creating the SECOORA AI Gateway, an interactive resource hub for applying artificial intelligence and machine learning to PAM networks, images, and video data bottlenecks. 

And it started with training the model to differentiate between natural marine sounds and anthropogenic (human-derived) ones—teaching it what a boat’s sounds look like in a spectrogram. The project’s Boat Detection Workflow is an open-source method for scientists to monitor boating and angling activities in large offshore areas. No teams of data scientists, machine learning engineers, or advanced coders are required. The SECOORA AI Gateway provides cloud-hosted Google Colaboratory (Colab) notebooks—such as the Boat Noise Workbook—so that any lab, researcher, or state wildlife agency can run Python code using their own audio and benefit from state-of-the-art AI.

Because the workflow was not built in isolation but on the foundational Perch framework, the same system can be trained to untangle human activity (boat noise) from marine activity (whale songs and fish spawning) within a single audio stream. This is revolutionary, as it shows environmental stakeholders how boating behavior is impacting marine biodiversity in real time.

The AI Gateway was built and shared over four years with managers, advisory councils, and Ocean Science conferences, and it expanded the network to include the dynamic, globally engaged Conservation AI community.

Another of the AI initiative’s goals was to aggregate and standardize regional PAM databases, including those of SECOORA’s Estuarine Soundscape Observing Network in the Southeast (ESONS), Florida Keys National Marine Sanctuary (FKNMS), West Florida Shelf, BioSound, SoundCoop, and Gulf of America Coastal Ocean Observing System (GCOOS). This benchmark regional PAM dataset followed AI-PAM standards. The project also leveraged NOAA National Centers for Environmental Information’s (NCEI) cloud resources, maintained PAM data collection at critical fish spawning sites such as Riley’s Hump, Eyeglass Bar (around 7 miles south of Key West), and Western Dry Rocks, and supported the Florida Atlantic Coast Telemetry (FACT) network instrumentation. FACT is a grassroots network of underwater receiver stations that shares data gathered from monitoring tagged animals.

Through the Standardizing Marine BioData Working Group (SMBD), Dr. McEachron’s team also developed documentation for publishing taxonomic occurrence data in Darwin Core for OBIS (Ocean Biodiversity Information System) and GBIF (Global Biodiversity Information Facility) submissions. Occurrence data provide scientific evidence that a specific organism was present at a particular time and place. These instructional materials guide the annual “BioData Mobilization Workshops” where Marine Biodiversity Observation Network (MBON) projects and researchers convert raw datasets into the Darwin Core format. By unifying disparate datasets into a standardized, open-access format, scientists can seamlessly use the data to map biodiversity, track the spread of invasive species, and model the impacts of climate change.

Standardizing AI/ML pipelines from the outset secures long-term benefits for marine research, fully aligning with MBON’s vision and adhering to FAIR (Findable, Accessible, Interoperable, and Reusable) data principles. This approach is critical as bioacoustic AI tools are advancing rapidly, evolving from early models like Perch 1.0 to highly specialized, cross-domain architectures.

By 2024, the SECOORA AI Gateway featured coded notebooks using Google Research and Google DeepMind’s SurfPerch, a marine bioacoustic AI foundational model trained on ReefSet—an open-access, standardized dataset compiled by marine scientists globally from thousands of hours of underwater reef audio recordings.

For Dr. McEachron, finding like-minded collaborators outside marine science has been one of the project’s most inspiring highlights. He pointed to the expansive Community Resources directory of the AI Gateway as a testament to this cross-industry synergy. “It all links to people that are building foundational datasets and labeled data sets, and collecting them, and curating them, and they’re not ecologists—they are computer scientists.”

“That was the coolest thing…Thinking, ‘Okay, we don’t have to figure this out by ourselves; we’re not alone.’”

Using AI and Machine Learning to Repopulate Florida’s Reefs

One of the Augmenting Ocean Observing through Artificial Intelligence initiative’s most vivid outcomes is an Edge computing demonstration applying AI, machine learning, video, and passive acoustic monitoring (PAM) to real-world applications that can help restore our reefs.

In collaboration with The Florida Aquarium (FLAQ), SECOORA, and Axiom Data Science (a Tetra Tech company), Dr. McEachron and his team first developed an automated coral spawning alert system that analyzes underwater video feeds at land-based nurseries, enabling instant alerts and saving staffers from taxing, often unproductive overnight stakeouts. The system’s custom detection algorithm flags the cloud of a spawning event as a visual anomaly and transmits instant alerts to the researchers. 

This automation system marked a major milestone in augmenting the lab’s coral reproduction, as researchers have only a few hours to collect and cross-fertilize the eggs and sperm before they degrade. The FLAQ is a pioneer in historic efforts to induce spawning to breed rescued corals by replicating ideal marine seasonal conditions with programmed LED lighting that copies sunrises and sunsets, shifting tank temperatures from ~21°C to ~30°C, and a “lunar light” that mimics monthly lunar cycles. In 2019, the aquarium’s Coral Conservation and Research Center became the first to successfully spawn the critically endangered pillar coral in a laboratory, and in 2020, it was the first to induce spawning of grooved brain coral. 

The FWC team’s success with an AI-driven video detection system in controlled laboratory conditions led Dr. McEachron’s team to scale the technology further. They replicated this real-time video and image processing capability in the open sea with the deployment of an autonomous data buoy in the FKNMS.

“In the aquarium, I was sending a signal through an Ethernet cable,” said Dr. McEachron, “and then, we were trying to set up the buoy to send a signal through a cell tower.”

“We thought, ‘Let’s take it a step further and really apply it to management; let’s do real-time or near real-time detections, autonomously, in the field, with a tremendous number of applications, and let’s see how far we can take this,’” he recounted with excitement.

This approach addresses the significant lag time of traditional marine sensors, which require researchers to physically retrieve the equipment from the water before they can download and process video or audio data.

David Kochan, PhD, a FWC Associate Research Scientist with the Fish and Wildlife Research Institute and project co-investigator, explained that the deployment of the autonomous data buoy at Delta Shoal, a reef about 3 miles offshore from Marathon in the Florida Keys, had been a huge effort for a staff made primarily of biologists.

“The technical team had to learn new programming languages and electrical engineering to get the camera and data transmission systems working together,” said Dr. Kochan. “We also had to figure out the logistics of mooring the buoy and installing the camera on the reef.”

_{Left: Jade Lee, Research Associate in the Center for Spatial Analysis at FWRI, solders components of the underwater red lights for the buoy deployment in Delta Shoal. Middle: Underwater camera assembled in its housing with internal components. Right: Components of the underwater camera that include a Raspberry Pi mini-computer connected to a monitor for programming. All photo credits: David Kochan/FWRI}

The project, which also receives funding from the Florida Department of Environmental Protection’s Coral Protection and Restoration Program, envisions creating a blueprint for a low-cost modular system that any scientist can build and customize.

Because it took nearly a year to complete the permitting process, only a few autonomous buoy deployments have been made. “We are still in the testing phase for the different components, and so far the deployments have been about 3 days at a time,” said Dr. Kochan.

The team’s current goal is to track FLAQ-raised black long-spined sea urchins (Diadema antillarum) released onto the reefs. In the past, whenever researchers returned to look for them, the urchins had vanished after only one night. During one of their first buoy deployments, when the video feed was up and running, 22 hours of video were collected documenting the aftermath of the urchin release.

“Seeing the urchins move across the reef in and out of shelters when we reviewed the video files was amazing,” said Dr. Kochan. “It was the culmination of years of work.” 

Dr. Kochan explained that to record the video, the team built an underwater camera powered by a small computer board, which was connected to a separate computer board inside the buoy. In addition, they built custom underwater battery-run lights designed by another FWC group.

“Finally, the buoy still had to be there the next morning for us to download the data!” he added.

Just last month, Dr. Kochan and his team deployed the buoy, camera, and lights for 3 days and 2 nights. On each of the three days, Dr. Kochan and his team went looking for the sea urchins during dive surveys.

“The FWRI Restoration Ecology team released 10 aquarium-raised urchins on the reef,” said Dr. Kochan. “On the third day, we found 6 of the 10 urchins, 2 of which had been eaten.”

Because the camera remained stationary, the video did not capture the predators that ate the urchins, though a dive survey did identify the culprit: a batwing crab. With each deployment, the team perfects the design and learns exponentially more about the delicate habitats, informing their efforts to restore balance to the Florida Reef.

In a new funding cycle, Dr. McEachron and Dr. Kochan aim to continue refining the FWC data buoy by integrating a hydrophone and adding onboard AI tools to obtain real-time PAM data.

“One of my visions for the project is setting up the buoy with a hydrophone near a spawning aggregation site for reef fish and getting an alert when the fish start to arrive and reproduce,” says Dr. Kochan. This will allow scientists to mobilize divers and put them in the water at exactly the right times to conduct their surveys. 

“Our permitting goals for the future will focus on longer-term deployments at multiple sites.”

The data buoy is also being supported by funding from the Florida Department of Environmental Protection’s Coral Protection and Restoration Program and all work was completed under appropriate permits from FDEP, FKNMS, and the Army Corps of Engineers

The U.S. Marine Biodiversity Observation Network (MBON) is co-organized by NOAA, NASA, BOEM, and ONR through the National Oceanographic Partnership Program (NOPP).