'Shazam for birds' app detects songs with incredible accuracy

Merlin Bird ID's machine learning-based Sound ID engine gives impressive results when it comes to recognizing individual bird songs despite noisy environments.

On the market today, there are many applications to identify birds based on images and sounds, with varying degrees of accuracy. There is an application that identifies every bird call as the sound of the northern mockingbird, which is a bird. specializing in mimicking the voices of other birds. And there are other apps, like Merlin Bird ID of the Cornell Lab of Ornithology , that are the first choice of bird lovers and researchers alike when it comes to identifying a bird based on photographs. Recently, Cornell Lab has expanded their service and provided users with a bird song recognition function. So what is so special about this machine learning-based sound recognition tool that is dubbed "Shazam for birds" ?

This application can recognize birds based on images and sounds, with a high degree of accuracy.

Experienced birders can easily identify a bird by listening to its song, but it can sometimes be quite difficult and time consuming and requires experience. The purpose of Merlin Bird ID is to solve this problem. " The cool thing about Merlin is that it's a non-critical companion, someone who can tell you that you've heard a sparrow sing 300 times and still be happy to say it again. it's like the first time, " said Drew Weber, Merlin Bird ID project coordinator.

Gizmodo reporter Ryan Mandelbaum tested the app over the weekend at Prospect Park in Brooklyn to make sure the last successful guess wasn't due to luck. Although Brooklyn's location and ecology make it a top birdwatcher's destination during the spring and fall, only a handful of songbird species are present here in the summer, so The application may have the advantage of only having to identify a few of the most common types of birds.

Mandelbaum stopped at a tree corner at the southwest entrance to the park, which is a rather noisy place, and spotted a Baltimore golden boy singing in a pine branch. He activated the Sound ID feature, pressed the record button, and held the phone high above his head. The app displays a spectrum image - that is, a graph of the frequency it records over time - and immediately gives the result "American robin".You think it's wrong, but there's actually an American robin singing behind Mandelbaum. He tried again, and this time a house sparrow began to speak. The application displays a picture of a house sparrow! He tried one last time, and just as his golden bird sang, a swallow came to disrupt the group; The application once again ignored the golden uncle and correctly identified the swallow's voice. Obviously, although it did not recognize the original target as expected, the above three correct guesss showed that the Merlin Bird ID was extremely sensitive. . However, Mandelbaum said he was a bit frustrated at not being able to identify the golden eagle, which is a very common bird, in such a simple setting.

Overall, Merlin Sound ID works as expected.

Deep into the park, Mandelbaum kept the app open in the background and jotted down any other birds he encountered. It successfully identified a northern cardinal through its "pew-pew-pew" song , but when the bird started singing in a higher pitch, the app immediately said it identified the bird's song. osprey, which is a large fish-eating falcon. The high-pitched, high- pitched "seee" of North American cedar waxwing birds appears in the spectrum, though unrecognizable, and is replaced by a warbling vireo (another North American bird species) appears when it begins to sing from a distance.

Overall, Merlin Sound ID works as expected; You can only faintly hear the song of a bird, the application immediately identifies whether the bird is of an uncommon type.

However, Merlin Birth ID is more than just a sound recognition app ; it is the result of tens of thousands of bird lovers and researchers night and day sending to Cornell's Macaulay library millions of recordings of bird songs through the eBird app over the past few years. Considering the huge amount of data, Weber and Macaulay library research engineer Grant Van Horn, along with other members of the Cornell Lab, wondered what it would be like if they created a bird song recognition feature for the app. using Merlin Bird ID?

In fact, sound recognition is a form of image recognition, according to Van Horn. Caltech and Cornell Tech engineers assembled a set of bird image recognition neural networks, with data from photos from the Macaulay library, to create the Merlin Photo ID feature. Sound ID converts the sounds into spectral images, processes them, and then traditional computer vision tools compare these spectrograms with those in available bird song recordings.

Playing a key role in the identification process is a "massive" training dataset, which requires the contribution of public researchers. Like the recordings of birdsong in the background that Mandelbaum obtained while walking in the park, the records of the Macaulay library also often contain many songs of birds around the user. A team of annotating volunteers sifted through a collection of spectrograms used to train more than 400 North American bird species, identifying and labeling each individual species' sound. As a result, they obtained a dataset of about 250,000 annotations, each label corresponding to only one species. App users can either directly upload a file or recording of bird songs, or the app will filter out each type of bird it hears from every 3 seconds in the recordings.The team also trained the algorithm with a large amount of background noise, including Google's extensive AudioSet dataset, so that the app knew what non-bird sounds are like.

As noted above, the market still has many other high-quality bird song identification applications - Cornell Lab, together with Chemnitz University of Technology, also develops the BirdNET Sound ID application . These apps, however, have distinct purposes: BirdNET is primarily used as a research tool for scientists, while Merlin is a community-contributed bird identification application, including functionality. visual identification, Q+A, field guides, and data from the public science database eBird related to bird habitats, sounds, and images. Data from eBird also helps with Merlin Sound and Photo ID features; they rely on public researchers' records of surrounding birds to make more accurate suggestions.

Merlin Sound ID still has many points for improvement.

Merlin Sound ID still has many points for improvement. There are about 10,000 species of birds, and the app can only identify about 400 species. Short songs are also difficult for the application because they can sound extremely similar to other species, and the application can mistake low-frequency songs at a certain level as background noise. But as datasets improve, so too do machine learning algorithms and application capabilities in general.

Van Horn is excited about the potential for its data sets and machine learning models. He plans to use this model in other areas of the Cornell Lab, like bird cameras and stabilized recording. Weber says that perhaps they can use this model to tell users what birds are flying around the city during migration season. Perhaps they could use the model to identify bird videos, too. Van Horn also said that this algorithm is only used in wildlife and is generated from data that users agree to give to Cornell through eBird, running on users' phones without sending data back to Cornell. .

• The acoustic detection feature built into one of the world's most popular bird identification apps is sure to be something bird lovers have been waiting for, and according to Mandelbaum, after trying it out, he's confident it works pretty well. Experienced birders may find their ears to be a little more accurate than the app, but at least for amateurs, this tool will be a great addition to the expensive hobby. their time and effort.