The mongoose-like creature has been called Durrell’s vontsira (Salanoia durrelli) in honour of conservationist Gerald Durrell.
Scientists found the creature in the wetlands of Lake Alaotra, the largest lake in Madagascar.
Its marsh habitat is under pressure from invasive species and pollution, and the team thinks it could be one of the world’s most threatened mammals.
They describe the cat-sized animal for the first time in the journal Systematics and Biodiversity.
A team from the Durrell Wildlife Conservation Trust first saw the mammal swimming in the lake during a field trip in 2004. Suspecting it might be a new species, they photographed it so their zoologist colleagues could examine it more closely.
A team then returned to the site in 2005, caught one of the animals and took detailed measurements and blood and tissue samples.
During the same expedition, the scientists sent one dead specimen to the Natural History Museum in London.
There, zoologists were able to compare the creature with its closest relative, the forest-dwelling brown-tailed vontsira, and finally confirm its identity.
Durrell’s vontsira and the brown-tailed vontsira are similar but have very different colouring, explained the Natural History Museum’s Dr Paula Jenkins, a member of the research team.
The discovery of mammal species is uncommon and finding a new carnivore species is “particularly unusual”, Dr Jenkins added.
“Durrell’s vontsira is incredibly rare,” she said.
“We know of only two animals in the wild. It has only been found in the wetlands of [Lake] Alaotra in Madagascar, so it lives in a very small area and is consequently vulnerable to the pressures on this threatened habitat.”
The researchers still know very little about the animal’s behaviour and biology.
They think it may be a mongoose-like creature specifically adapted for an aquatic or semi-aquatic environment.
Professor John Fa, chief conservation officer at Durrell, told BBC News: “If that is the case, it’s very interesting indeed; mongooses normally live in arid or forested areas.
“We think it feeds on fish and small mammals in the lake and if it’s a mongoose that catches fish – that’s very unusual.”
The scientists hope to return to the lake to carry out a more detailed, systematic trapping study, and possibly to tag and follow the small mammals to see if their habitat is confined to the lake.
“This just shows how much biodiversity Madagascar is still throwing at us,” Professor Fa added.
Since 2006, new mammal species found in this biodiversity hotspot have included three new species of mouse lemur (Microcebus jollyae, M. mittermeieri and M. simmoni) and a bat (Scotophilus marovaza).
But the last carnivore discovered on the island was Grandider’s vontsira (Galidictis grandidieri), described in 1986.
It is classified as Endangered on The IUCN Red List.
The team also included researchers from Nature Heritage in Jersey and Conservation International (CI).