The research, published in Science, also explains why those who recover from the virus have much worse symptoms if they catch it again.
Dengue fever is a viral infection spread by a mosquito bite. It is a major cause of illness worldwide, and cases are on the rise.
There is currently no licensed vaccine or drug treatment.
The researchers, based in the UK and Thailand, took blood samples from infected volunteers.
They found antibodies produced in response to the virus do not do a very effective job.
Rather than neutralising the virus, they actually help it infect more cells, springing into action when a person is infected a second time by a different strain of the virus.
This phenomenon accounts for why a second bout of dengue fever can be more severe and dangerous.
It also provides new insight into how to design a vaccine for dengue fever.
The authors of the Science paper say vaccines that steer clear of a key viral protein involved in the immune response should be the most effective.
Professor Gavin Screaton, head of the Department of Medicine at Imperial College London, led the study.
He said: “Our new research gives us some key information about what is and what is not likely to work when trying to combat the dengue virus.
“We hope that our findings will bring scientists one step closer to creating an effective vaccine.”
Professor Screaton said one of the major challenges was developing a vaccine for a virus that has four very different strains.
“The need for vaccines is enormous but the challenge is that in this case you need to hit four bugs all at once down a single needle,” he added.