I read a book last year, 'Into the Drowning Deep' by Mira Grant, and it completely redefined scary mermaids for me. The suspense wasn't just jump-scares. It was this slow, creeping dread built on scientific plausibility. They're not singing sirens; they're pack-hunting apex predators with a biological reason for luring humans. The tension comes from the characters realizing, piece by piece, that every old sailor's myth was a garbled warning about a real animal.
What works so well is the environment. The deep ocean is the ultimate locked-room mystery. You're trapped on a ship or a crumbling rig, surrounded by an element you can't survive in, while something that belongs there watches you. The suspense is in the distorted sonar pings, the shadows moving just beyond the submersible lights, and the awful understanding that you're not at the top of the food chain out here. The mermaid doesn't need to be supernatural to be terrifying; it just needs to be perfectly adapted to a world that will kill you in minutes.
That biological angle also plays on a deeper, almost visceral fear of being prey. There's a scene where a character realizes the 'songs' are complex hunting calls that manipulate sound waves. That moment of intellectual horror, where curiosity turns into the certainty of being hunted, is where the real thriller engine kicks in. It's less about a monster jump-out and more about the dreadful confirmation that you are, definitively, on the menu.