Gillian Flynn's 'Gone Girl' set a standard, but I actually find Paula Hawkins' 'The Girl on the Train' does the psychological tension better for me. Nick and Amy's twisted game feels a bit theatrical, while Rachel's booze-fogged, unreliable narration digs into something more mundane and terrifying. You're never quite sure if she's a witness or a perpetrator, and that grey area is where the real chill is.
Lately, I've been more drawn to stuff that doesn't rely on a big twist. Iain Reid's 'I'm Thinking of Ending Things' is basically a masterclass in sustained, creeping dread. It's less about a killer chasing someone and more about the slow erosion of reality inside a car. You finish it and just sit there, questioning everything you just read, which is a special kind of psychological workout.
On a totally different note, Tana French's Dublin Murder Squad books, especially 'In the Woods', build tension through memory and trauma. The procedural elements are there, but the real hook is the detective's own crumbling psyche as a childhood event bleeds into his current case. It's a slower burn, but the payoff is a profound unease rather than a jump scare.