For me it’s hard to say for sure, because my scariest book wasn’t meant as horror, but Communion by Whitley Streiber is definitely responsible for the most sleepless hours.
Probably isn't really horror but I've never felt more uncomfortable than when I read House of Leaves. Just being alone in an apartment felt creepy while reading certain parts.