When I GM an RPG like D&D my players frequently describe them as tense, scary, creepy, or sometimes even being a bit like "Scooby Doo". I'm quite pleased with this since I set out to include those kind of elements in my games. I thought I'd share some of the things I've learned over the years about what helps add a touch of horror to an RPG.
Description vs Dice
It's the little things that create a creepy atmosphere in a game. Try and make sure you're giving your players enough descriptive language to properly imagine the scene in their minds eye. You don't need to give them long winded descriptions or read from pre-written box text, but you should do more than give them bare bones accounts of thirty foot long hallways leading to the next standard door. Try to include all 5 senses if you can. Cold breezes, damp odours, and echoing sounds can make an otherwise basic description feel a bit more real.
Avoid rolling the dice for checks to find traps and secret doors. Ask the players what they're doing and answer their questions about the scenery they're examining. This is a great opportunity to go into more detail with your descriptions since the players have asked for more info anyway. The more you are describing the scene and the players are telling you what they're doing there the better. Lots of dice rolling, including combat, can take away from this a bit. Combat is the action part of a movie, as opposed to the scenes where you build tension.
Suspense vs Surprise
Make a saving throw! Roll for Initiative! You take 13 points of damage! These don't make your game as scary as when players are nervous about the potential for those things to happen. The sudden appearance of a slavering monster doesn't leave your players much time to be afraid of it before they switch gears to thinking about the combat system and how to defeat it. Monsters that you can hear approaching, or see climbing out of their coffins are much better - the players have time to think about what might happen if they face the monster. Having to decide whether to edge carefully around a pit when you know there's a chance of falling in is much scarier than a covered pit trap that you suddenly fall into. A sudden explosion of yellow mold spores isn't scary. A sickly yellow mist that fills the hallway ahead is.
If you are exploring a dungeon and you find a room with a closet door, open it, and a gorilla jumps out, you're going to feel like it's arbitrary and the tension of the scene will be gone. If instead you receive clues before opening the door that gets you ready to accept what you find, it'll work much better. The room has a strange musky smell. You hear some scratching sounds from behind the door. If you listen at it you might hear breathing, and so on.
Mystery vs Mundane
When players know too much about the mechanics behind the monsters they stop being scary. Once they've memorized all the stats, read the ecology, and know all of the game fiction about a Bugbear or a Gremlin it's not scary anymore. It's an exotic animal at best, and just a pile of game rules to be predictably dealt with at worst. If the players think you will only give them encounters (monsters, traps or hazards) that are "level appropriate" and manageable, they're not going to think twice about charging headlong into the fray. If there is some mystery to what they're encountering it adds a lot of tension to the game.
Clues vs Chaos
The flip side of mystery is that it can't just be random and arbitrary. If it is you'll break your player's suspension of disbelief and they'll stop buying into the reality of the fictional world and the suspense of the game. This is exactly the same thing that happens in a movie when you think "what? come on, that doesn't make any sense!" - when that happens you're "out of the movie" and it's the same thing in a tabletop RPG. The more your world feels internally consistent the more players will buy into it. Don't confuse this with "realism" - you can have suspension of disbelief in a fantastic world.
How did the zombies get in this room, and why aren't they attacking the dwarves playing cards in the last room? Sure you can make up some explanation for that, but if your players are thinking about that at the table they aren't thinking about the zombies being scary. On the other hand, passing by some recently unearthed graves before you run into the zombies gets you ready to accept their appearance.
Fleeing vs Fighting
As soon as players commit to fighting with a monster it becomes a little less scary. It might still be a tough opponent but that's just a relative level of challenge. Combat typically runs much slower than the other parts of the game, so the pacing will also change and that will also affect your level of tension. It's almost an inverse of what you'd see in a movie - where fight scenes are often the fastest paced on the screen, they' can be some of the slowest paced in an RPG.
If your players get an opportunity to choose whether to fight a monster and decide to run away from it on the other hand, the tension increases. The players (not just the characters) have a fear of what would happen to their characters in the combat encounter. So whenever possible try and make sure the PCs have a choice about combat, and that running is seen as a viable option.
Again, this won't work if your players think you'll only give them 'appropriate' challenges and/or you'll fudge the dice to keep their character alive if things go bad for them. That's when you get players attacking everything without a second thought. Having some player controlled retainers along with the party can be a good way of letting the players see the results of bad choices without taking their main character (and thus the player) out of the game.
I find this works best when there's a time limit under which the players need to decide whether to flee or not, and not simply break off from combat whenever they feel like it. Perhaps the monster is breaking down the bars of it's cage to get out and attack the players. Skeletons are pulling the heavy stone slab off their crypt. Or Animated Statues are heading down the hall towards the room leaving the players a few moments to decide between fight or flee.
Characters in scary movies and games spend more time running from monsters than fighting them...
