After posting about the importance of running from dangerous monsters in our games, Rob (who's a player in our Weird West campaign) mentioned that in the 4e Dark Sun campaign he's running he's sure his players wouldn't run if he introduced that kind of threat in his game and it would end up with the entire party being killed.
So why will some games see characters running frequently, and others where every single encounter is treated like the Battle of Thermopylae? I think there's a number of reasons, and a number of steps you can take to address them (if you choose to).
If players believe the object of the game is to win each battle they are presented with, rather than choose when to fight or flee to achieve victory in the overall adventure, then they will not choose to "lose" the game by fleeing from an enemy.
The longer the combat takes, and the more prep work that goes into setting it up (including battlemats, miniatures, custom statblocks, etc) the more players will tend to believe they are expected to engage in combat when presented with an encounter.
Experience rewards that are given for defeating enemies instead of treasure, exploration, or story based rewards, will also discourage running from a battle as it can be seen as "losing the game" to many players. Furthermore, when the expectation is that each encounter will be balanced such that the players have a reasonable chance of victory, then not even trying to "win" the encounter could be seen as the mark of a rather poor player.
And if the GM is rolling dice behind a screen then players will tend to believe the GM is a safety net rather than an unbiased referee for the battle. If things are going against the players too badly the GM will "correct" the dice so as not to wipe out the party. All the more reason to charge the hideous monster!
So what do you do to have encounters where the characters run from an overwhelmingly dangerous enemy rather than stand their ground and get wiped out, or (much worse) have you fudge the dice and stats to let them win against an enemy they shouldn't have?
Be clear about how the game works and don't back down to let the players win.
First, make sure everyone understand what genre the game is. "Fantasy" or "Medieval Fantasy" isn't specific enough. Is it heroic fantasy? SUPERheroic fantasy? Horror Fantasy? Survival Horror Fantasy? etc. Talking about books or movies with the genre and tone you want for your game can be a quick way to get everyone on the same page.
Don't roll dice behind the screen. Or rather, don't roll them behind the screen for numbers/results you're immediately going to give to the players. Attack rolls, damage, saving throws, morale checks - do this all on the table. If you want your players to make choices about fighting or fleeing they need to be able to base that on consistent information: the statistics and probabilities of the game and dice.
Warn your players encounters are not balanced. If you want players to run from some encounters tell them up front that you are not balancing things to ensure a fair chance at winning. Whenever an enemy is encountered the players will think about whether this is one they can defeat.
Delay putting down the battlement and miniatures. If you are using a tactical grid and minis or tokens for combat, avoid putting this on the table until the players have decided to engage the enemy. Movement on a grid suggests tactics and order. "OMG RUN AWAY" suggests a bit more chaos.
Reward players for more than just winning battles. If the players can "win" the game or simply increase in level by means other than combat then many people will think twice about throwing their character into the meat grinder if it's not necessary.
If players are still not backing down and you're not ready to let the dice fall where they may and see some player characters devoured by slathering beasts then consider introducing some henchmen, retainers or minions. Make them useful, and let the players have a lot of control over their actions. They might still be NPCs but they'll follow orders from the PCs. They'll be great assets to the players… unless they get them killed by not keeping the party out of fights they can't win.
When the players understand all the risks and STILL choose not to back down it's much more dramatic, and while it may end in their defeat it won't feel as much like a random TPK but rather an epic battle they chose on their own.
