Now to my mind combat is an integral component of classic table top RPG play. War game roots, conflict requiring randomized resolution, immediately recognizable as risky, fulfills male power fantasies common among gamer types etc. Of course there's the question of what kind of combat one might have?
I am by inclination and lack of newer gameplay pretty stalwart in my devotion to B/X ODD OSR combat and gameplay, so I am not really talking about other editions or systems here (like story games of the tactical focus I hear is part of 3.5E on). Yet, within even simple (and deemed by some to be deadly boring) B/X style fights there's a lot of house rules and variations that effect play. In order to make them work it seems one has to figure out what one wants.
What is Combat for and How should it feel?
1. Lethal vs. Obstacle
How dangerous should combat be? LBB combat, Boothill and Warhammer Fantasy have reputations (deservedly) for having incredibly lethal combat mechanics. Failure or bad luck is likely to result in character death or permanent maiming. On the other hand, you have systems where loss in combat is merely a setback, resulting in capture, failure to pass an obstacle or the loss of resources.
Many modifications and house rules eliminate lethality, even when they're simply meant to add interesting variation to character types. In D&D every change from the Little Brown Books on has resulted in less lethal combat (at least that's how it seems to me). Modding on your own is likely to do the same. This isn't a bad thing necessarily - if you can mess up in fight and still survive things get less cautious, but this can be counter-intuitive and create odd results.
Example: If I were doing a scenario about vikings I might be tempted to make everything super lethal - this is about badass dudes in furs hacking each other with axes - yet super lethal viking battles would hardly encourage berzerker heroics and wild abandon, instead fights would become trading bow shots from cover, fleeing if injured and constantly trying to talk Rangvar the Bloodyhanded out of fights. Lately less lethality has been discussed in the OSR blogsphere - and I think the easiest/best regarded among people I play with way of doing low lethality is "It get's worse" where normal Basic/Expert rules are used, but you don't die at zero HP - it just gets worse.
What this Encourages ? Lethality effects fight avoidance. Incredibly lethal fights encourage players not to fight. Diplomacy, subterfuge, assassination and just walking on by all seem like good options when the there's a good chance of death in a fight. Low lethality encourages more fighting, because the consequences for bad decision/luck aren't as high. This can also make a game less brutally violent and more cartoon-like as Players are less likely to kill opponents when they know killing is rare and taboo in the game's rules.
What game feel ? When every arrow, bullet or sword slash might have your name on it all opposition is dangerous - a lucky blow by a goblin or the cursed sword of the lich king - it doesn't matter, either way they hit you and you're rolling on the death and dismemberment table, so you want to avoid them unless you can't. This encourages creative and cautious play, possibly to the point of being really boring. High lethality also may encourage a lack of character development or investment. I.E. when three PCs die in a session its hard to get excited about "Fighty IV the 1st level fighter." Low lethality makes fighting a more viable solution to problems that might best be solved another way - sometimes to the detriment of the game, sometimes not, depending on desired game feel.
2. Unpredictable vs. Managed
Unpredictable combat is highly random - things like critical hits, fumbles, exploding dice, or Carcosa dice (roll all the dice for damage, the system picks one for damage) encourage this sort of unpredictability. Easily combined with high lethality, but not exactly the same assuming getting knocked out of a fight isn't lethal.
Managed Combat is about resource management - the outcome is likely known in general terms, the question is how many of one's feats, resources and special abilities will one need to use to prevail or escape?
What this Encourages ? Unpredictability effects pacing, as one doesn't know which fights will be hard or easy - maybe the "boss" fight is a let down because of a lucky roll. It also gets players planning, attempting to introduce factors into fights to eliminate the unpredictability or maximize one's advantages (also a result of complexity). Unpredictable fights also mean that its hard for a GM to scale a dungeon - a monster that is "too strong" for the party might fall to a lucky exploding dice pistol shot and conversely a harmless wandering monster might kill a PC with an unlucky critical.
When the fight is managed and the system allows easy or accurate gauging of a fight's outcome planning or tactics become less important and pacing is steady - the party can guess how many more tussles they can handle before their resources are exhausted and they have to retreat. This can make things easier for GM and makes strategic play more accessible because the question isn't how a fight will go, but if the party can successful figure out the puzzle of how to fight through to a larger goal before they exhaust their resources. Trick monsters (weapon immunities, odd special abilities) are also a lot easier to introduce into managed systems, as one knows how hard they will be to stop without figuring out the trick, and can judge how much combat the party can handle and so how long they have to figure out the trick.
What game feel? Unpredictability may encourage a more freewheeling sense of play, but because any conflict can go either way, pacing is determined by the dice so setting hard goals and defining defeat and victory may become harder. Perhaps it's ideal for picaresque play.
Managed combat has the ability to feel epic and drive stories if done right. The party knows they can take on the big bad, and has an idea how bad the fight will be, but has to struggle and plan to get there. I think it might be especially effective for military style scenarios or games where resource management is key.
3. Simple vs. Complicated
Simple combat has only a few rule, a lack of fiddly parts. OD&D combat is very simple for example - two to three rolls per round of combat per combatant (including initiative). The number of bonuses involved are also limited. As one adds things, from Strength bonuses, to feats and tactics combat gets more complicated - and combat takes longer. I would argue anything added to combat makes it more complicated. Though that isn't bad, complexity helps create character roles within the party - where everyone hits on a 12 or better and does 1D6 points of damage, why put the fighter in the front? Complexity creates drama and tactical choice - though too much can be overwhelming and simply make the game into tactical combat.
What this Encourages? Complexity and Simplicity effect the Importance of Combat - If combat is simple, it's unlikely to be an interesting central activity, and moreover it shouldn't take up most of the time in the game. It doesn't matter how lethal it is, puzzle traps are deadly to, but when combat is three rolls the puzzle trap will be seen as more of the central challenge.
Complex combat makes the game about combat, but it also makes the occurrence of combat something that can't just be random, because who wants to spend most of the session fighting that wandering monster rather than getting into the meat of scenario that the GM has built and players are excited about.
Final Thoughts and Examples
What one is looking for depends on the game one wants to play, a game about high school bullies and crushes shouldn't really have a lot of lethality, whereas one about World War I should be instantly deadly all the time with the goal of capturing the absurdity and horror. Note that both these games would want to have unpredictable combat, because sometimes the nerd has to be able to get in a lucky haymaker or go Zangif kid, and sometimes a soldier walks away unscathed when everyone around is annihilated by a shell. Simplicity I'm not so sure about, I suspect high-school troublemakers might be a bit the marital arts spectacle so complexity and skills are appropriate, each fight being a big deal, whereas in trench warfare land (I am starting to suspect each player gets a group of characters there) it's fairly constant and should be quick and unexpected. Thus we have two combat feels we can talk about Non-Lethal, Unpredictable and Complicated vs. Deadly, Unpredictable and Simple.
Now what would something weird be good for - A Lethal, Predictable and Complicated Combat for example? Perhaps a battle mecha game or naval battle scenario. If your ship goes down, or your mecha brews up - it's over. Yet these scenarios would be combat focused and there's a lot of cool fiddly bits before one gets to the inevitable part, but then again in either case it's also partially a management game - do you use the store of Armor piercing shells on an enemy destroyer - surely not, you use the homing torpedoes!
For two campaigns I am running I think the right levels are as follows:
HMS Apollyon - Moderately Lethal, Unpredictable and Simple (It's currently Non-lethal, Unpredictable and Complex). This needs to change. All those lovely leveling skill tables make things hard, and are moot in a flail-snails game anyhow, but mostly it's the recovery rules and the rate of fire of guns that are making trouble. I can't predict how deadly the party will be because of the variability of the characters and the sheer nastiness of exploding dice damage.
ASE - Mostly Deadly, Unpredictable and Simple - I think I've got this down so far. Though no one's died, this is partially because the players have been lucky.