Horror is an interesting thing. It relies on our perception of the world. It relies on what we consider normal, what we consider sacred, and what we consider so mysterious and frightening.
If we consider a setting like the World of Darkness, set in modern times, we have an enormous assumptions as players, because the setting is our world. We know that in the game world the laws of physics work in expected ways, and that we can easily sympathize with characters, since they are humans, just like us. We can understand how our human character feel when they are hurt, feel love, feel pain, and witness horrific, though mundane tragedies. We can sympathize with the sense of loss, hate, and fear. And all of these things are rooted in our understanding of the world. And our world is mundane.
The Joker: It’s the schemers that put you where you are. You were a schemer, you had plans, and uh, look where that got you. I just did what I do best. I took your plan and I turned it on itself. Look what I did, to this city with a few drums of gas and a couple of bullets. Hm? You know what, you know what I noticed? Nobody panics when things go according to plan. Even if the plan is horrifying. If tomorrow I tell the press that like a gang banger, will get shot, or a truckload of soldiers will be blown up, nobody panics, because it’s all, part of the plan. But when I say that one, little old mayor will die, well then everyone loses their minds!
Horror of course does exist in our own mundane world. Terrible accidents occur, which shatter our sense of safety, be it within our own home, our country, the mode of transport we use. Of course, the events may not be directly effecting us, but if the events are so extreme, so unexpected, that we experience the horror of what has happened. Contrast that to expected events, such as soldiers dying in war, or criminals being shot by police, though tragic, we don't fee the horror, since such things are part of life - bad things happen in those sorts of situations.
Horror of course is different to terror. From wikipedia;
Terror is an emotion that a person gets when they are in an immediate fear. There is a feeling of revulsion that is absent in terror. Terror is provoked by danger and menace, as when someone suddenly find their self in a jungle in front of a tiger. Terror is the feeling that is experienced by people confronted with terrorists, robbers and murderers. Terror activates the sympathetic nervous system and prepares the body for a fight-or-flight response response.
On the other hand, horror evokes a feeling of disgust and is more disturbing and psychological in nature. With a feeling of horror, a person may have nausea or a revulsion, as one might feel when they see something bizarre and horrifying, such as worms inside wounds of an organism, facing a deadly animal and their phobia, and even the supernatural.
Definitively speaking, terror would bring panic, an overwhelming feeling of fear and anxiety to the person (bomb threats, death threat, being held hostage, and, enduring natural disasters and terrorism). Whilst horror gives a person an overwhelming and rather painful feeling caused by something frightfully shocking, terrifying, or revolting (corpses, zombies, grotesque images).
So of course we can experience terror from mundane sources and what can be normal events. But horror is more rooted in our world view being shattered and causes us to reject this revelation or situation.
Mundane horror can of course be due to people, who we expect to be able to sympathize with, performing bizarre and gruesome acts. A perfect example of this are bizarre serial killers, who take to mutilating their victims in strange ways, cannibalism, and other foul acts.
So of course the supernatural is related to horror, and is perhaps even more powerful through the horror it induces. Finding a person dead, perhaps near some train tracks, is shocking. But in the normal world we may well reason that it was the result of a terrible accident. Or at worst a mortal killer left the body. However, if there is some sign of the inhuman, the impossible, such as organs having been removed without any sign of entry or exit from the body, the body being impossibly drained of blood, then we have a sense of horror. All the normal expectations of reality have been broken. What should be impossible has invaded our world, and desecrated it. We are faced with inhuman entities and powers, some which are foul mockeries of ourselves, and some which are so strange and alien, that they cannot be reasoned with and so we are repulsed by and wish to be as far away from. Vampires are a mockery of humans, and also represent some of our worst crimes, such as blood drinking and the violation of our bodies and self control. Werewolves are a violation of the barrier between nature and ourselves and perhaps our own sanity. The list continues.
But in the case of World of Darkness games we are often playing characters that are supernatural in nature. This means we, as Storytellers, should ensure that we reinforce what is considered mundane for the characters, and how what they do in horrific for the mortal world. As a vampire, over time, feeding becomes a normal event. The horror of drinking blood is replaced with a degree of acceptance. We understand that there others just like us, and that they too feed on blood. We begin to accept that there are many strange powers at our disposal, and we come to accept the weird things vampires can be capable of. But there is still horror. We can maintain a sense of horror in the vampiric condition by comparing our player characters to other vampires. Some vampires have learnt some truly vile powers that cause foul afflictions and monstrous transformations. If we are playing mages, then some mages can be shown to be so full of hubris that they are willing do violate the laws of reality, and desecrate souls and humans, to get what they want. Some changelings are so unhinged, and disconnected from reality, we can reveal through their dreams, that when they are chewing the fat from a babies leg, they think they are eating a sumptuous leg of lamb.
This sense of horror then relies on establishing what is normal, what is not normal for your character but may still be normal for the rest of their society, and then what is possible for one of their kind but which is rare and extreme and unknown.
Horror for supernatural player characters can also be portrayed in another manner that shows how even though are no longer human, they still retain a semblance of their humanity, or a sense of morality, or what is acceptable for their world. A vampire, who feeds on humans, but tries not to kill, keeps to the masquerade, and tends to use rather normal powers (normal meaning that he is not transmogrifying people, or turning into liquid shadow, or summoning the dead, or creating balefire) may find the acts committed by mortals to be horrific. A vampire may have specific habits, killing only certain people, reasoning when doing so is acceptable, but also knowing that it should not get out of hand. All the while this same vampire may watch from a far, and keep safe, his mortal family, be they direct members, or just the current generation of the family. So when presented by the seemingly random, bizarre and vile murders of a serial killer, the vampire player has a sense of horror - this horror being that even humans are capable of worse acts than even his vampire character, a character that is considered a monster. The horror of ones own existence, even when it has become the norm, can be reinforce once more by having the horror of it being revealed by a sympathetic mortal character - the player character is finally reminded of a portion of his humanity and in turn how far he has fallen from it.
Horror for supernatural player characters can also come from other supernatural sources. The trick here is to ensure that the players divorce themselves about what they know about the setting. You may or may not be running a World of Darkness setting where all other supernatural creatures are exactly portrayed as they are within their own game. Your vampire game may have magic using mortals. But are there the Traditions and the Technocracy? Are Werewolves creatures innately tied to the Umbra. Players knowledgeable about the setting my make use of this out of character knowledge and so assume certain things. This can mean that what should be a horrific event, instead becomes something mundane as the player says 'Oh it's the work of changelings'. If this happens, the horror of the scene can be shattered. This is where it is important for the Storyteller to 'own their game'. Make it your own. Don't follow the metaplot. Change up what things are considered myth, rumour, fact. It will help you retain your sense of mystery and what is considered strange and unusual, things that are essential to create an atmosphere of horror. If this means stopping players reading books, do it. If they need to know stuff from a book, copy paste or write some bridge notes. In a setting like CWoD this is perhaps more of an issue as players may have bought into the setting, and so read a lot about it as they have followed the metaplot story. But even CWoD, just like NWoD, with a bit of work, can be remixed to re-establish the mystery and otherworldliness of entities and events, and the horror of these things.
The other important thing to remember is that familiarity breeds normality. So if your players are constantly fighting antagonists that are using a plethora of strange powers, and this is every story and every session, the impact of the weirdness of these powers and of vampires is dulled. Instead use horrific powers sparingly, try to build the tension until the full shock of these things can have the most impact, perhaps by luring the group into a false sense of security.
When it comes to running horror games in non-modern settings, we have to be consider once more what is mundane in our setting. In World of Darkness we have many of these options, such as either historical settings, such as Dark Ages, or Noir, or Victorian. But in World of Darkness: Mirrors we are also shown ways that the World of Darkness can be used applied to more fantastical, near future, or space opera settings. When considering any of these settings it changes what is considered normal, possible and supernatural. In scifi settings technology often means that certain events can be explained, or mimic supernatural powers. But I think often the biggest challenge is running settings that fall into 'fantasy'.
Fantasy settings are varied and diverse, ranging from high to low, gritty and dark, or all gleaming towers. There could be elves, orcs, magic, gods, demons and more. And this can make it difficult to portray horror. If magic is commonplace, the horror of such things as the undead, is reduced, because while the undead are still fearsome enemies, it is easy for characters to reason that the creatures they are facing are the result of magic, and so can easily be thwarted with magic. And so there is a need for establishing what is mundane magic, and what is not. If magic is a rare thing, and all the strange creatures have been banished to the darkest parts of the forests, then characters will have grown up with certain assumptions. Talk of gods, dragons and other beasts will be considered myth and folklore. The result is that while magic is a part of the world, it makes its presence and actions either miraculous, or in the case of horror, unnatural and otherworldly. Again, familiarity breeds normality, and in turn it means that all manner of terrifying, and rightly horrific creatures, are treated with a degree of contempt, as they can easily be dealt with by the party 'priest/wizard'.
One of the best things we can do in such fantasy setting then to reinforce the horror, is to reinforce the similarities between our world and the setting, so that when the horror occurs, we can envision ourselves in the scenario. The other important thing to also focus on, is that no matter the reason for the horror, be it magic or science or the supernatural, we should never forget that these creatures and magics and items represent a perversion of something more fundamental than even the basics of the setting - the soul. If we are reminded of this, and how many powers, magics, and creatures, represent the way that the soul has been damaged, twisted, coerced and destroyed. And this is what horror is ultimately about - how external threats present a danger to our very sense of being.