Most computer games, as well as tabletop games, handle magic like an everyday tool. After you've seen a fireball or two, there's little "magic" in it; you might as well have a rocket launcher.
There's nothing really wrong with this, as such; but it doesn't really feel like.. magic, you know? I've pondered on this for a while, and I'll present a couple approaches to the matter. I don't know if these would make playing more fun or more frustrating, though.
First off, the magic in games is generally on/off kind of a deal. You get an effect, or you don't. Perhaps your spellcaster gets a health or stamina hit on a really fumbled spell. But that doesn't really feel like magic.. so let's think of it like this:
I've simply tossed some spell effects around a circle, to give you the basic idea. Let's make magic work in a bit more fuzzy manner. Let's say you try to heal someone, and fumble a bit; the target is healed, but also gets some freezing effect. Or you try to cause an explosion, but fumble in a major way, and it also starts to rain.
Heavy-duty, dangerous spells like ressurrection could be paired up with spells like disintegrate, to make their casting trickier. Curing a poison and causing poisoning might be inverted effects, or pretty close to each other. Failing to cure a disease might result in a very healthy frog.
How to end up with the different values of a spell then? Easiest solution would be to just add some randomness to it, which might be a good idea in any case, but to be fun, more control needs to be given to the players. What follows are a few ideas that I've got. Mix and match as needed, or think up your own variations.
The first idea I thought of was a spell research game. Kind of like the spell rune hunt in 'Dungeon Master', but taken to another level. Let's say you have this carving on a wall..
..and since it's partially illegible, you scribble a few guesses in your spellbook.
Of course, you should find several copies of at least the most common spells around the game, so you can put together the correct spell. As for the less common spells, well.. playing with magic is dangerous.
The main problem with this approach is that checking how close two strings are to each other is actually a rather complicated research issue. You could just say that ok, these letters are right, these are wrong, but what if only a couple of letters are swapped? And it's one thing to ask how near string A is from string B, and another to ask the distances from string A to strings B, C, D, E and F, to calculate the 'magic coordinates' of the spell.
One way to tackle this would be to score the strings based on a 'mastermind' scale, i.e. correct letters in correct place get a high score, and correct letters in wrong place get a partial score. Not perfect by any means, and requires lots of planning in the spell string definitions to achieve the kind of spell morphing we're trying to strive for.
Easier to implement is the reaction game. This could work even for a realtime game, as your spellcaster needs to concentrate to cast the spell. If the caster is disturbed, who knows what might happen?
As an example, the common "golf club meter". The difficulty of the spell could affect the speed at whe bar moves at; and nobody says it should only move in one direction..
Another source for a variable spells would be gestures.
The games Arx Fatalis by Arkane Studios and Black & White by Lionhead implemented spellcasting using gestures. I don't know anyone who actually liked the gestures in black & white, but still, they are a viable alternative, especially if you wish to make magic use rare. When detecting the gestures, it's very likely that you can also detect just how perfect the gesture was, and use that as the variable.
Any comments etc. can be emailed to me.