Gameplay
It's uncertain how this gameplay mechanic will work, but the name hints at the Wizard's lore, and also its functionality.
Most likely each spell cast will cause "instability" due to the dangerous and volatile nature of the spells the Wizard wields.
Hypothesis 1
Too much instability will hurt her or make her unable to perform certain tasks or perhaps modify how a spell is cast or what spell is cast (eg: casting Disintegrate will produce Spectral Blades instead). It could just be a power gauge inspired by the now defunct Mana Burst, dealing damage based upon how much/little control she has over the instability. Remember Mana Burst was a passive skill from the previous Arcane Skill Tree, Tier 4. It was listed with a description that supposed to add 25% damage to any spell casted by a Wizzard with a full mana orb. Now, there isn't mana anymore for the Wizzard, and probably they removed this skill too. Perhaps the new system would be something like this spell: just making more damage from spells if you got high Instability, and less if you got and empty orb. Also, it would increase the damage taken if you got tones of Instability.
This theory has merit because most resource systems use one or another way of gaining a resource and using skills by spending it.
Hypothesis 2
An alternate form would be that the instability somehow helps the Wizard to cast spells better in some way. The more spells cast, the instability will actually help by causing ripples in time or similar effects.
The only problem with this is that it could be quite unbalanced if the Wizard would continuously gain power the more she did, especially in PvP.
Hypothesis 3
Instability could also take the form of a meter that fills up as the Wizard casts spells, representing the buildup of excess unstable magic as she pushes her limits. When this meter (or, more likely, orb) fills to a certain point, the Wizard begins taking damage from each spell while increasing the damage of that spell by the same amount. For example, a spell deals a base of 1000 damage. When the Instability orb reaches 30% percent, when she casts that spell, she take 2% of that spell's damage, but the total damage that spell deals is also increased by 2%. Then the Instability orb would fill another X%.