Area of Effect, or AoE, refers to a spell or other effect that covers a location, rather than hitting just one individual monster. AoE attacks are generally less damaging per target, but they make up for it by potentially hitting many targets at once, often with damage over time, for greater total damage.
Generally speaking, these types of attacks are best used on multiple targets, especially when they're grouped by some expert crowd control. AoE attacks are usually less effective on single targets, especially ones with a lot of hit points like a boss.
Some skills seem to combine direct damage with AoE; Fireballs with splash damage, for instance. In those cases the extra damage dealt to nearby monsters is somewhat of a bonus, rather than the reason for using the spell in the first place.