Why they Exist
While there isn't a given reason in the lore (yet) for these power-ups floating in mid air like an arcade game, the reason for their existence in the gameplay is to increase speed of gameplay without relying so heavily on potions. The development team wanted to keep Diablo III fast paced and filled with action, not waiting idle for health or mana to regenerate. Jay Wilson also said that the old potion-based combat, where potions basically were extra health and mana to your pools, they wanted it to be more interactive.
Game Mechanics
The drop rate is quite low, but with the amount of monsters coming at players, there should rarely be a lack of orbs, and when they are far between, there is always potions. Additional tweaks have also been promised.
SourceThat's how it works!
We're also playing around with some alternate functionality for when/how they're picked up while you're at full health, as well as some visual tweaks. Regardless we're going to make sure it's as fun and balanced as possible.
Controversy
There has been some small level of controversy about these Health Orbs as some players do not like the idea of "power-ups" in a RPG game. It is nowhere near the levels of the art controversy, but some people are genuinely upset.
In either way, it is highly unlikely that Blizzard will change Health Orbs significantly at this point.