To overextend means to push too far forward, out of the range of your healers. This means that you are in a vulnerable position and your healers will have to move up to heal you, taking additional time and placing them in danger. Using skills that increase the damage you take (e.g. Frenzy) while overextended is particularly dangerous. Overextending differs from splitting in that it is unintentional and the character is typically unable to support themselves.
Overextension happens any time you move out of the range of your healers but in PvP it commonly occurs when a frontline player (e.g. a warrior) runs forward to attack an opposition backline player (e.g. a monk) and can be exacerbated by continuing to attack even as the target is kiting backwards. In PvE it can occur when a monster breaks aggro and a player runs after it. This is dangerous when mobs who had previously been aggroed by a tank then aggro a soft target. It can also be seen in PvE when a melee player runs through the AI lines to attack a backline monster (e.g. a monk boss).