How loud is loud? I know loud when I hear it, but if you want a number, I'd say at home anything over 90 dB is getting up there, and might annoy neighbors in adjacent apartments, especially after 10 p.m. If you live in a house, 90+ dB would definitely disturb other family members not watching the movie or listening to music. Of course, the volume at concerts and movie theaters is much, much louder than most people would ever tolerate at home. Loud music, games, and home theater takes on an almost physical quality; you don't just hear it, you feel it with your entire body.
A lot of folks find loud noise at home painful, but enjoy it at concerts or in their car. Those who prefer whisper-quiet volume are lucky; softer-volume listening doesn't make demands on the gear and sounds fine with tiny speakers.
Some speakers sound better than others when played at high volumes. Big speakers have the edge over little ones, and few really small speakers (under 5 inches tall), or ones with 3-inch or smaller woofers sound great when played really loud. When looking at a given speaker's specifications, the "sensitivity" spec will be more relevant than "power handling" for predicting an aptitude for high volume. The higher the number, the louder the speaker will play with a given number of watts. Most speakers are have around 86 dB sensitivity, but 88 or 89 dB won't be much better. Ones in the low 90s are more likely candidates, but specs never tell the whole story. When in doubt, check with the manufacturer. The Klipsch RF-62 II towers ($500 each) have ultra-high sensitivity (97 dB) and can play very loud with a 25-watt amp, and still handle 500 "peak" watts!
The sensitivity spec is only important to consider if you either like to play music or movies really loud, or your room is large (and if it is, don't buy small speakers), or your room has a lot of sound absorbing things like heavy drapes, overstuffed furniture, thick carpeting, etc. Sound quality is a different matter, and unrelated to sensitivity. For home theater applications, large subs, with 10-inch or larger woofers, are recommended for high-volume bass.
There's loud, and then there's really loud, and sustained high volume listening at home, concerts, bars, or movie theaters over the long run can affect your hearing and is definitely not recommended. If you start to regularly experience "ringing in the ears," clicks, buzzing, etc. (tinnitus) during or after exposure to loud sound, you may be starting to lose your hearing. Turn the volume down, and have your hearing checked by an audiologist.