ARTISTdirect Network (The Ultimate Band List): A good source of links to sites about most any band you can name.
The Internet Underground Music Archive (IUMA): One of the places where you can either try to get your start or listen to others who are doing just that. Step into the unknown; you might just be rewarded.
MP3.Com: All about your favourite audio compression format, including whacks of free tracks by new and unknown bands.
Noisembryo: Beginner's Guide to Noise World: A Japanese page about noise from all around the world. Some parts are in Japanese only, but others, including the links page, are in English.
Rough Guides Music: Another resource for trying to pinpoint info about your favourites.
Theremin World: Everything you wanted to know about one of the most unique electronic musical instruments in the world, including plans. if you want to try building your own.
Top Sampling Groups: Find out who steals from what to make what you pay them money for.