Most iconic is probably Super Mario Bros. even if it is just World 1-1 that people remember.

My favourite and also highly iconic is The Legend Of Zelda: Ocarina Of Time.

Final Fantasy VIII stood out for me more then VII, but that's probably due to the age I was when I played VIII. Also the opening cut scene for VIII is a thing of cinematic beauty that tied in heavily to how I remember the game and I think was also utilised in the TV adverts for the game.

As for games that make the soundtrack part of the gameplay, those dance mat games were massive, but Guitar Hero took things up a notch, however I'd struggle to recall any of the tracklisting for the first three Guitar Hero games bar Through The Fire And The Flames by Dragonforce.