And go into the big Temple structure. Not the castle, but the temple.

You go into the Temple of Time play the song of time at the altar,and the door of time
will open...

NOTE:KOKORI STONE:you get it after you kill the boss inside of the deku tree(who dies
after you save him which is kind of sad:<)
GORON STONE:you get it after you defeat the boss in the dodongos cavern(you need bombs to
open the cavern and to kill the boss)
ZORAS STONE:you get from princess ruto after you kill the boss inside of lord jabu
jabu(but then the stone actually means u two are engaged in her mind!)
YOU NEED THE SONG OF TIME AND THEN PLAY IT IN FRONT OF THE DOOR OF TIME AND PLACE THE GEMS
IN THE THING IN FRONT OF IT!!!!!!!!!