You do not need to beat the game 4 times to reach the secret garden. All you have to do is familiarize yourself with the diagonal jump. When you grab a climbable wall (usually indicated by the moss growing on it), you have to hold jump and the left analog stick diagonally up until your character swings his arm, depending on what direction you tilted the analog stick. Once he does this, let go of jump and you will scale the wall diagonally without using a large amount of stamina.

Before you decide to reach the secret garden, it is advisable to complete about 7 colossi for the sole reason of acquiring enough stamina to take on this task. You could also search for white-tailed lizards as an alternative to gain stamina.

To reach the secret garden, you have to locate the wall of ivy on the northeast side of the temple shrine. If you use your horse and jump off of him towards the wall of ivy, you will save even more stamina. Now, follow it all the way up until you reach a ledge on the right. Go around it while still using the diagonal jump to save stamina. On the opposite side, leap across to the visible moss and keep going around the second ledge above. Once you get to other side, you have to go around yet another ledge, which is a little bit below your position. You will reach a circular pillar that you have to jump onto. This is easily done by situating yourself directly opposite of it and then jumping across. Hold on and keep going up. The moss will lead you a little bit towards the left to a small walkway. Follow the stairs to the secret garden. Enjoy, but do not eat the fruit, as it will decrease your health permanently.