He color of the blocks can only go down the hole with the same color. The colors cycle each time you beat the golem,so it's not that hard to find the color you're looking for.Hope it helps!