You need to press R, A, B for a "certain kill" shot or your "Special" However, all you need to do is hold A, press and hold B, then tap R. Timing is critical. This works for any type of special move.

With Goku level 1, when your power is 100%, press L + R. Also, when your power is 100%, be at the very top of the battle screen and press A + B + R. Goku will perform Spirit Bomb or Genki Dama.

With Goku levels 2 and 3, when your power is 100%, you must be at the very top of the battle screen and press A + B + R. Goku will perform a Kamehameha Wave with an Instant Transmission. Try doing this; when your power is 100%, be at the very top of the battle screen and press A + B + R simultaneously. This can be done with other characters besides Goku.