I believe you need to be at least level 3 and.or have at least 10-50 fame (exact number
i'm not sure of), before you are granted access to the isles.

Even with these requirements met you still have to speak with the bravil guard who is
there to trigger a small event in which another adventurer has gone through the door and
comes out insane and subsenquently killed. You must also listen to the Madgod's monologue
before being able to enter.

i like barely started, walked there, and talked to the guy and then the gate was talkin
and sed come in

If its still not letting you in then it must be a matter of level and/or fame. I'd suggest
to leave it alone for now and do a few other quests until you gain a couple of levels and
gain a little fame and then try again to enter.

Also bear in mind that most rewards you receive from completing quests in the Isles are
leveled meaning that the higher level you are when you receive the reward the better the
item will be. The enchantments will be better and in the case of weapons they will deal
more damage.

Of course that is simply a suggestion , since oblivion has a very high replayability you
can wait till future playthroughs to get the best versions of rewards.

i can go there anytime no matter level or fame, looks like you got a bug