Since you seem to know what the quest is and how to do it , i will stick with the possible
bugs that could explain your current problem.


BUGS :

Do not cast the soultrap spell on two different beings (or twice on the same being) within
the 120 second time-frame of the spell. This will cause the spell to stop functioning
correctly on all future casts regardless of whether you kill the NPCs that you double-cast
on or not.

Occasionally you will not be able to soul trap one of the races.
- Saving your game to a new save (be sure to not overwrite the last save made before the
start of the quest), quitting out of the game entirely, restarting, and reloading can fix
this problem.
- Another method is possible. Cast Hermaeus Mora's Soul Trap spell on an NPC two times
within 120 seconds, and within the same time period, cast the spell again on the NPC you
wish to capture. You will not get a message stating that you "You cannot cast this spell
on multiple targets", but the 2nd NPC is now able to be trapped.
- If saving and reloading does not work then you may have to load a game that was saved
prior to the start of Hermaeus Mora quest and redo the quest.
- A reliable fix to the bug rather than keep reloading a prior save is to cast the spell
on two separate NPCs until you are interrupted by a message saying "You cannot cast the
spell on two separate souls at once." Once you've gotten that message, do not cast the
spell again and find a safe place to wait(select button). After that, the spell will work
as it is supposed to without any bug or glitch.


Hopefully this will help you.

The exact smae thing happend to me, deoraven is right