After inflicting enough damage to Trollsegur, you'll get a cutscene where the music will change and the ship overpowers its crew and starts fighting on its own. After this point if you wait long enough the ship will become so unstable it will destroy itself without you needing to finish it off, essentially allowing you to just stay in a corner of the map away from harm until the victory cutscene triggers.