Stronghold 2 Requests

OK, I've beaten Edwin and now have to clean up his estate and get 100 residents. I've built a travelling fair, increased food rations to double, and have food and resources coming in from everywhere. I can't get my popularity up. My tax gives me +4 on popularity, food +8, but gong, rats and crime are even greater negatives. Within minutes of starting, my popularity takes a nose dive. How do I get it back up? My honour is over 200 to start, so I don't need to work on that part. Any suggestions?