Stop ducking AFTER sweeper ends, not BEFORE.
When the audio is ducked for a sweeper, it begins ramping it back up to full volume before the sweeper ends, meaning that when the sweeper is at the end, the audio isn't ducked. It should begin to fade back up AFTER the sweeper ends, not before.