smelly exhaust fumes.

[Burbnasty]

So, I have a 86 burb with a tailgate. I have replaced the tailgate one with one off of my parts truck because it was rust free and the track in the old one was all messed up.

If I roll down the drivers window only, I get exhaust fumes in the truck. If I roll down the rear window only I cant smell any exhaust fumes. This is all with the tailgate glass rolled up.
Sometimes if sitting at a traffic light I smell fumes.
Now I'm concerned about having family in burb especially baby. If I cant drive my weekend family cruiser, with the family Im going to be a sad guy.

My question Did I do something wrong when swapping gates, I seem to remember always having this problem even with the original gate?
The seals appear to be in good shape, my exhaust is in great shape, no holes or anything, the floor boards are in good shape.

How can I stop fumes from getting in?

I only smell this from time to time, and I could just be paranoid, but I want the family to be safe and not take any chances.

Help so i don't have to go two tracking in my wife's lame VW.