The problem with that reasoning is that the particle is always going to be detected at one slot or the other, and so even when no detectors are involved we would have to presume that the particle in some sense passes through one or the other, and that the other slot "shouldn't matter." It also doesn't take into account the similar results attained with atoms and molecules, where the collision or interaction with the detector has a much more subtle effect on the trajectory or energy state of the object and doesn't involve any re-emission. I would agree with you if we were closing one slit entirely, and some experiments have been crude enough that they have this effect, but that is not generally the case.