My mother used to employ a saying that she used all the time.

She would mention "Don't throw the baby away with the tub water". I'll admit that for a very long time the single reaction of mine was "Huh?