Get rid of them now. They WILL ruin your lawn, especially if given a chance to build a burrow under a deck, or some other inaccessible-to-humans place. Once they get hunkered down, it's difficult to get them out.
Just tell the wife if she's OK with having them get in the house, then you're OK leaving them on the lawn.