Elections, being a human endeavor, cannot be perfect. There will always be legitimate voters who are prevented from voting and there will always be people who vote improperly. Either of these can undermine the validity of the election if they are widespread, and both of them can undermine confidence that the election is valid, even if they are fairly rare, if people are convinced that they're happening.
What I want to get at is the question of which of the two is more important. Setting aside the question of exactly who should be allowed to vote, where in principle should the balance be between being sure no one who is supposed to be allowed to vote is excluded and being sure no one who isn't does? Is it better to turn away a hundred legitimate voters than to allow one fraudulent vote in? Is it better to allow a hundred fraudulent votes in than to turn away one legitimate person? Let's just imagine you're an election commissioner and you have a box of 100 ballots and there is a controversy about the box. You know that some fraudulent ballots got put into the box along with some valid ones, but there is absolutely no way to tell which ones are which. You must either accept the whole box or reject the whole box. But you have learned the exact number of bogus ballots that got in the box. If you accept the box, N illegal votes are counted. If you reject it, 100-N legitimate voters lose their votes. Where do you draw the line? Other than who the votes are probably for, does the nature of the nature of the fraudulent votes affect your decision? Does it matter who the voters who would be disenfranchised are? Discuss.
For me, it does matter if the illegal votes are inside or outside the principle of one person, one vote. I do want the rules to be followed, but I'm less worried about a tourist from France casting a ballot while he's here on vacation than I am about a fully qualified citizen getting to put in more than one vote (either by forging a ballot or by forcing a real voter's choice).
As I actually try to answer my own question, I find it's tougher than I thought. It makes a big difference how evenly divided the legitimate votes are likely to be. If there were 49 real votes for each candidate, they cancel and the two fraudulent votes are all that's left, so it would be better to throw out the box even though it has only 2 bad ballots. If there were 65 real votes for one candidate and 5 for the other, then the 60 vote difference in the real votes probably justifies counting the 30 bad ones. But I'm cheating on my own question to argue that way.
Without any additional information, if the illegal votes are all from people who actually live in the community but aren't allowed to vote (and each only voted once, there), I'd probably accept the box if I knew it had 50 bogus votes. If I knew that 20 ballots were from people who had gotten on a bus after voting in their own precinct, voted in this one, and went home, I'd throw the box out. If I knew that 10 ballots were put in by one single cheater, I'd throw the box out. (I reserve the right to change my answer if I get information on how the real votes in the box are likely distributed, i.e., how many of them cancel each other.)