Dealing with Directory Harvest Attacks

One of my domains is the .com of a common surname.

I get these constant spam attacks where the spammer sends to every possible combination of names and words @mydomain.com I think this kind of thing is called a directory harvest attack, correct?

They come from a wide variety of ips and different addresses.

I only use five addresses on this domain. All these other emails are rejected or bounced away be postgrey or the server responding that user doesn't exist.

However, these things are coming in sometimes several per second. Is there a more efficient way to handle them? Like perhaps something in front of postgrey that says 'if not these specific addresses, ignore'?

Chris