Assuming you are using the latest version of Mailscan for Exchange [5.6]
There are two ways to reject mails from a particular domain. The first is SMTP level and the other at Mail scan Level.
Please follow the steps mentioned below :
Open SMTP Administrator -> Goto User and Mail Restrictions -> Goto Restrictions From User Tab.
Under "Do not accept mails from the following IP/Hosts/Users :
Add the domain that has to be rejected.
Eg.
*@domain.com
This will be rejected at the SMTP level.
Alternative way is :
Open Mailscan Administrator-> Goto Scan Control
Under Do not Scan Mails from Users, check the Delete Mails from following users.
Add the particular domain.
Eg.
*@domain.com
This will reject the domain at mailscan.