He is just trying to scare you. He cannot annul the marriage or divorce you without your being served with a Summons and Complaint. If he tries to do it without notifying you, you can have it reversed.
He can sue you for an annulment, but it is up to a judge to grant him his request, and the judge will want to hear your side of the story before he makes a decision.
If you are living in the NYC metro area, please call me to discuss mediating your annulment.
Leonard Weiner, Esq./Divorce Solutions