They could sue you for breach of contract - it's unlikely they will but there is that risk.
I think you should just hand in your notice (a week?) and keep going in until your finish date- you may need a reference from them one day.
As an aside, I don't think they have been unreasonable regarding giving you 3 weeks to provide your references, and they are under no obligation to give you a day off for a wedding so soon after starting- maybe you should have mentioned it as a condition for accepting the job



