Why not use this as an opportunity?
Get him alone and say in the nicest way possible that you have no desire to go because of his constant, passive-aggressive bullshit. I realise the wording needs to be considerably more subtle but that's the nub.
LP: I am afraid I won't be attending the bonfire event.
Professor Asshat: Why not?
LP: Because on occasion A you did Z, on B you did Y, on C you did X, on D you did W......... and for this reason I don't feel particularly like I am a valued member of the lab/respected in any way by you/want to spend my free time in your company etc etc. You don't treat me with the respect I know I deserve and for this reason I have no desire to spend my free time in your company.
If you are concerned about that biting you on the ass then take someone in with you that you trust or alternatively put it in an e-mail and copy in anyone else you feel will have your back should he up the Asshat ante any time in the future.
Regardless of the method by all accounts he is a bully. Stand up to him. And if he escalates then keep fronting him and when you do make sure it is front of other people and you have all the facts to back up your case.
Good luck. I've had to deal with workplace bullies in the past and it ain't fun. I hate to think of anyone else, particularly someone as brown and nummy as you, having to put up with that shit.