I would return the deposit.
In business, the customer is NOT always right, but $50 is a small price to pay for potentially bad word-of-mouth, etc. Especially if you're just starting out.
It might be different if you spent hours on a job and they went their money back, but this is a deposit for work you haven't even done yet.
I would just make it clear that you don't normally do this, but you'll make an exception, and be completely pleasant and smile the entire time.
Hopefully they'll be thankful and will remember you in the future, or potentially send other work your way via a referral, etc.