Born in Memphis, Tennessee, Browder grew up in Charlotte, North Carolina. His parents were race car owners and operators. He attended Furman University in Greenville, South Carolina and graduated with a degree in psychology. He was a star player on the Furman football team. Browder met his wife, actress Francesca Buller, while studying at the Central School of Speech and Drama in London.