Brown was named coach of the Cavs in the summer of 2005. They went to the playoffs in his first season and in 2007, the Cavs beat the Detroit Pistons to make the Eastern Conference Finals. The Cavs won more than 60 games in Brown's final two years with the team, eventually being fired in 2010 for not being able to get the team over the hump.

Brown has had a relatively unusual head coaching career. Rather than starting out with a young team and growing slowly, he was thrown into the fire as the head coach of the Cleveland Cavaliers right as LeBron James began to grow into a superstar.