SAN FRANCISCO — DVD-by-mail service Netflix has canceled a sequel to a $1 million movie-recommendation contest, avoiding a potential courtroom drama…