Moving the discussion with @amueller from pydata/patsy#77 (comment). Proposing to: add an optional 'labels' argument to log_loss add an argument to make_scorer to enable passing labels argument to ...