uqlm.utils.tuner.Tuner#

class uqlm.utils.tuner.Tuner#

Bases: object

__init__()#

Class for tuning weights and threshold for UQEnsemble

Methods

__init__()

Class for tuning weights and threshold for UQEnsemble

tune_params(score_lists, correct_indicators)

Tunes weights and threshold parameters on a set of user-provided graded responses.

tune_threshold(y_scores, correct_indicators)

Conducts 1-dimensional grid search for threshold.
tune_params(score_lists, correct_indicators, weights_objective='roc_auc', thresh_objective='fbeta_score', thresh_bounds=(0, 1), n_trials=100, step_size=0.01, fscore_beta=1)#

Tunes weights and threshold parameters on a set of user-provided graded responses.

Return type:

Dict[str, Any]

Parameters:

  • score_lists (list of lists of floats) – A list of lists of floats. Each interior list is a list of component-specific scores.

  • correct_indicators (list of bool) – A list of boolean indicators of whether self.original_responses are correct.

  • weights_objective ({'fbeta_score', 'accuracy_score', 'balanced_accuracy_score', 'roc_auc', 'log_loss', 'average_precision', 'brier_score'}, default='roc_auc') – Objective function for optimization of weights. Must match thresh_objective if one of ‘fbeta_score’, ‘accuracy_score’, ‘balanced_accuracy_score’. If same as thresh_objective, joint optimization will be done.

  • thresh_objective ({'fbeta_score', 'accuracy_score', 'balanced_accuracy_score'}, default='fbeta_score') – Objective function for threshold optimization via grid search.

  • thresh_bounds (tuple of floats, default=(0,1)) – Bounds to search for threshold.

  • n_trials (int, default=100) – Indicates how many candidates to search over with optuna optimizer.

  • step_size (float, default=0.01) – Indicates step size in grid search, if used.

  • fscore_beta (float, default=1) – Value of beta in fbeta_score.

Returns:

Dictionary containing optimized weights and threshold.

Return type:

Dict

tune_threshold(y_scores, correct_indicators, thresh_objective='fbeta_score', fscore_beta=1, bounds=(0, 1), step_size=0.01)#

Conducts 1-dimensional grid search for threshold.

Return type:

float

Parameters:

  • y_scores (list of floats) – List of confidence scores.

  • correct_indicators (list of bool) – A list of boolean indicators of whether self.original_responses are correct.

  • thresh_objective ({'fbeta_score', 'accuracy_score', 'balanced_accuracy_score'}, default='fbeta_score') – Objective function for threshold optimization via grid search.

  • fscore_beta (float, default=1) – Value of beta in fbeta_score.

  • bounds (tuple of floats, default=(0,1)) – Bounds to search for threshold.

  • step_size (float, default=0.01) – Indicates step size in grid search, if used.

Returns:

Optimized threshold.

Return type:

float

References