thresholding

class AdaptiveThreshold(utility)

Adaptive agent that balances exploration with exploitation with respect to setting and adjusting thresholds.

When exploring, the threshold is 0, effectively letting anything through. This produces unbiased data that can then be used to set a more optimal threshold in subsequent rounds. The agent seeks to balance the opportunity cost of running an experiment with the utility gained over subsequent rounds using the information gained from this experiment.

__call__(ground_truth, score)

Args are ignored if previous threshold was not 0. Otherwise, the score is added as a potential threhold and ground_truth noted to help identify the optimal threshold.

Parameters:

• ground_truth (bool) –

• score (float) –

__init__(utility)

Parameters:

utility (function) – Function that takes in true/false positive/negative rates. Specifically (tp, fp, tn fn) -> float representing utility.

get_best_threshold()

expected_utility(utility, data, N=4096, credibility=0.5)

Get the utility distribution over possible thresholds.

Parameters:

• utility (function) – utility function that ingests true/false positive/negative rates.

• data (list-like) – iterable of list-likes of the form (ground truth, score). Feedback is null when an alert is not triggered.

• credibility (float) – Credibility level for a credible interval. This interval will be centered about the mean and have a credibility chance of containing the true utility.

Returns:

• candidate thresholds

• mean expected utility

• upper and lower quantile of estimate of expected utility associated with each threshold

Return type:

tuple of three elements

exploration_proportion(thresholds, N)

optimal_threshold(utility, data, N=4096)

plot_err(scores, utility_mean, utility_err, color=None, label=None, alpha=0.5)

sample_utilities(utility, data, N=4096)

Get distribution of utilities.

Parameters:

• utility (float) – utility function that ingests true/false positive/negative rates.

• data (list-like) – iterable of of iterables of the form (ground truth, score). Feedback is null when an alert is not triggered.

returns: thresholds, utilities

thompson_sample(utility, data, N=1024, quantile=False)

update_rates(rates, ground_truth)