detoxai.metrics

Submodules

detoxai.metrics.bias_metrics submodule

class detoxai.metrics.bias_metrics.BiasMetrics(value)

Bases: Enum

TPR_GAP = 'TPR_GAP'

FPR_GAP = 'FPR_GAP'

TNR_GAP = 'TNR_GAP'

FNR_GAP = 'FNR_GAP'

EO_GAP = 'EO_GAP'

DP_GAP = 'DP_GAP'

detoxai.metrics.bias_metrics.stabilize(x, epsilon=0.0001)

Parameters:

• x

• epsilon – (Default value = 1e-4)

Returns:

detoxai.metrics.bias_metrics.calculate_bias_metric_torch(metric: BiasMetrics | str, y_pred: Tensor, y_true: Tensor, protected_attribute: Tensor) → Tensor

Parameters:

• metric – BiasMetrics | str:

• y_pred – torch.Tensor:

• y_true – torch.Tensor:

• protected_attribute – torch.Tensor:

Returns:

detoxai.metrics.distance_metrics submodule

detoxai.metrics.distance_metrics.cosine_similarities_batch(a, cav, eps=1e-07)

Parameters:

• a

• cav

• eps – (Default value = 1e-7)

Returns:

detoxai.metrics.distance_metrics.euclidean_dist(x, y, dim=-1)

Parameters:

• x

• y

• dim – (Default value = -1)

Returns:

detoxai.metrics.distance_metrics.cosine_dist(x, y, dim=-1)

Parameters:

• x

• y

• dim – (Default value = -1)

Returns:

detoxai.metrics.distance_metrics.largest_vals(x, y, dim=-1)

Parameters:

• x

• y

• dim – (Default value = -1)

Returns:

detoxai.metrics.fairness_metrics submodule

detoxai.metrics.fairness_metrics.stabilize(x, eps=1e-08)

Parameters:

• x

• eps – (Default value = 1e-8)

Returns:

detoxai.metrics.fairness_metrics.results_to_tidy_df(results, metrics_config)

Parameters:

• results

• metrics_config

Returns:

class detoxai.metrics.fairness_metrics.PerformanceMetrics

Bases: object

classmethod infer_metrics(metrics_config: Dict[str, Dict[str, List[str]]], class_labels: List[str])

Parameters:

• metrics_config – Dict[str:

• Dict[str

• List[str]]]

• class_labels – List[str]:

Returns:

class detoxai.metrics.fairness_metrics.AllMetrics(metrics_config: Dict[str, Dict[str, Dict[str, List[str]]]], class_labels: List[str], num_groups: int | None = None)

Bases: object

get_performance_metrics() → PerformanceMetrics

get_fairness_metrics() → MetricCollection

class detoxai.metrics.fairness_metrics.BinaryGroupStatRatesUnwrapped(**kwargs)

Bases: BinaryGroupStatRates

update(preds: Tensor, target: Tensor, sensitive_features: Tensor)

Parameters:

• preds – torch.Tensor:

• target – torch.Tensor:

• sensitive_features – torch.Tensor:

Returns:

compute()

class detoxai.metrics.fairness_metrics.FairnessMetrics(metrics_spec: Dict[str, Dict[str, List[str | None]]], *args, **kwargs)

Bases: BinaryGroupStatRatesUnwrapped

Computes derived metrics (like TPR, FPR, etc.) for each group and allows for ratio and difference comparisons between groups to assess fairness. Supports various fairness metrics such as Equalized Odds.

Parameters:

• metrics_spec (metrics_spec_dict_type) – Dictionary specifying metrics to calculate.

• args – Additional arguments for the base class initialization.

• **kwargs –

  Additional arguments for the base class initialization.

Supported metrics: - True Positive Rate (TPR), False Positive Rate (FPR), Error Rate (ER), etc. - Equalized Odds can be calculated with either a ‘ratio’ or ‘difference’ reduction. Example usage:

Returns:

>>> preds = torch.tensor([1, 1, 1, 0, 1, 0, 1, 0])
    >>> target = torch.tensor([1, 1, 1, 1, 0, 0, 0, 0])
    >>> sensitive_features = torch.tensor([1, 1, 1, 1, 0, 0, 0, 0])

>>> metrics_spec = {
...     "TPR": {"reduce": ["ratio", "difference", "per_group"]},
...     "FPR": {"reduce": ["ratio", "difference"]},
...     "EO": {"reduce": ["difference"]},
...     "ACC": {"reduce":  ["per_group"]},
... }
>>> derived_metrics = DerivedMetrics(metrics_spec, num_groups=2)
>>> results = derived_metrics(preds, target, sensitive_features) #forward calls update and compute internally
>>> results
{
    'TPR_0_group': tensor(1.),
    'TPR_1_group': tensor(0.6667),
    'TPR_ratio': tensor(nan),
    'TPR_difference': tensor(nan),
    'FPR_ratio': tensor(1.),
    'FPR_difference': tensor(0.),
    'EqualizedOdds_difference': tensor(nan),
    'ACC_0_group': tensor(0.5000),
    'ACC_1_group': tensor(0.5000),
}

compute()

calculate_equalized_odds(derived_metrics)

Parameters:

derived_metrics

Returns:

calculate_demographic_parity(derived_metrics)

Parameters:

derived_metrics

Returns:

calculate_equality_of_opportunity(derived_metrics)

Parameters:

derived_metrics

Returns:

calculate_treatment_equality(stats)

Parameters:

stats

Returns:

detoxai.metrics.metrics submodule

detoxai.metrics.metrics.stabilize_torch(x: Tensor, eps: float = 0.0001) → Tensor

Stabilize a tensor by adding a small epsilon

Parameters:

• x – torch.Tensor:

• eps – float: (Default value = 1e-4)

Returns:

detoxai.metrics.metrics.balanced_accuracy_torch(y_true: Tensor, y_pred: Tensor) → Tensor

Calculate the balanced accuracy metric

Parameters:

• y_true – torch.Tensor:

• y_pred – torch.Tensor:

Returns:

detoxai.metrics.metrics.comprehensive_metrics_torch(y_true: Tensor, y_pred: Tensor, prot_attr: Tensor | None = None, return_torch: bool = True) → dict[str, Tensor | float]

Calculate a comprehensive set of metrics

Parameters:

• y_true – torch.Tensor:

• y_pred – torch.Tensor:

• prot_attr – torch.Tensor | None: (Default value = None)

• return_torch – bool: (Default value = True)

Returns:

Module contents