shap_enhanced.explainers.ECSHAP

EC-SHAP: Empirical Conditional SHAP for Discrete Data

Theoretical Explanation

Empirical Conditional SHAP (EC-SHAP) is a feature attribution method tailored for discrete data types, including binary, categorical, and one-hot encoded features. Unlike classical SHAP methods that often rely on unconditional sampling or simplistic imputations, EC-SHAP imputes masked features based on the empirical conditional distribution derived from a background dataset.

For each coalition (a subset of masked features), EC-SHAP seeks background samples that match the unmasked features, ensuring that imputed instances remain within the data manifold and reflect realistic, observed patterns.

Key Concepts

  • Empirical Conditional Imputation:

    Masked features are filled by matching the unmasked portion of an input to background data. If no exact match exists, the algorithm can either skip the coalition or use the closest match (by Hamming distance).

  • Valid Discrete Patterns:

    All imputations correspond to real, observed combinations in the background dataset—preserving the statistical validity and interpretability of the perturbed inputs.

  • Fallback for Continuous Features:

    If features appear continuous (e.g., many unique values), EC-SHAP automatically falls back to mean imputation.

  • Additivity Normalization:

    Attributions are scaled such that their sum equals the difference in model outputs between the original and fully-masked inputs.

Algorithm

  1. Initialization:

    • Accepts a model, background dataset, device context, and configuration for skipping or relaxing matches (e.g., using closest match).

  2. Conditional Imputation:
    • For each coalition (subset of features to mask):

      • Identify background samples where the unmasked features match.

      • If a match exists, use it to fill in masked features.

      • If no match:

        • Optionally use the nearest match (by Hamming distance), or

        • Fallback to mean imputation (for continuous features), or

        • Skip the coalition.

  3. SHAP Value Estimation:
    • For each feature:

      • Sample random coalitions of other features.

      • Impute both:

        • The coalition alone, and

        • The coalition plus the target feature.

      • Compute the difference in model outputs.

      • Average the differences to estimate marginal contribution.

  4. Normalization: - Ensure the sum of feature attributions equals the difference in model output between the original and fully-masked input.

References

  • Lundberg & Lee (2017), “A Unified Approach to Interpreting Model Predictions” [SHAP foundation—coalitional feature attribution framework]

  • Aas, Jullum & Løland (2021), “Explaining individual predictions when features are dependent: More accurate approximations to Shapley values” [Introduces empirical conditional sampling for dependent/discrete features] :contentReference[oaicite:1]{index=1}

  • Redelmeier, Villani et al. (2020), “Explaining predictive models with mixed features using Shapley values and conditional inference trees” [Uses conditional inference trees to impute masked discrete/categorical features realistically] :contentReference[oaicite:2]{index=2}

  • “Estimating conditional Shapley values” —Springer comparison study (2023) [Systematizes Monte Carlo and regression methods for conditional Shapley estimation when feature dependence is unknown] :contentReference[oaicite:3]{index=3}

  • Molnar, _Interpretable Machine Learning_ (2022), SHAP chapter [Discusses trade-offs and techniques for conditional vs marginal sampling in SHAP; practical guidance for discrete and mixed data] :contentReference[oaicite:4]{index=4}

Classes

EmpiricalConditionalSHAPExplainer(model, ...)

Empirical Conditional SHAP (EC-SHAP) Explainer for Discrete Data
class shap_enhanced.explainers.ECSHAP.EmpiricalConditionalSHAPExplainer(model, background, skip_unmatched=True, use_closest=False, device=None)[source]

Bases: BaseExplainer

Empirical Conditional SHAP (EC-SHAP) Explainer for Discrete Data

This explainer estimates Shapley values for discrete (e.g., categorical, binary, or one-hot) feature inputs by imputing masked features from a background dataset using conditional matching. It ensures perturbed samples remain within the data manifold, preserving interpretability.

Parameters:

  • model (Any) – Model to explain, must support PyTorch tensors as input.

  • background (np.ndarray or torch.Tensor) – Background dataset used for empirical conditional imputation.

  • skip_unmatched (bool) – If True, skip coalitions where no matching background sample exists.

  • use_closest (bool) – If True, use the closest (Hamming distance) background sample when no exact match is found.

  • device (Optional[str]) – Device on which to run the model (‘cpu’ or ‘cuda’).

property expected_value

Optional property returning the expected model output on the background dataset.

Returns:

Expected value if defined by the subclass, else None.

Return type:

float or None

explain(X, **kwargs)

Alias to shap_values for flexibility and API compatibility.

Parameters:

  • X (Union[np.ndarray, torch.Tensor, list]) – Input samples to explain.

  • kwargs – Additional arguments.

Returns:

SHAP values.

Return type:

Union[np.ndarray, list]

shap_values(X, nsamples=100, check_additivity=True, random_seed=42, **kwargs)[source]

Estimate SHAP values using empirical conditional imputation.

For each feature-time index (t, f), this method: - Samples coalitions of other features. - Finds background samples matching the unmasked portion of the input. - Imputes masked values with corresponding values from the matched sample. - Computes model output with and without the target feature masked. - Averages the differences over multiple coalitions.

Normalization ensures:

\[\sum_{t=1}^T \sum_{f=1}^F \phi_{t,f} \approx f(x) - f(x_{\text{masked}})\]

Note

If no exact match is found and use_closest is False, the coalition may be skipped. For continuous-looking data, the method will fallback to mean imputation.

Parameters:

  • X (np.ndarray or torch.Tensor) – Input data of shape (T, F) or (B, T, F)

  • nsamples (int) – Number of coalitions to sample per feature.

  • check_additivity (bool) – Whether to rescale SHAP values to match model output difference.

  • random_seed (int) – Seed for reproducibility.

Returns:

SHAP values of shape (T, F) or (B, T, F)

Return type:

np.ndarray