BaseModelEvaluator

Bases: EvaluatorMethods, ABC

Abstract base class for evaluating machine learning model performance.

This class provides methods for calculating model performance metrics, plotting confusion matrices, and evaluating feature importance, with options for handling one-hot encoded features and aggregating SHAP values.

Inherits

• EvaluatorMethods: Provides prediction and encoding methods.
• ABC: Specifies abstract methods for subclasses to implement.

Parameters:

Name	Type	Description	Default
X	DataFrame	The test dataset features.	required
y	Series	The test dataset labels.	required
model	BaseEstimator	A trained sklearn model instance for single-model evaluation.	required
encoding	Optional[str]	Encoding type for categorical features, e.g., 'one_hot' or 'target', used for labeling and grouping in plots.	required
aggregate	bool	If True, aggregates the importance values of multi-category encoded features for interpretability.	required

Attributes:

Name	Type	Description
X	DataFrame	Holds the test dataset features for evaluation.
y	Series	Holds the test dataset labels for evaluation.
model	Optional[BaseEstimator]	The primary model instance used for evaluation, if single-model evaluation is performed.
encoding	Optional[str]	Indicates the encoding type used, which impacts plot titles and feature grouping in evaluations.
aggregate	bool	Indicates whether to aggregate importance values of multi-category encoded features, enhancing interpretability in feature importance plots.

Methods:

Name	Description
calibration_plot	Plots calibration plot for model probabilities.
brier_score_groups	Calculates Brier score within specified groups.
bss_comparison	Compares Brier Skill Score of model with baseline.
plot_confusion_matrix	Generates a styled confusion matrix heatmap for the test data and model predictions.

Inherited Methods

• brier_scores: Calculates Brier score for each instance in the evaluator's dataset based on the model's predicted probabilities. Returns series of Brier scores indexed by instance.
• model_predictions: Generates model predictions for evaluator's feature set, applying threshold-based binarization if specified, and returns predictions as a series indexed by instance.

Abstract Methods

• evaluate_feature_importance: Abstract method for evaluating feature importance across models.
• analyze_brier_within_clusters: Abstract method for analyzing Brier score distribution within clusters.

Source code in periomod/evaluation/_baseeval.py

class BaseModelEvaluator(EvaluatorMethods, ABC):
    """Abstract base class for evaluating machine learning model performance.

    This class provides methods for calculating model performance metrics,
    plotting confusion matrices, and evaluating feature importance, with options
    for handling one-hot encoded features and aggregating SHAP values.

    Inherits:
        - `EvaluatorMethods`: Provides prediction and encoding methods.
        - `ABC`: Specifies abstract methods for subclasses to implement.

    Args:
        X (pd.DataFrame): The test dataset features.
        y (pd.Series): The test dataset labels.
        model (sklearn.base.BaseEstimator): A trained sklearn model instance
            for single-model evaluation.
        encoding (Optional[str]): Encoding type for categorical features, e.g.,
            'one_hot' or 'target', used for labeling and grouping in plots.
        aggregate (bool): If True, aggregates the importance values of multi-category
            encoded features for interpretability.

    Attributes:
        X (pd.DataFrame): Holds the test dataset features for evaluation.
        y (pd.Series): Holds the test dataset labels for evaluation.
        model (Optional[sklearn.base.BaseEstimator]): The primary model instance used
            for evaluation, if single-model evaluation is performed.
        encoding (Optional[str]): Indicates the encoding type used, which impacts
            plot titles and feature grouping in evaluations.
        aggregate (bool): Indicates whether to aggregate importance values of
            multi-category encoded features, enhancing interpretability in feature
            importance plots.

    Methods:
        calibration_plot: Plots calibration plot for model probabilities.
        brier_score_groups: Calculates Brier score within specified groups.
        bss_comparison: Compares Brier Skill Score of model with baseline.
        plot_confusion_matrix: Generates a styled confusion matrix heatmap
            for the test data and model predictions.

    Inherited Methods:
        - `brier_scores`: Calculates Brier score for each instance in the evaluator's
            dataset based on the model's predicted probabilities. Returns series of
            Brier scores indexed by instance.
        - `model_predictions`: Generates model predictions for evaluator's feature
            set, applying threshold-based binarization if specified, and returns
            predictions as a series indexed by instance.

    Abstract Methods:
        - `evaluate_feature_importance`: Abstract method for evaluating feature
            importance across models.
        - `analyze_brier_within_clusters`: Abstract method for analyzing Brier
            score distribution within clusters.
    """

    def __init__(
        self,
        X: pd.DataFrame,
        y: pd.Series,
        model: Union[
            RandomForestClassifier,
            LogisticRegression,
            MLPClassifier,
            XGBClassifier,
        ],
        encoding: Optional[str],
        aggregate: bool,
    ) -> None:
        """Initialize the FeatureImportance class."""
        super().__init__(X=X, y=y, model=model, encoding=encoding, aggregate=aggregate)

    def calibration_plot(
        self,
        n_bins: int = 10,
        tight_layout: bool = False,
        task: Optional[str] = None,
    ) -> None:
        """Generates calibration plots for the model's predicted probabilities.

        Creates a calibration plot for binary classification or one for each
        class in a multiclass setup.

        Args:
            n_bins (int): Number of bins for plotting.
            tight_layout (bool): If True, applies tight layout to the plot.
                Defaults to False.
            task (Optional[str]): Task name to apply label mapping for the plot.
                Defaults to None.

        Raises:
            ValueError: If the model does not support probability predictions.
        """
        classification = "binary" if self.y.nunique() == 2 else "multiclass"
        probas = get_probs(self.model, classification=classification, X=self.X)

        if task is not None:
            label_mapping = _label_mapping(task)
            plot_title = (
                f"Calibration Plot \n {task_map.get(task, 'Binary')}"
                if classification == "binary"
                else f"Calibration Plot \n{task_map.get(task, 'Multiclass Task')}"
            )
        else:
            plot_title = "Calibration Plot"
            label_name = None

        if classification == "multiclass":
            num_classes = probas.shape[1]
            plt.figure(figsize=(4, 4), dpi=300)
            for class_idx in range(num_classes):
                class_probas = probas[:, class_idx]
                binarized_y = (self.y == class_idx).astype(int)
                prob_true, prob_pred = calibration_curve(
                    binarized_y, class_probas, n_bins=n_bins, strategy="uniform"
                )
                if task is not None:
                    label_name = label_mapping.get(class_idx, f"Class {class_idx}")
                plt.plot(prob_pred, prob_true, marker="o", label=label_name)
            plt.plot([0, 1], [0, 1], "k--", label="Perfect Calibration")
            plt.xlabel("Mean Predicted Probability", fontsize=12)
            plt.ylabel("Fraction of Positives", fontsize=12)
            ax = plt.gca()
            ax.set_aspect("equal", adjustable="box")
            ax.spines["top"].set_visible(False)
            ax.spines["right"].set_visible(False)
            plt.title(plot_title, fontsize=12)
            plt.xticks(fontsize=12)
            plt.yticks(fontsize=12)
            plt.ylim(0, 1)
            plt.xlim(0, 1)
            plt.legend(frameon=False)
            if tight_layout:
                plt.tight_layout()
            plt.show()
        else:
            prob_true, prob_pred = calibration_curve(
                self.y, probas, n_bins=n_bins, strategy="uniform"
            )
            plt.figure(figsize=(4, 4), dpi=300)
            plt.plot(prob_pred, prob_true, marker="o", label="Model", color="#078294")
            plt.plot([0, 1], [0, 1], "k--", label="Perfect Calibration")
            plt.xlabel("Mean Predicted Probability", fontsize=12)
            plt.ylabel("Fraction of Positives", fontsize=12)
            ax = plt.gca()
            ax.set_aspect("equal", adjustable="box")
            ax.spines["top"].set_visible(False)
            ax.spines["right"].set_visible(False)
            plt.xticks(fontsize=12)
            plt.yticks(fontsize=12)
            plt.title(plot_title, fontsize=12)
            plt.ylim(0, 1)
            plt.xlim(0, 1)
            plt.legend(frameon=False)

            if tight_layout:
                plt.tight_layout()
            plt.show()

    def brier_score_groups(
        self,
        group_by: str = "y",
        task: Optional[str] = None,
        tight_layout: bool = False,
    ) -> None:
        """Calculates and displays Brier score within groups.

        Args:
            group_by (str): Grouping variable for calculating Brier scores.
                Defaults to "y".
            tight_layout (bool): If True, applies tight layout to the plot.
                Defaults to False.
            task (Optional[str]): Task name to apply label mapping for the plot.
                Defaults to None.
        """
        data = pd.DataFrame({group_by: self.y, "Brier_Score": self.brier_scores()})
        if task is not None:
            data[group_by] = data[group_by].map(_label_mapping(task))
        data_grouped = data.groupby(group_by)
        summary = data_grouped["Brier_Score"].agg(["mean", "median"]).reset_index()
        print(f"Average and Median Brier Scores by {group_by}:\n{summary}")

        plt.figure(figsize=(4, 4), dpi=300)
        plt.figure(figsize=(4, 4), dpi=300)
        sns.violinplot(
            x=group_by,
            y="Brier_Score",
            data=data,
            linewidth=0.5,
            color="#078294",
            inner_kws={"box_width": 4, "whis_width": 0.5},
        )
        sns.despine(top=True, right=True)
        plt.title("Distribution of Brier Scores", fontsize=12)
        plt.xlabel(f'{"y" if group_by == "y" else group_by}', fontsize=12)
        plt.ylabel("Brier Score", fontsize=12)
        plt.ylim(0, 1)
        plt.xticks(fontsize=12)
        plt.yticks(fontsize=12)
        if tight_layout:
            plt.tight_layout()
        plt.show()

    def bss_comparison(
        self,
        baseline_models: Dict[Tuple[str, str], Any],
        classification: str,
        tight_layout: bool = False,
        num_patients: Optional[int] = None,
    ) -> None:
        """Compares the Brier Skill Scores (BSS) of the model with baseline models.

        Args:
            baseline_models (Dict[Tuple[str, str], Any]): A dictionary containing
                the baseline models. Keys are tuples of model name and type, and
                values are the trained model objects.
            classification (str): Classification type ('binary' or 'multiclass').
            tight_layout (bool): If True, applies tight layout to the plot.
                Defaults to False.
            num_patients (Optional[int]): Number of unique patients in the test set.
                Defaults to None.

        Raises:
            ValueError: If the model or any baseline model cannot predict probabilities.
        """
        trained_probs = (
            get_probs(self.model, classification=classification, X=self.X)
            if hasattr(self.model, "predict_proba")
            else None
        )

        if classification == "binary":
            trained_brier = brier_score_loss(y_true=self.y, y_proba=trained_probs)
        elif classification == "multiclass":
            trained_brier = brier_loss_multi(y=self.y, probs=trained_probs)
        else:
            raise ValueError(f"Unsupported classification type: {classification}")

        bss_data = []
        brier_scores = [{"Model": "Tuned Model", "Brier Score": trained_brier}]

        for model_name, model in baseline_models.items():
            baseline_probs = (
                get_probs(model, classification=classification, X=self.X)
                if hasattr(model, "predict_proba")
                else None
            )
            if classification == "binary":
                baseline_brier = brier_score_loss(y_true=self.y, y_proba=baseline_probs)
            else:
                baseline_brier = brier_loss_multi(y=self.y, probs=baseline_probs)

            bss = 1 - (trained_brier / baseline_brier)
            bss_data.append({"Model": model_name[0], "Brier Skill Score": bss})
            brier_scores.append({"Model": model_name[0], "Brier Score": baseline_brier})

        bss_df, brier_df = pd.DataFrame(bss_data), pd.DataFrame(brier_scores)
        df1_melted = brier_df.melt(
            id_vars=["Model"], var_name="Score", value_name="Value"
        )
        df2_melted = bss_df.melt(
            id_vars=["Model"], var_name="Score", value_name="Value"
        )

        combined_df = pd.concat([df1_melted, df2_melted], ignore_index=True)

        fig, axes = plt.subplots(figsize=(8, 4), dpi=300)

        model_order = [
            "Tuned Model",
            "Dummy Classifier",
            "Logistic Regression",
            "Random Forest",
        ]

        g = sns.barplot(
            data=combined_df,
            x="Model",
            y="Value",
            hue="Score",
            order=model_order,
            linewidth=1,
            edgecolor="black",
        )

        plt.axvline(x=0.5, color="gray", linestyle="--", linewidth=1)
        plt.axhline(y=0, color="black", linestyle="-", linewidth=1)

        plt.gca().spines["top"].set_visible(False)
        plt.gca().spines["bottom"].set_visible(False)
        plt.gca().spines["right"].set_visible(False)

        for _, e in enumerate(g.patches):
            if e.get_height() > 0:
                g.annotate(
                    f"{e.get_height():.2f}",
                    (e.get_x() + e.get_width() / 2, e.get_height()),
                    ha="center",
                    va="center",
                    fontsize=8,
                    color="black",
                    xytext=(0, 5),
                    textcoords="offset points",
                )
            if e.get_height() < 0:
                g.annotate(
                    f"{e.get_height():.2f}",
                    (e.get_x() + e.get_width() / 2, e.get_height()),
                    ha="center",
                    va="center",
                    fontsize=8,
                    color="black",
                    xytext=(0, -10),
                    textcoords="offset points",
                )

        plt.legend(
            title="Metric",
            frameon=False,
            fontsize=10,
            bbox_to_anchor=(1.05, 0.5),
            loc="upper left",
        )
        if num_patients is not None:
            plt.title(
                f"Baseline Comparison \n"
                f"Number of Patients {num_patients}; Number of Sites: {len(self.y)}"
            )
        else:
            plt.title(f"Baseline Comparison \n Number of Sites: {len(self.y)}")

        labels = [label.get_text() for label in g.get_xticklabels()]
        g.set_xticklabels([label.replace(" ", "\n") for label in labels])

        if tight_layout:
            plt.tight_layout()
        plt.show()

    def plot_confusion_matrix(
        self,
        col: Optional[pd.Series] = None,
        y_label: str = "True",
        normalize: str = "rows",
        tight_layout: bool = False,
        task: Optional[str] = None,
    ) -> plt.Figure:
        """Generates a styled confusion matrix for the given model and test data.

        Args:
            col (Optional[pd.Series]): Column for y label. Defaults to None.
            y_label (str): Description of y label. Defaults to "True".
            normalize (str, optional): Normalization method ('rows' or 'columns').
                Defaults to 'rows'.
            tight_layout (bool): If True, applies tight layout to the plot.
                Defaults to False.
            task (Optional[str]): Task name to apply label mapping for the plot.
                Defaults to None.

        Returns:
            Figure: Confusion matrix heatmap plot.
        """
        y_true = pd.Series(col if col is not None else self.y)
        pred = self.model_predictions()

        if task is not None:
            label_mapping = _label_mapping(task)
            y_true = y_true.map(label_mapping)
            pred = pd.Series(pred).map(label_mapping).values
            labels = list(label_mapping.values())
        else:
            labels = None

        cm = confusion_matrix(y_true=y_true, y_pred=pred, labels=labels)

        if normalize == "rows":
            row_sums = cm.sum(axis=1, keepdims=True)
            normalized_cm = (cm / row_sums) * 100
        elif normalize == "columns":
            col_sums = cm.sum(axis=0, keepdims=True)
            normalized_cm = (cm / col_sums) * 100
        else:
            raise ValueError("Invalid value for 'normalize'. Use 'rows' or 'columns'.")

        custom_cmap = LinearSegmentedColormap.from_list(
            "teal_cmap", ["#FFFFFF", "#078294"]
        )

        plt.figure(figsize=(6, 4), dpi=300)
        sns.heatmap(
            normalized_cm,
            cmap=custom_cmap,
            fmt="g",
            linewidths=0.5,
            square=True,
            annot=False,
            cbar_kws={"label": "Percent"},
            xticklabels=labels if labels else range(cm.shape[1]),
            yticklabels=labels if labels else range(cm.shape[0]),
        )

        for i in range(len(cm)):
            for j in range(len(cm)):
                plt.text(
                    j + 0.5,
                    i + 0.5,
                    cm[i, j],
                    ha="center",
                    va="center",
                    color="white" if normalized_cm[i, j] > 50 else "black",
                )

        plt.title("Confusion Matrix", fontsize=12)
        plt.xlabel("Predicted", fontsize=12)
        plt.ylabel(y_label, fontsize=12)

        ax = plt.gca()
        ax.xaxis.set_ticks_position("top")
        ax.xaxis.set_label_position("top")
        cbar = ax.collections[0].colorbar
        cbar.outline.set_edgecolor("black")
        cbar.outline.set_linewidth(1)

        ax.add_patch(
            Rectangle(
                (0, 0), cm.shape[1], cm.shape[0], fill=False, edgecolor="black", lw=2
            )
        )

        plt.tick_params(axis="both", which="major", labelsize=12)
        if tight_layout:
            plt.tight_layout()
        plt.show()

    @abstractmethod
    def evaluate_feature_importance(self, importance_types: List[str]):
        """Evaluate the feature importance for a list of trained models.

        Args:
            importance_types (List[str]): Methods of feature importance evaluation:
                'shap', 'permutation', 'standard'.
        """

    @abstractmethod
    def analyze_brier_within_clusters(
        self,
        clustering_algorithm: Type,
        n_clusters: int,
    ):
        """Analyze distribution of Brier scores within clusters formed by input data.

        Args:
            clustering_algorithm (Type): Clustering algorithm class from sklearn to use
                for clustering.
            n_clusters (int): Number of clusters to form.
        """

__init__(X, y, model, encoding, aggregate)

Initialize the FeatureImportance class.

Source code in periomod/evaluation/_baseeval.py

def __init__(
    self,
    X: pd.DataFrame,
    y: pd.Series,
    model: Union[
        RandomForestClassifier,
        LogisticRegression,
        MLPClassifier,
        XGBClassifier,
    ],
    encoding: Optional[str],
    aggregate: bool,
) -> None:
    """Initialize the FeatureImportance class."""
    super().__init__(X=X, y=y, model=model, encoding=encoding, aggregate=aggregate)

analyze_brier_within_clusters(clustering_algorithm, n_clusters) abstractmethod

Analyze distribution of Brier scores within clusters formed by input data.

Parameters:

Name	Type	Description	Default
clustering_algorithm	Type	Clustering algorithm class from sklearn to use for clustering.	required
n_clusters	int	Number of clusters to form.	required

Source code in periomod/evaluation/_baseeval.py

@abstractmethod
def analyze_brier_within_clusters(
    self,
    clustering_algorithm: Type,
    n_clusters: int,
):
    """Analyze distribution of Brier scores within clusters formed by input data.

    Args:
        clustering_algorithm (Type): Clustering algorithm class from sklearn to use
            for clustering.
        n_clusters (int): Number of clusters to form.
    """

brier_score_groups(group_by='y', task=None, tight_layout=False)

Calculates and displays Brier score within groups.

Parameters:

Name	Type	Description	Default
group_by	str	Grouping variable for calculating Brier scores. Defaults to "y".	'y'
tight_layout	bool	If True, applies tight layout to the plot. Defaults to False.	False
task	Optional[str]	Task name to apply label mapping for the plot. Defaults to None.	None

Source code in periomod/evaluation/_baseeval.py

def brier_score_groups(
    self,
    group_by: str = "y",
    task: Optional[str] = None,
    tight_layout: bool = False,
) -> None:
    """Calculates and displays Brier score within groups.

    Args:
        group_by (str): Grouping variable for calculating Brier scores.
            Defaults to "y".
        tight_layout (bool): If True, applies tight layout to the plot.
            Defaults to False.
        task (Optional[str]): Task name to apply label mapping for the plot.
            Defaults to None.
    """
    data = pd.DataFrame({group_by: self.y, "Brier_Score": self.brier_scores()})
    if task is not None:
        data[group_by] = data[group_by].map(_label_mapping(task))
    data_grouped = data.groupby(group_by)
    summary = data_grouped["Brier_Score"].agg(["mean", "median"]).reset_index()
    print(f"Average and Median Brier Scores by {group_by}:\n{summary}")

    plt.figure(figsize=(4, 4), dpi=300)
    plt.figure(figsize=(4, 4), dpi=300)
    sns.violinplot(
        x=group_by,
        y="Brier_Score",
        data=data,
        linewidth=0.5,
        color="#078294",
        inner_kws={"box_width": 4, "whis_width": 0.5},
    )
    sns.despine(top=True, right=True)
    plt.title("Distribution of Brier Scores", fontsize=12)
    plt.xlabel(f'{"y" if group_by == "y" else group_by}', fontsize=12)
    plt.ylabel("Brier Score", fontsize=12)
    plt.ylim(0, 1)
    plt.xticks(fontsize=12)
    plt.yticks(fontsize=12)
    if tight_layout:
        plt.tight_layout()
    plt.show()

bss_comparison(baseline_models, classification, tight_layout=False, num_patients=None)

Compares the Brier Skill Scores (BSS) of the model with baseline models.

Parameters:

Name	Type	Description	Default
baseline_models	Dict[Tuple[str, str], Any]	A dictionary containing the baseline models. Keys are tuples of model name and type, and values are the trained model objects.	required
classification	str	Classification type ('binary' or 'multiclass').	required
tight_layout	bool	If True, applies tight layout to the plot. Defaults to False.	False
num_patients	Optional[int]	Number of unique patients in the test set. Defaults to None.	None

Raises:

Type	Description
ValueError	If the model or any baseline model cannot predict probabilities.

Source code in periomod/evaluation/_baseeval.py

def bss_comparison(
    self,
    baseline_models: Dict[Tuple[str, str], Any],
    classification: str,
    tight_layout: bool = False,
    num_patients: Optional[int] = None,
) -> None:
    """Compares the Brier Skill Scores (BSS) of the model with baseline models.

    Args:
        baseline_models (Dict[Tuple[str, str], Any]): A dictionary containing
            the baseline models. Keys are tuples of model name and type, and
            values are the trained model objects.
        classification (str): Classification type ('binary' or 'multiclass').
        tight_layout (bool): If True, applies tight layout to the plot.
            Defaults to False.
        num_patients (Optional[int]): Number of unique patients in the test set.
            Defaults to None.

    Raises:
        ValueError: If the model or any baseline model cannot predict probabilities.
    """
    trained_probs = (
        get_probs(self.model, classification=classification, X=self.X)
        if hasattr(self.model, "predict_proba")
        else None
    )

    if classification == "binary":
        trained_brier = brier_score_loss(y_true=self.y, y_proba=trained_probs)
    elif classification == "multiclass":
        trained_brier = brier_loss_multi(y=self.y, probs=trained_probs)
    else:
        raise ValueError(f"Unsupported classification type: {classification}")

    bss_data = []
    brier_scores = [{"Model": "Tuned Model", "Brier Score": trained_brier}]

    for model_name, model in baseline_models.items():
        baseline_probs = (
            get_probs(model, classification=classification, X=self.X)
            if hasattr(model, "predict_proba")
            else None
        )
        if classification == "binary":
            baseline_brier = brier_score_loss(y_true=self.y, y_proba=baseline_probs)
        else:
            baseline_brier = brier_loss_multi(y=self.y, probs=baseline_probs)

        bss = 1 - (trained_brier / baseline_brier)
        bss_data.append({"Model": model_name[0], "Brier Skill Score": bss})
        brier_scores.append({"Model": model_name[0], "Brier Score": baseline_brier})

    bss_df, brier_df = pd.DataFrame(bss_data), pd.DataFrame(brier_scores)
    df1_melted = brier_df.melt(
        id_vars=["Model"], var_name="Score", value_name="Value"
    )
    df2_melted = bss_df.melt(
        id_vars=["Model"], var_name="Score", value_name="Value"
    )

    combined_df = pd.concat([df1_melted, df2_melted], ignore_index=True)

    fig, axes = plt.subplots(figsize=(8, 4), dpi=300)

    model_order = [
        "Tuned Model",
        "Dummy Classifier",
        "Logistic Regression",
        "Random Forest",
    ]

    g = sns.barplot(
        data=combined_df,
        x="Model",
        y="Value",
        hue="Score",
        order=model_order,
        linewidth=1,
        edgecolor="black",
    )

    plt.axvline(x=0.5, color="gray", linestyle="--", linewidth=1)
    plt.axhline(y=0, color="black", linestyle="-", linewidth=1)

    plt.gca().spines["top"].set_visible(False)
    plt.gca().spines["bottom"].set_visible(False)
    plt.gca().spines["right"].set_visible(False)

    for _, e in enumerate(g.patches):
        if e.get_height() > 0:
            g.annotate(
                f"{e.get_height():.2f}",
                (e.get_x() + e.get_width() / 2, e.get_height()),
                ha="center",
                va="center",
                fontsize=8,
                color="black",
                xytext=(0, 5),
                textcoords="offset points",
            )
        if e.get_height() < 0:
            g.annotate(
                f"{e.get_height():.2f}",
                (e.get_x() + e.get_width() / 2, e.get_height()),
                ha="center",
                va="center",
                fontsize=8,
                color="black",
                xytext=(0, -10),
                textcoords="offset points",
            )

    plt.legend(
        title="Metric",
        frameon=False,
        fontsize=10,
        bbox_to_anchor=(1.05, 0.5),
        loc="upper left",
    )
    if num_patients is not None:
        plt.title(
            f"Baseline Comparison \n"
            f"Number of Patients {num_patients}; Number of Sites: {len(self.y)}"
        )
    else:
        plt.title(f"Baseline Comparison \n Number of Sites: {len(self.y)}")

    labels = [label.get_text() for label in g.get_xticklabels()]
    g.set_xticklabels([label.replace(" ", "\n") for label in labels])

    if tight_layout:
        plt.tight_layout()
    plt.show()

calibration_plot(n_bins=10, tight_layout=False, task=None)

Generates calibration plots for the model's predicted probabilities.

Creates a calibration plot for binary classification or one for each class in a multiclass setup.

Parameters:

Name	Type	Description	Default
n_bins	int	Number of bins for plotting.	10
tight_layout	bool	If True, applies tight layout to the plot. Defaults to False.	False
task	Optional[str]	Task name to apply label mapping for the plot. Defaults to None.	None

Raises:

Type	Description
ValueError	If the model does not support probability predictions.

Source code in periomod/evaluation/_baseeval.py

def calibration_plot(
    self,
    n_bins: int = 10,
    tight_layout: bool = False,
    task: Optional[str] = None,
) -> None:
    """Generates calibration plots for the model's predicted probabilities.

    Creates a calibration plot for binary classification or one for each
    class in a multiclass setup.

    Args:
        n_bins (int): Number of bins for plotting.
        tight_layout (bool): If True, applies tight layout to the plot.
            Defaults to False.
        task (Optional[str]): Task name to apply label mapping for the plot.
            Defaults to None.

    Raises:
        ValueError: If the model does not support probability predictions.
    """
    classification = "binary" if self.y.nunique() == 2 else "multiclass"
    probas = get_probs(self.model, classification=classification, X=self.X)

    if task is not None:
        label_mapping = _label_mapping(task)
        plot_title = (
            f"Calibration Plot \n {task_map.get(task, 'Binary')}"
            if classification == "binary"
            else f"Calibration Plot \n{task_map.get(task, 'Multiclass Task')}"
        )
    else:
        plot_title = "Calibration Plot"
        label_name = None

    if classification == "multiclass":
        num_classes = probas.shape[1]
        plt.figure(figsize=(4, 4), dpi=300)
        for class_idx in range(num_classes):
            class_probas = probas[:, class_idx]
            binarized_y = (self.y == class_idx).astype(int)
            prob_true, prob_pred = calibration_curve(
                binarized_y, class_probas, n_bins=n_bins, strategy="uniform"
            )
            if task is not None:
                label_name = label_mapping.get(class_idx, f"Class {class_idx}")
            plt.plot(prob_pred, prob_true, marker="o", label=label_name)
        plt.plot([0, 1], [0, 1], "k--", label="Perfect Calibration")
        plt.xlabel("Mean Predicted Probability", fontsize=12)
        plt.ylabel("Fraction of Positives", fontsize=12)
        ax = plt.gca()
        ax.set_aspect("equal", adjustable="box")
        ax.spines["top"].set_visible(False)
        ax.spines["right"].set_visible(False)
        plt.title(plot_title, fontsize=12)
        plt.xticks(fontsize=12)
        plt.yticks(fontsize=12)
        plt.ylim(0, 1)
        plt.xlim(0, 1)
        plt.legend(frameon=False)
        if tight_layout:
            plt.tight_layout()
        plt.show()
    else:
        prob_true, prob_pred = calibration_curve(
            self.y, probas, n_bins=n_bins, strategy="uniform"
        )
        plt.figure(figsize=(4, 4), dpi=300)
        plt.plot(prob_pred, prob_true, marker="o", label="Model", color="#078294")
        plt.plot([0, 1], [0, 1], "k--", label="Perfect Calibration")
        plt.xlabel("Mean Predicted Probability", fontsize=12)
        plt.ylabel("Fraction of Positives", fontsize=12)
        ax = plt.gca()
        ax.set_aspect("equal", adjustable="box")
        ax.spines["top"].set_visible(False)
        ax.spines["right"].set_visible(False)
        plt.xticks(fontsize=12)
        plt.yticks(fontsize=12)
        plt.title(plot_title, fontsize=12)
        plt.ylim(0, 1)
        plt.xlim(0, 1)
        plt.legend(frameon=False)

        if tight_layout:
            plt.tight_layout()
        plt.show()

evaluate_feature_importance(importance_types) abstractmethod

Evaluate the feature importance for a list of trained models.

Parameters:

Name	Type	Description	Default
importance_types	List[str]	Methods of feature importance evaluation: 'shap', 'permutation', 'standard'.	required

Source code in periomod/evaluation/_baseeval.py

@abstractmethod
def evaluate_feature_importance(self, importance_types: List[str]):
    """Evaluate the feature importance for a list of trained models.

    Args:
        importance_types (List[str]): Methods of feature importance evaluation:
            'shap', 'permutation', 'standard'.
    """

plot_confusion_matrix(col=None, y_label='True', normalize='rows', tight_layout=False, task=None)

Generates a styled confusion matrix for the given model and test data.

Parameters:

Name	Type	Description	Default
col	Optional[Series]	Column for y label. Defaults to None.	None
y_label	str	Description of y label. Defaults to "True".	'True'
normalize	str	Normalization method ('rows' or 'columns'). Defaults to 'rows'.	'rows'
tight_layout	bool	If True, applies tight layout to the plot. Defaults to False.	False
task	Optional[str]	Task name to apply label mapping for the plot. Defaults to None.	None

Returns:

Name	Type	Description
Figure	Figure	Confusion matrix heatmap plot.

Source code in periomod/evaluation/_baseeval.py

def plot_confusion_matrix(
    self,
    col: Optional[pd.Series] = None,
    y_label: str = "True",
    normalize: str = "rows",
    tight_layout: bool = False,
    task: Optional[str] = None,
) -> plt.Figure:
    """Generates a styled confusion matrix for the given model and test data.

    Args:
        col (Optional[pd.Series]): Column for y label. Defaults to None.
        y_label (str): Description of y label. Defaults to "True".
        normalize (str, optional): Normalization method ('rows' or 'columns').
            Defaults to 'rows'.
        tight_layout (bool): If True, applies tight layout to the plot.
            Defaults to False.
        task (Optional[str]): Task name to apply label mapping for the plot.
            Defaults to None.

    Returns:
        Figure: Confusion matrix heatmap plot.
    """
    y_true = pd.Series(col if col is not None else self.y)
    pred = self.model_predictions()

    if task is not None:
        label_mapping = _label_mapping(task)
        y_true = y_true.map(label_mapping)
        pred = pd.Series(pred).map(label_mapping).values
        labels = list(label_mapping.values())
    else:
        labels = None

    cm = confusion_matrix(y_true=y_true, y_pred=pred, labels=labels)

    if normalize == "rows":
        row_sums = cm.sum(axis=1, keepdims=True)
        normalized_cm = (cm / row_sums) * 100
    elif normalize == "columns":
        col_sums = cm.sum(axis=0, keepdims=True)
        normalized_cm = (cm / col_sums) * 100
    else:
        raise ValueError("Invalid value for 'normalize'. Use 'rows' or 'columns'.")

    custom_cmap = LinearSegmentedColormap.from_list(
        "teal_cmap", ["#FFFFFF", "#078294"]
    )

    plt.figure(figsize=(6, 4), dpi=300)
    sns.heatmap(
        normalized_cm,
        cmap=custom_cmap,
        fmt="g",
        linewidths=0.5,
        square=True,
        annot=False,
        cbar_kws={"label": "Percent"},
        xticklabels=labels if labels else range(cm.shape[1]),
        yticklabels=labels if labels else range(cm.shape[0]),
    )

    for i in range(len(cm)):
        for j in range(len(cm)):
            plt.text(
                j + 0.5,
                i + 0.5,
                cm[i, j],
                ha="center",
                va="center",
                color="white" if normalized_cm[i, j] > 50 else "black",
            )

    plt.title("Confusion Matrix", fontsize=12)
    plt.xlabel("Predicted", fontsize=12)
    plt.ylabel(y_label, fontsize=12)

    ax = plt.gca()
    ax.xaxis.set_ticks_position("top")
    ax.xaxis.set_label_position("top")
    cbar = ax.collections[0].colorbar
    cbar.outline.set_edgecolor("black")
    cbar.outline.set_linewidth(1)

    ax.add_patch(
        Rectangle(
            (0, 0), cm.shape[1], cm.shape[0], fill=False, edgecolor="black", lw=2
        )
    )

    plt.tick_params(axis="both", which="major", labelsize=12)
    if tight_layout:
        plt.tight_layout()
    plt.show()