BaseBenchmark

Bases: BaseConfig

Base class for benchmarking models on specified tasks with various settings.

This class initializes common parameters for benchmarking, including task specifications, encoding and sampling methods, tuning strategies, and model evaluation criteria.

Inherits

• BaseConfig: Base configuration class providing configuration loading.

Parameters:

Name	Type	Description	Default
task	str	Task for evaluation (pocketclosure', 'pocketclosureinf', 'improvement', or 'pdgrouprevaluation'.).	required
learners	List[str]	List of models or algorithms to benchmark, including 'xgb', 'rf', 'lr' or 'mlp'.	required
tuning_methods	List[str]	List of tuning methods for model training, such as 'holdout' or 'cv'.	required
hpo_methods	List[str]	Hyperparameter optimization strategies to apply, includes 'rs' and 'hebo'.	required
criteria	List[str]	List of evaluation criteria ('f1', 'macro_f1', 'brier_score').	required
encodings	List[str]	Encoding types to transform categorical features, can either be 'one_hot' or 'target' encoding.	required
sampling	Optional[List[Union[str, None]]]	Sampling strategies to handle class imbalance, options include None, 'upsampling', 'downsampling', or 'smote'.	required
factor	Optional[float]	Factor specifying the amount of sampling to apply during resampling, if applicable.	required
n_configs	int	Number of configurations to evaluate in hyperparameter tuning.	required
n_jobs	int	Number of parallel jobs to use for processing; set to -1 to utilize all available cores.	required
cv_folds	int	Number of cross-validation folds for model training. Defaults to None.	required
racing_folds	Optional[int]	Number of racing folds to use in Random Search (rs) for optimized tuning.	required
test_seed	int	Random seed for reproducible train-test splits.	required
test_size	float	Fraction of the dataset to allocate to test set.	required
val_size	float	Fraction of the dataset to allocate to validation in a holdout setup.	required
cv_seed	int	Seed for cross-validation splitting.	required
mlp_flag	bool	If True, enables Multi-Layer Perceptron (MLP) training with early stopping.	required
threshold_tuning	bool	Enables decision threshold tuning for binary classification when optimizing for 'f1'.	required
verbose	bool	Enables detailed logging of processes if set to True.	required
path	Path	Directory path where processed data will be stored.	required

Attributes:

Name	Type	Description
task	str	Task used for model classification or regression evaluation.
learners	List[str]	Selected models or algorithms for benchmarking.
tuning_methods	List[str]	List of model tuning approaches.
hpo_methods	List[str]	Hyperparameter optimization techniques to apply.
criteria	List[str]	Criteria used to evaluate model performance.
encodings	List[str]	Encoding methods applied to categorical features.
sampling	Optional[List[Union[str, None]]]	Sampling strategies employed to address class imbalance.
factor	Optional[float]	Specifies the degree of sampling applied within the chosen strategy.
n_configs	int	Number of configurations assessed during hyperparameter optimization.
n_jobs	int	Number of parallel processes for model training and evaluation.
cv_folds	int	Number of cross-validation folds for model training.
racing_folds	Optional[int]	Racing folds used in tuning with cross-validation and random search..
test_seed	int	Seed for consistent test-train splitting.
test_size	float	Proportion of the data set aside for testing.
val_size	float	Proportion of data allocated to validation in holdout tuning.
cv_seed	int	Seed for cross-validation splitting.
mlp_flag	bool	Flag for MLP training with early stopping.
threshold_tuning	bool	Enables threshold adjustment for optimizing F1 in binary classification tasks.
verbose	bool	Flag to enable detailed logging during training and evaluation.
path	Path	Path where processed data is saved.

Source code in periomod/benchmarking/_basebenchmark.py

class BaseBenchmark(BaseConfig):
    """Base class for benchmarking models on specified tasks with various settings.

    This class initializes common parameters for benchmarking, including task
    specifications, encoding and sampling methods, tuning strategies, and model
    evaluation criteria.

    Inherits:
        - `BaseConfig`: Base configuration class providing configuration loading.

    Args:
        task (str): Task for evaluation (pocketclosure', 'pocketclosureinf',
            'improvement', or 'pdgrouprevaluation'.).
        learners (List[str]): List of models or algorithms to benchmark,
            including 'xgb', 'rf', 'lr' or 'mlp'.
        tuning_methods (List[str]): List of tuning methods for model training,
            such as 'holdout' or 'cv'.
        hpo_methods (List[str]): Hyperparameter optimization strategies to apply,
            includes 'rs' and 'hebo'.
        criteria (List[str]): List of evaluation criteria ('f1', 'macro_f1',
            'brier_score').
        encodings (List[str]): Encoding types to transform categorical features,
            can either be 'one_hot' or 'target' encoding.
        sampling (Optional[List[Union[str, None]]]): Sampling strategies to handle
            class imbalance, options include None, 'upsampling', 'downsampling', or
            'smote'.
        factor (Optional[float]): Factor specifying the amount of sampling to apply
            during resampling, if applicable.
        n_configs (int): Number of configurations to evaluate in hyperparameter tuning.
        n_jobs (int): Number of parallel jobs to use for processing; set
            to -1 to utilize all available cores.
        cv_folds (int): Number of cross-validation folds for model
            training. Defaults to None.
        racing_folds (Optional[int]): Number of racing folds to use in Random Search
            (rs) for optimized tuning.
        test_seed (int): Random seed for reproducible train-test splits.
        test_size (float): Fraction of the dataset to allocate to test set.
        val_size (float): Fraction of the dataset to allocate to validation
            in a holdout setup.
        cv_seed (int): Seed for cross-validation splitting.
        mlp_flag (bool): If True, enables Multi-Layer Perceptron (MLP)
            training with early stopping.
        threshold_tuning (bool): Enables decision threshold tuning for binary
            classification when optimizing for 'f1'.
        verbose (bool): Enables detailed logging of processes if set to True.
        path (Path): Directory path where processed data will be stored.

    Attributes:
        task (str): Task used for model classification or regression evaluation.
        learners (List[str]): Selected models or algorithms for benchmarking.
        tuning_methods (List[str]): List of model tuning approaches.
        hpo_methods (List[str]): Hyperparameter optimization techniques to apply.
        criteria (List[str]): Criteria used to evaluate model performance.
        encodings (List[str]): Encoding methods applied to categorical features.
        sampling (Optional[List[Union[str, None]]]): Sampling strategies employed
            to address class imbalance.
        factor (Optional[float]): Specifies the degree of sampling applied
            within the chosen strategy.
        n_configs (int): Number of configurations assessed during hyperparameter
            optimization.
        n_jobs (int): Number of parallel processes for model training
            and evaluation.
        cv_folds (int): Number of cross-validation folds for model training.
        racing_folds (Optional[int]): Racing folds used in tuning with cross-validation
            and random search..
        test_seed (int): Seed for consistent test-train splitting.
        test_size (float): Proportion of the data set aside for testing.
        val_size (float): Proportion of data allocated to validation
            in holdout tuning.
        cv_seed (int): Seed for cross-validation splitting.
        mlp_flag (bool): Flag for MLP training with early stopping.
        threshold_tuning (bool): Enables threshold adjustment for optimizing F1
            in binary classification tasks.
        verbose (bool): Flag to enable detailed logging during training and evaluation.
        path (Path): Path where processed data is saved.

    """

    def __init__(
        self,
        task: str,
        learners: List[str],
        tuning_methods: List[str],
        hpo_methods: List[str],
        criteria: List[str],
        encodings: List[str],
        sampling: Optional[List[Union[str, None]]],
        factor: Optional[float],
        n_configs: int,
        n_jobs: int,
        cv_folds: Optional[int],
        racing_folds: Optional[int],
        test_seed: int,
        test_size: float,
        val_size: Optional[float],
        cv_seed: Optional[int],
        mlp_flag: Optional[bool],
        threshold_tuning: Optional[bool],
        verbose: bool,
        path: Path,
    ) -> None:
        """Initialize the base benchmark class with common parameters."""
        super().__init__()
        self.task = task
        self.learners = learners
        self.tuning_methods = tuning_methods
        self.hpo_methods = hpo_methods
        self.criteria = criteria
        self.encodings = encodings
        self.sampling = sampling
        self.factor = factor
        self.n_configs = n_configs
        self.n_jobs = n_jobs
        self.verbose = verbose
        self.cv_folds = cv_folds
        self.racing_folds = racing_folds
        self.test_seed = test_seed
        self.test_size = test_size
        self.val_size = val_size
        self.cv_seed = cv_seed
        self.mlp_flag = mlp_flag
        self.threshold_tuning = threshold_tuning
        self.path = path
        self._validate_task()

    def _validate_task(self) -> None:
        """Validates the task type for the model.

        Raises:
            ValueError: If `self.task` is not one of the recognized task types.

        Supported task types:
            - "pocketclosure"
            - "pocketclosureinf"
            - "improvement"
            - "pdgrouprevaluation"
        """
        if self.task not in {
            "pocketclosure",
            "pocketclosureinf",
            "improvement",
            "pdgrouprevaluation",
        }:
            raise ValueError(
                f"Unknown task: {self.task}. Unable to determine classification."
            )

__init__(task, learners, tuning_methods, hpo_methods, criteria, encodings, sampling, factor, n_configs, n_jobs, cv_folds, racing_folds, test_seed, test_size, val_size, cv_seed, mlp_flag, threshold_tuning, verbose, path)

Initialize the base benchmark class with common parameters.

Source code in periomod/benchmarking/_basebenchmark.py

def __init__(
    self,
    task: str,
    learners: List[str],
    tuning_methods: List[str],
    hpo_methods: List[str],
    criteria: List[str],
    encodings: List[str],
    sampling: Optional[List[Union[str, None]]],
    factor: Optional[float],
    n_configs: int,
    n_jobs: int,
    cv_folds: Optional[int],
    racing_folds: Optional[int],
    test_seed: int,
    test_size: float,
    val_size: Optional[float],
    cv_seed: Optional[int],
    mlp_flag: Optional[bool],
    threshold_tuning: Optional[bool],
    verbose: bool,
    path: Path,
) -> None:
    """Initialize the base benchmark class with common parameters."""
    super().__init__()
    self.task = task
    self.learners = learners
    self.tuning_methods = tuning_methods
    self.hpo_methods = hpo_methods
    self.criteria = criteria
    self.encodings = encodings
    self.sampling = sampling
    self.factor = factor
    self.n_configs = n_configs
    self.n_jobs = n_jobs
    self.verbose = verbose
    self.cv_folds = cv_folds
    self.racing_folds = racing_folds
    self.test_seed = test_seed
    self.test_size = test_size
    self.val_size = val_size
    self.cv_seed = cv_seed
    self.mlp_flag = mlp_flag
    self.threshold_tuning = threshold_tuning
    self.path = path
    self._validate_task()