Documentation

Documentation

valor_lite.object_detection.BoundingBox dataclass

Represents a bounding box with associated labels and optional scores.

Parameters:

Name	Type	Description	Default
uid	str	A unique identifier.	required
xmin	float	The minimum x-coordinate of the bounding box.	required
xmax	float	The maximum x-coordinate of the bounding box.	required
ymin	float	The minimum y-coordinate of the bounding box.	required
ymax	float	The maximum y-coordinate of the bounding box.	required
labels	list of str	List of labels associated with the bounding box.	required
scores	list of float	Confidence scores corresponding to each label. Defaults to an empty list.	list()

Examples:

Ground Truth Example:

>>> bbox = BoundingBox(uid="xyz", xmin=10.0, xmax=50.0, ymin=20.0, ymax=60.0, labels=['cat'])

Prediction Example:

>>> bbox = BoundingBox(
...     uid="abc",
...     xmin=10.0, xmax=50.0, ymin=20.0, ymax=60.0,
...     labels=['cat', 'dog'], scores=[0.9, 0.1]
... )

Source code in valor_lite/object_detection/annotation.py

@dataclass
class BoundingBox:
    """
    Represents a bounding box with associated labels and optional scores.

    Parameters
    ----------
    uid : str
        A unique identifier.
    xmin : float
        The minimum x-coordinate of the bounding box.
    xmax : float
        The maximum x-coordinate of the bounding box.
    ymin : float
        The minimum y-coordinate of the bounding box.
    ymax : float
        The maximum y-coordinate of the bounding box.
    labels : list of str
        List of labels associated with the bounding box.
    scores : list of float, optional
        Confidence scores corresponding to each label. Defaults to an empty list.

    Examples
    --------
    Ground Truth Example:

    >>> bbox = BoundingBox(uid="xyz", xmin=10.0, xmax=50.0, ymin=20.0, ymax=60.0, labels=['cat'])

    Prediction Example:

    >>> bbox = BoundingBox(
    ...     uid="abc",
    ...     xmin=10.0, xmax=50.0, ymin=20.0, ymax=60.0,
    ...     labels=['cat', 'dog'], scores=[0.9, 0.1]
    ... )
    """

    uid: str
    xmin: float
    xmax: float
    ymin: float
    ymax: float
    labels: list[str]
    scores: list[float] = field(default_factory=list)

    def __post_init__(self):
        if len(self.scores) == 0 and len(self.labels) != 1:
            raise ValueError(
                "Ground truths must be defined with no scores and a single label. If you meant to define a prediction, then please include one score for every label provided."
            )
        if len(self.scores) > 0 and len(self.labels) != len(self.scores):
            raise ValueError(
                "If scores are defined, there must be a 1:1 pairing with labels."
            )

    @property
    def extrema(self) -> tuple[float, float, float, float]:
        """
        Returns the annotation's data representation.

        Returns
        -------
        tuple[float, float, float, float]
            A tuple in the form (xmin, xmax, ymin, ymax).
        """
        return (self.xmin, self.xmax, self.ymin, self.ymax)

extrema property

Returns the annotation's data representation.

Returns:

Type	Description
tuple[float, float, float, float]	A tuple in the form (xmin, xmax, ymin, ymax).

valor_lite.object_detection.Polygon dataclass

Represents a polygon shape with associated labels and optional scores.

Parameters:

Name	Type	Description	Default
uid	str	A unique identifier.	required
shape	Polygon	A Shapely polygon object representing the shape.	required
labels	list of str	List of labels associated with the polygon.	required
scores	list of float	Confidence scores corresponding to each label. Defaults to an empty list.	list()

Examples:

Ground Truth Example:

>>> from shapely.geometry import Polygon as ShapelyPolygon
>>> shape = ShapelyPolygon([(0, 0), (1, 0), (1, 1), (0, 1)])
>>> polygon = Polygon(uid="xyz", shape=shape, labels=['building'])

Prediction Example:

>>> polygon = Polygon(
...     uid="abc", shape=shape, labels=['building'], scores=[0.95]
... )

Source code in valor_lite/object_detection/annotation.py

@dataclass
class Polygon:
    """
    Represents a polygon shape with associated labels and optional scores.

    Parameters
    ----------
    uid : str
        A unique identifier.
    shape : shapely.geometry.Polygon
        A Shapely polygon object representing the shape.
    labels : list of str
        List of labels associated with the polygon.
    scores : list of float, optional
        Confidence scores corresponding to each label. Defaults to an empty list.

    Examples
    --------
    Ground Truth Example:

    >>> from shapely.geometry import Polygon as ShapelyPolygon
    >>> shape = ShapelyPolygon([(0, 0), (1, 0), (1, 1), (0, 1)])
    >>> polygon = Polygon(uid="xyz", shape=shape, labels=['building'])

    Prediction Example:

    >>> polygon = Polygon(
    ...     uid="abc", shape=shape, labels=['building'], scores=[0.95]
    ... )
    """

    uid: str
    shape: ShapelyPolygon
    labels: list[str]
    scores: list[float] = field(default_factory=list)

    def __post_init__(self):
        if not isinstance(self.shape, ShapelyPolygon):
            raise TypeError("shape must be of type shapely.geometry.Polygon.")
        if self.shape.is_empty:
            raise ValueError("Polygon is empty.")

        if len(self.scores) == 0 and len(self.labels) != 1:
            raise ValueError(
                "Ground truths must be defined with no scores and a single label. If you meant to define a prediction, then please include one score for every label provided."
            )
        if len(self.scores) > 0 and len(self.labels) != len(self.scores):
            raise ValueError(
                "If scores are defined, there must be a 1:1 pairing with labels."
            )

valor_lite.object_detection.Bitmask dataclass

Represents a binary mask with associated labels and optional scores.

Parameters:

Name	Type	Description	Default
uid	str	A unique identifier.	required
mask	NDArray[bool_]	A NumPy array of boolean values representing the mask.	required
labels	list of str	List of labels associated with the mask.	required
scores	list of float	Confidence scores corresponding to each label. Defaults to an empty list.	list()

Examples:

Ground Truth Example:

>>> import numpy as np
>>> mask = np.array([[True, False], [False, True]], dtype=np.bool_)
>>> bitmask = Bitmask(uid="abc", mask=mask, labels=['tree'])

Prediction Example:

>>> bitmask = Bitmask(
...     uid="xyz", mask=mask, labels=['tree'], scores=[0.85]
... )

Source code in valor_lite/object_detection/annotation.py

@dataclass
class Bitmask:
    """
    Represents a binary mask with associated labels and optional scores.

    Parameters
    ----------
    uid : str
        A unique identifier.
    mask : NDArray[np.bool_]
        A NumPy array of boolean values representing the mask.
    labels : list of str
        List of labels associated with the mask.
    scores : list of float, optional
        Confidence scores corresponding to each label. Defaults to an empty list.

    Examples
    --------
    Ground Truth Example:

    >>> import numpy as np
    >>> mask = np.array([[True, False], [False, True]], dtype=np.bool_)
    >>> bitmask = Bitmask(uid="abc", mask=mask, labels=['tree'])

    Prediction Example:

    >>> bitmask = Bitmask(
    ...     uid="xyz", mask=mask, labels=['tree'], scores=[0.85]
    ... )
    """

    uid: str
    mask: NDArray[np.bool_]
    labels: list[str]
    scores: list[float] = field(default_factory=list)

    def __post_init__(self):

        if (
            not isinstance(self.mask, np.ndarray)
            or self.mask.dtype != np.bool_
        ):
            raise ValueError(
                "Expected mask to be of type `NDArray[np.bool_]`."
            )
        elif not self.mask.any():
            raise ValueError("Mask does not define any object instances.")

        if len(self.scores) == 0 and len(self.labels) != 1:
            raise ValueError(
                "Ground truths must be defined with no scores and a single label. If you meant to define a prediction, then please include one score for every label provided."
            )
        if len(self.scores) > 0 and len(self.labels) != len(self.scores):
            raise ValueError(
                "If scores are defined, there must be a 1:1 pairing with labels."
            )

valor_lite.object_detection.Detection dataclass

Bases: Generic[AnnotationType]

Detection data structure holding ground truths and predictions for object detection tasks.

Parameters:

Name	Type	Description	Default
uid	str	Unique identifier for the image or sample.	required
groundtruths	list[BoundingBox] | list[Polygon] | list[Bitmask]	List of ground truth annotations.	required
predictions	list[BoundingBox] | list[Polygon] | list[Bitmask]	List of predicted annotations.	required

Examples:

>>> bbox_gt = BoundingBox(xmin=10, xmax=50, ymin=20, ymax=60, labels=['cat'])
>>> bbox_pred = BoundingBox(
...     xmin=12, xmax=48, ymin=22, ymax=58, labels=['cat'], scores=[0.9]
... )
>>> detection = Detection(
...     uid='image_001',
...     groundtruths=[bbox_gt],
...     predictions=[bbox_pred]
... )

Source code in valor_lite/object_detection/annotation.py

@dataclass
class Detection(Generic[AnnotationType]):
    """
    Detection data structure holding ground truths and predictions for object detection tasks.

    Parameters
    ----------
    uid : str
        Unique identifier for the image or sample.
    groundtruths : list[BoundingBox] | list[Polygon] | list[Bitmask]
        List of ground truth annotations.
    predictions : list[BoundingBox] | list[Polygon] | list[Bitmask]
        List of predicted annotations.

    Examples
    --------
    >>> bbox_gt = BoundingBox(xmin=10, xmax=50, ymin=20, ymax=60, labels=['cat'])
    >>> bbox_pred = BoundingBox(
    ...     xmin=12, xmax=48, ymin=22, ymax=58, labels=['cat'], scores=[0.9]
    ... )
    >>> detection = Detection(
    ...     uid='image_001',
    ...     groundtruths=[bbox_gt],
    ...     predictions=[bbox_pred]
    ... )
    """

    uid: str
    groundtruths: list[AnnotationType]
    predictions: list[AnnotationType]

    def __post_init__(self):
        for prediction in self.predictions:
            if len(prediction.scores) != len(prediction.labels):
                raise ValueError(
                    "Predictions must provide a score for every label."
                )

valor_lite.object_detection.DataLoader

Object Detection DataLoader

Source code in valor_lite/object_detection/manager.py

class DataLoader:
    """
    Object Detection DataLoader
    """

    def __init__(self):
        self._evaluator = Evaluator()
        self.pairs: list[NDArray[np.float64]] = list()

    def _add_datum(self, datum_id: str) -> int:
        """
        Helper function for adding a datum to the cache.

        Parameters
        ----------
        datum_id : str
            The datum identifier.

        Returns
        -------
        int
            The datum index.
        """
        if datum_id not in self._evaluator.datum_id_to_index:
            if len(self._evaluator.datum_id_to_index) != len(
                self._evaluator.index_to_datum_id
            ):
                raise InternalCacheError("datum cache size mismatch")
            idx = len(self._evaluator.datum_id_to_index)
            self._evaluator.datum_id_to_index[datum_id] = idx
            self._evaluator.index_to_datum_id.append(datum_id)
        return self._evaluator.datum_id_to_index[datum_id]

    def _add_groundtruth(self, annotation_id: str) -> int:
        """
        Helper function for adding a ground truth annotation identifier to the cache.

        Parameters
        ----------
        annotation_id : str
            The ground truth annotation identifier.

        Returns
        -------
        int
            The ground truth annotation index.
        """
        if annotation_id not in self._evaluator.groundtruth_id_to_index:
            if len(self._evaluator.groundtruth_id_to_index) != len(
                self._evaluator.index_to_groundtruth_id
            ):
                raise InternalCacheError("ground truth cache size mismatch")
            idx = len(self._evaluator.groundtruth_id_to_index)
            self._evaluator.groundtruth_id_to_index[annotation_id] = idx
            self._evaluator.index_to_groundtruth_id.append(annotation_id)
        return self._evaluator.groundtruth_id_to_index[annotation_id]

    def _add_prediction(self, annotation_id: str) -> int:
        """
        Helper function for adding a prediction annotation identifier to the cache.

        Parameters
        ----------
        annotation_id : str
            The prediction annotation identifier.

        Returns
        -------
        int
            The prediction annotation index.
        """
        if annotation_id not in self._evaluator.prediction_id_to_index:
            if len(self._evaluator.prediction_id_to_index) != len(
                self._evaluator.index_to_prediction_id
            ):
                raise InternalCacheError("prediction cache size mismatch")
            idx = len(self._evaluator.prediction_id_to_index)
            self._evaluator.prediction_id_to_index[annotation_id] = idx
            self._evaluator.index_to_prediction_id.append(annotation_id)
        return self._evaluator.prediction_id_to_index[annotation_id]

    def _add_label(self, label: str) -> int:
        """
        Helper function for adding a label to the cache.

        Parameters
        ----------
        label : str
            The label associated with the annotation.

        Returns
        -------
        int
            Label index.
        """
        label_id = len(self._evaluator.index_to_label)
        if label not in self._evaluator.label_to_index:
            if len(self._evaluator.label_to_index) != len(
                self._evaluator.index_to_label
            ):
                raise InternalCacheError("label cache size mismatch")
            self._evaluator.label_to_index[label] = label_id
            self._evaluator.index_to_label.append(label)
            label_id += 1
        return self._evaluator.label_to_index[label]

    def _add_data(
        self,
        detections: list[Detection],
        detection_ious: list[NDArray[np.float64]],
        show_progress: bool = False,
    ):
        """
        Adds detections to the cache.

        Parameters
        ----------
        detections : list[Detection]
            A list of Detection objects.
        detection_ious : list[NDArray[np.float64]]
            A list of arrays containing IOUs per detection.
        show_progress : bool, default=False
            Toggle for tqdm progress bar.
        """
        disable_tqdm = not show_progress
        for detection, ious in tqdm(
            zip(detections, detection_ious), disable=disable_tqdm
        ):
            # cache labels and annotation pairs
            pairs = []
            datum_idx = self._add_datum(detection.uid)
            if detection.groundtruths:
                for gidx, gann in enumerate(detection.groundtruths):
                    groundtruth_idx = self._add_groundtruth(gann.uid)
                    glabel_idx = self._add_label(gann.labels[0])
                    if (ious[:, gidx] < 1e-9).all():
                        pairs.extend(
                            [
                                np.array(
                                    [
                                        float(datum_idx),
                                        float(groundtruth_idx),
                                        -1.0,
                                        float(glabel_idx),
                                        -1.0,
                                        0.0,
                                        -1.0,
                                    ]
                                )
                            ]
                        )
                    for pidx, pann in enumerate(detection.predictions):
                        prediction_idx = self._add_prediction(pann.uid)
                        if (ious[pidx, :] < 1e-9).all():
                            pairs.extend(
                                [
                                    np.array(
                                        [
                                            float(datum_idx),
                                            -1.0,
                                            float(prediction_idx),
                                            -1.0,
                                            float(self._add_label(plabel)),
                                            0.0,
                                            float(pscore),
                                        ]
                                    )
                                    for plabel, pscore in zip(
                                        pann.labels, pann.scores
                                    )
                                ]
                            )
                        if ious[pidx, gidx] >= 1e-9:
                            pairs.extend(
                                [
                                    np.array(
                                        [
                                            float(datum_idx),
                                            float(groundtruth_idx),
                                            float(prediction_idx),
                                            float(self._add_label(glabel)),
                                            float(self._add_label(plabel)),
                                            ious[pidx, gidx],
                                            float(pscore),
                                        ]
                                    )
                                    for glabel in gann.labels
                                    for plabel, pscore in zip(
                                        pann.labels, pann.scores
                                    )
                                ]
                            )
            elif detection.predictions:
                for pidx, pann in enumerate(detection.predictions):
                    prediction_idx = self._add_prediction(pann.uid)
                    pairs.extend(
                        [
                            np.array(
                                [
                                    float(datum_idx),
                                    -1.0,
                                    float(prediction_idx),
                                    -1.0,
                                    float(self._add_label(plabel)),
                                    0.0,
                                    float(pscore),
                                ]
                            )
                            for plabel, pscore in zip(pann.labels, pann.scores)
                        ]
                    )

            data = np.array(pairs)
            if data.size > 0:
                self.pairs.append(data)

    def add_bounding_boxes(
        self,
        detections: list[Detection[BoundingBox]],
        show_progress: bool = False,
    ):
        """
        Adds bounding box detections to the cache.

        Parameters
        ----------
        detections : list[Detection]
            A list of Detection objects.
        show_progress : bool, default=False
            Toggle for tqdm progress bar.
        """
        ious = [
            compute_bbox_iou(
                np.array(
                    [
                        [gt.extrema, pd.extrema]
                        for pd in detection.predictions
                        for gt in detection.groundtruths
                    ],
                    dtype=np.float64,
                )
            ).reshape(len(detection.predictions), len(detection.groundtruths))
            for detection in detections
        ]
        return self._add_data(
            detections=detections,
            detection_ious=ious,
            show_progress=show_progress,
        )

    def add_polygons(
        self,
        detections: list[Detection[Polygon]],
        show_progress: bool = False,
    ):
        """
        Adds polygon detections to the cache.

        Parameters
        ----------
        detections : list[Detection]
            A list of Detection objects.
        show_progress : bool, default=False
            Toggle for tqdm progress bar.
        """
        ious = [
            compute_polygon_iou(
                np.array(
                    [
                        [gt.shape, pd.shape]
                        for pd in detection.predictions
                        for gt in detection.groundtruths
                    ]
                )
            ).reshape(len(detection.predictions), len(detection.groundtruths))
            for detection in detections
        ]
        return self._add_data(
            detections=detections,
            detection_ious=ious,
            show_progress=show_progress,
        )

    def add_bitmasks(
        self,
        detections: list[Detection[Bitmask]],
        show_progress: bool = False,
    ):
        """
        Adds bitmask detections to the cache.

        Parameters
        ----------
        detections : list[Detection]
            A list of Detection objects.
        show_progress : bool, default=False
            Toggle for tqdm progress bar.
        """
        ious = [
            compute_bitmask_iou(
                np.array(
                    [
                        [gt.mask, pd.mask]
                        for pd in detection.predictions
                        for gt in detection.groundtruths
                    ]
                )
            ).reshape(len(detection.predictions), len(detection.groundtruths))
            for detection in detections
        ]
        return self._add_data(
            detections=detections,
            detection_ious=ious,
            show_progress=show_progress,
        )

    def finalize(self) -> Evaluator:
        """
        Performs data finalization and some preprocessing steps.

        Returns
        -------
        Evaluator
            A ready-to-use evaluator object.
        """
        if not self.pairs:
            raise EmptyEvaluatorError()

        n_labels = len(self._evaluator.index_to_label)
        n_datums = len(self._evaluator.index_to_datum_id)

        self._evaluator._detailed_pairs = np.concatenate(self.pairs, axis=0)
        if self._evaluator._detailed_pairs.size == 0:
            raise EmptyEvaluatorError()

        # order pairs by descending score, iou
        indices = np.lexsort(
            (
                -self._evaluator._detailed_pairs[:, 5],  # iou
                -self._evaluator._detailed_pairs[:, 6],  # score
            )
        )
        self._evaluator._detailed_pairs = self._evaluator._detailed_pairs[
            indices
        ]
        self._evaluator._label_metadata = compute_label_metadata(
            ids=self._evaluator._detailed_pairs[:, :5].astype(np.int32),
            n_labels=n_labels,
        )
        self._evaluator._ranked_pairs = rank_pairs(
            detailed_pairs=self._evaluator._detailed_pairs,
            label_metadata=self._evaluator._label_metadata,
        )
        self._evaluator._metadata = Metadata.create(
            detailed_pairs=self._evaluator._detailed_pairs,
            number_of_datums=n_datums,
            number_of_labels=n_labels,
        )
        return self._evaluator

add_bitmasks(detections, show_progress=False)

Adds bitmask detections to the cache.

Parameters:

Name	Type	Description	Default
detections	list[Detection]	A list of Detection objects.	required
show_progress	bool	Toggle for tqdm progress bar.	False

Source code in valor_lite/object_detection/manager.py

def add_bitmasks(
    self,
    detections: list[Detection[Bitmask]],
    show_progress: bool = False,
):
    """
    Adds bitmask detections to the cache.

    Parameters
    ----------
    detections : list[Detection]
        A list of Detection objects.
    show_progress : bool, default=False
        Toggle for tqdm progress bar.
    """
    ious = [
        compute_bitmask_iou(
            np.array(
                [
                    [gt.mask, pd.mask]
                    for pd in detection.predictions
                    for gt in detection.groundtruths
                ]
            )
        ).reshape(len(detection.predictions), len(detection.groundtruths))
        for detection in detections
    ]
    return self._add_data(
        detections=detections,
        detection_ious=ious,
        show_progress=show_progress,
    )

add_bounding_boxes(detections, show_progress=False)

Adds bounding box detections to the cache.

Parameters:

Name	Type	Description	Default
detections	list[Detection]	A list of Detection objects.	required
show_progress	bool	Toggle for tqdm progress bar.	False

Source code in valor_lite/object_detection/manager.py

def add_bounding_boxes(
    self,
    detections: list[Detection[BoundingBox]],
    show_progress: bool = False,
):
    """
    Adds bounding box detections to the cache.

    Parameters
    ----------
    detections : list[Detection]
        A list of Detection objects.
    show_progress : bool, default=False
        Toggle for tqdm progress bar.
    """
    ious = [
        compute_bbox_iou(
            np.array(
                [
                    [gt.extrema, pd.extrema]
                    for pd in detection.predictions
                    for gt in detection.groundtruths
                ],
                dtype=np.float64,
            )
        ).reshape(len(detection.predictions), len(detection.groundtruths))
        for detection in detections
    ]
    return self._add_data(
        detections=detections,
        detection_ious=ious,
        show_progress=show_progress,
    )

add_polygons(detections, show_progress=False)

Adds polygon detections to the cache.

Parameters:

Name	Type	Description	Default
detections	list[Detection]	A list of Detection objects.	required
show_progress	bool	Toggle for tqdm progress bar.	False

Source code in valor_lite/object_detection/manager.py

def add_polygons(
    self,
    detections: list[Detection[Polygon]],
    show_progress: bool = False,
):
    """
    Adds polygon detections to the cache.

    Parameters
    ----------
    detections : list[Detection]
        A list of Detection objects.
    show_progress : bool, default=False
        Toggle for tqdm progress bar.
    """
    ious = [
        compute_polygon_iou(
            np.array(
                [
                    [gt.shape, pd.shape]
                    for pd in detection.predictions
                    for gt in detection.groundtruths
                ]
            )
        ).reshape(len(detection.predictions), len(detection.groundtruths))
        for detection in detections
    ]
    return self._add_data(
        detections=detections,
        detection_ious=ious,
        show_progress=show_progress,
    )

finalize()

Performs data finalization and some preprocessing steps.

Returns:

Type	Description
Evaluator	A ready-to-use evaluator object.

Source code in valor_lite/object_detection/manager.py

def finalize(self) -> Evaluator:
    """
    Performs data finalization and some preprocessing steps.

    Returns
    -------
    Evaluator
        A ready-to-use evaluator object.
    """
    if not self.pairs:
        raise EmptyEvaluatorError()

    n_labels = len(self._evaluator.index_to_label)
    n_datums = len(self._evaluator.index_to_datum_id)

    self._evaluator._detailed_pairs = np.concatenate(self.pairs, axis=0)
    if self._evaluator._detailed_pairs.size == 0:
        raise EmptyEvaluatorError()

    # order pairs by descending score, iou
    indices = np.lexsort(
        (
            -self._evaluator._detailed_pairs[:, 5],  # iou
            -self._evaluator._detailed_pairs[:, 6],  # score
        )
    )
    self._evaluator._detailed_pairs = self._evaluator._detailed_pairs[
        indices
    ]
    self._evaluator._label_metadata = compute_label_metadata(
        ids=self._evaluator._detailed_pairs[:, :5].astype(np.int32),
        n_labels=n_labels,
    )
    self._evaluator._ranked_pairs = rank_pairs(
        detailed_pairs=self._evaluator._detailed_pairs,
        label_metadata=self._evaluator._label_metadata,
    )
    self._evaluator._metadata = Metadata.create(
        detailed_pairs=self._evaluator._detailed_pairs,
        number_of_datums=n_datums,
        number_of_labels=n_labels,
    )
    return self._evaluator

valor_lite.object_detection.Evaluator

Object Detection Evaluator

Source code in valor_lite/object_detection/manager.py

class Evaluator:
    """
    Object Detection Evaluator
    """

    def __init__(self):

        # external reference
        self.datum_id_to_index: dict[str, int] = {}
        self.groundtruth_id_to_index: dict[str, int] = {}
        self.prediction_id_to_index: dict[str, int] = {}
        self.label_to_index: dict[str, int] = {}

        self.index_to_datum_id: list[str] = []
        self.index_to_groundtruth_id: list[str] = []
        self.index_to_prediction_id: list[str] = []
        self.index_to_label: list[str] = []

        # temporary cache
        self._temp_cache: list[NDArray[np.float64]] | None = []

        # internal cache
        self._detailed_pairs = np.array([[]], dtype=np.float64)
        self._ranked_pairs = np.array([[]], dtype=np.float64)
        self._label_metadata: NDArray[np.int32] = np.array([[]])
        self._metadata = Metadata()

    @property
    def ignored_prediction_labels(self) -> list[str]:
        """
        Prediction labels that are not present in the ground truth set.
        """
        glabels = set(np.where(self._label_metadata[:, 0] > 0)[0])
        plabels = set(np.where(self._label_metadata[:, 1] > 0)[0])
        return [
            self.index_to_label[label_id] for label_id in (plabels - glabels)
        ]

    @property
    def missing_prediction_labels(self) -> list[str]:
        """
        Ground truth labels that are not present in the prediction set.
        """
        glabels = set(np.where(self._label_metadata[:, 0] > 0)[0])
        plabels = set(np.where(self._label_metadata[:, 1] > 0)[0])
        return [
            self.index_to_label[label_id] for label_id in (glabels - plabels)
        ]

    @property
    def metadata(self) -> Metadata:
        """
        Evaluation metadata.
        """
        return self._metadata

    def create_filter(
        self,
        datums: list[str] | NDArray[np.int32] | None = None,
        groundtruths: list[str] | NDArray[np.int32] | None = None,
        predictions: list[str] | NDArray[np.int32] | None = None,
        labels: list[str] | NDArray[np.int32] | None = None,
    ) -> Filter:
        """
        Creates a filter object.

        Parameters
        ----------
        datum : list[str] | NDArray[int32], optional
            An optional list of string ids or indices representing datums to keep.
        groundtruth : list[str] | NDArray[int32], optional
            An optional list of string ids or indices representing ground truth annotations to keep.
        prediction : list[str] | NDArray[int32], optional
            An optional list of string ids or indices representing prediction annotations to keep.
        labels : list[str] | NDArray[int32], optional
            An optional list of labels or indices to keep.
        """
        mask_datums = np.ones(self._detailed_pairs.shape[0], dtype=np.bool_)

        # filter datums
        if datums is not None:
            # convert to indices
            if isinstance(datums, list):
                datums = np.array(
                    [self.datum_id_to_index[uid] for uid in datums],
                    dtype=np.int32,
                )

            # validate indices
            if datums.size == 0:
                raise EmptyFilterError(
                    "filter removes all datums"
                )  # validate indices
            elif datums.min() < 0:
                raise ValueError(
                    f"datum index cannot be negative '{datums.min()}'"
                )
            elif datums.max() >= len(self.index_to_datum_id):
                raise ValueError(
                    f"datum index cannot exceed total number of datums '{datums.max()}'"
                )

            # apply to mask
            mask_datums = np.isin(self._detailed_pairs[:, 0], datums)

        filtered_detailed_pairs = self._detailed_pairs[mask_datums]
        n_pairs = self._detailed_pairs[mask_datums].shape[0]
        mask_groundtruths = np.zeros(n_pairs, dtype=np.bool_)
        mask_predictions = np.zeros_like(mask_groundtruths)

        # filter by ground truth annotation ids
        if groundtruths is not None:
            # convert to indices
            if isinstance(groundtruths, list):
                groundtruths = np.array(
                    [
                        self.groundtruth_id_to_index[uid]
                        for uid in groundtruths
                    ],
                    dtype=np.int32,
                )

            # validate indices
            if groundtruths.size == 0:
                warnings.warn("filter removes all ground truths")
            elif groundtruths.min() < 0:
                raise ValueError(
                    f"groundtruth annotation index cannot be negative '{groundtruths.min()}'"
                )
            elif groundtruths.max() >= len(self.index_to_groundtruth_id):
                raise ValueError(
                    f"groundtruth annotation index cannot exceed total number of groundtruths '{groundtruths.max()}'"
                )

            # apply to mask
            mask_groundtruths[
                ~np.isin(
                    filtered_detailed_pairs[:, 1],
                    groundtruths,
                )
            ] = True

        # filter by prediction annotation ids
        if predictions is not None:
            # convert to indices
            if isinstance(predictions, list):
                predictions = np.array(
                    [self.prediction_id_to_index[uid] for uid in predictions],
                    dtype=np.int32,
                )

            # validate indices
            if predictions.size == 0:
                warnings.warn("filter removes all predictions")
            elif predictions.min() < 0:
                raise ValueError(
                    f"prediction annotation index cannot be negative '{predictions.min()}'"
                )
            elif predictions.max() >= len(self.index_to_prediction_id):
                raise ValueError(
                    f"prediction annotation index cannot exceed total number of predictions '{predictions.max()}'"
                )

            # apply to mask
            mask_predictions[
                ~np.isin(
                    filtered_detailed_pairs[:, 2],
                    predictions,
                )
            ] = True

        # filter by labels
        if labels is not None:
            # convert to indices
            if isinstance(labels, list):
                labels = np.array(
                    [self.label_to_index[label] for label in labels]
                )

            # validate indices
            if labels.size == 0:
                raise EmptyFilterError("filter removes all labels")
            elif labels.min() < 0:
                raise ValueError(
                    f"label index cannot be negative '{labels.min()}'"
                )
            elif labels.max() >= len(self.index_to_label):
                raise ValueError(
                    f"label index cannot exceed total number of labels '{labels.max()}'"
                )

            # apply to mask
            labels = np.concatenate([labels, np.array([-1])])  # add null label
            mask_groundtruths[
                ~np.isin(filtered_detailed_pairs[:, 3], labels)
            ] = True
            mask_predictions[
                ~np.isin(filtered_detailed_pairs[:, 4], labels)
            ] = True

        filtered_detailed_pairs, _, _ = filter_cache(
            self._detailed_pairs,
            mask_datums=mask_datums,
            mask_ground_truths=mask_groundtruths,
            mask_predictions=mask_predictions,
            n_labels=len(self.index_to_label),
        )

        number_of_datums = (
            datums.size
            if datums is not None
            else np.unique(filtered_detailed_pairs[:, 0]).size
        )

        return Filter(
            mask_datums=mask_datums,
            mask_groundtruths=mask_groundtruths,
            mask_predictions=mask_predictions,
            metadata=Metadata.create(
                detailed_pairs=filtered_detailed_pairs,
                number_of_datums=number_of_datums,
                number_of_labels=len(self.index_to_label),
            ),
        )

    def filter(
        self, filter_: Filter
    ) -> tuple[NDArray[np.float64], NDArray[np.float64], NDArray[np.int32],]:
        """
        Performs filtering over the internal cache.

        Parameters
        ----------
        filter_ : Filter
            The filter parameterization.

        Returns
        -------
        NDArray[float64]
            Filtered detailed pairs.
        NDArray[float64]
            Filtered ranked pairs.
        NDArray[int32]
            Label metadata.
        """
        return filter_cache(
            detailed_pairs=self._detailed_pairs,
            mask_datums=filter_.mask_datums,
            mask_ground_truths=filter_.mask_groundtruths,
            mask_predictions=filter_.mask_predictions,
            n_labels=len(self.index_to_label),
        )

    def compute_precision_recall(
        self,
        iou_thresholds: list[float],
        score_thresholds: list[float],
        filter_: Filter | None = None,
    ) -> dict[MetricType, list[Metric]]:
        """
        Computes all metrics except for ConfusionMatrix

        Parameters
        ----------
        iou_thresholds : list[float]
            A list of IOU thresholds to compute metrics over.
        score_thresholds : list[float]
            A list of score thresholds to compute metrics over.
        filter_ : Filter, optional
            A collection of filter parameters and masks.

        Returns
        -------
        dict[MetricType, list]
            A dictionary mapping MetricType enumerations to lists of computed metrics.
        """
        if not iou_thresholds:
            raise ValueError("At least one IOU threshold must be passed.")
        elif not score_thresholds:
            raise ValueError("At least one score threshold must be passed.")

        if filter_ is not None:
            _, ranked_pairs, label_metadata = self.filter(filter_=filter_)
        else:
            ranked_pairs = self._ranked_pairs
            label_metadata = self._label_metadata

        results = compute_precion_recall(
            ranked_pairs=ranked_pairs,
            label_metadata=label_metadata,
            iou_thresholds=np.array(iou_thresholds),
            score_thresholds=np.array(score_thresholds),
        )
        return unpack_precision_recall_into_metric_lists(
            results=results,
            label_metadata=label_metadata,
            iou_thresholds=iou_thresholds,
            score_thresholds=score_thresholds,
            index_to_label=self.index_to_label,
        )

    def compute_confusion_matrix(
        self,
        iou_thresholds: list[float],
        score_thresholds: list[float],
        filter_: Filter | None = None,
    ) -> list[Metric]:
        """
        Computes confusion matrices at various thresholds.

        Parameters
        ----------
        iou_thresholds : list[float]
            A list of IOU thresholds to compute metrics over.
        score_thresholds : list[float]
            A list of score thresholds to compute metrics over.
        filter_ : Filter, optional
            A collection of filter parameters and masks.

        Returns
        -------
        list[Metric]
            List of confusion matrices per threshold pair.
        """
        if not iou_thresholds:
            raise ValueError("At least one IOU threshold must be passed.")
        elif not score_thresholds:
            raise ValueError("At least one score threshold must be passed.")

        if filter_ is not None:
            detailed_pairs, _, _ = self.filter(filter_=filter_)
        else:
            detailed_pairs = self._detailed_pairs

        if detailed_pairs.size == 0:
            return []

        results = compute_confusion_matrix(
            detailed_pairs=detailed_pairs,
            iou_thresholds=np.array(iou_thresholds),
            score_thresholds=np.array(score_thresholds),
        )
        return unpack_confusion_matrix_into_metric_list(
            results=results,
            detailed_pairs=detailed_pairs,
            iou_thresholds=iou_thresholds,
            score_thresholds=score_thresholds,
            index_to_datum_id=self.index_to_datum_id,
            index_to_groundtruth_id=self.index_to_groundtruth_id,
            index_to_prediction_id=self.index_to_prediction_id,
            index_to_label=self.index_to_label,
        )

    def evaluate(
        self,
        iou_thresholds: list[float] = [0.1, 0.5, 0.75],
        score_thresholds: list[float] = [0.5],
        filter_: Filter | None = None,
    ) -> dict[MetricType, list[Metric]]:
        """
        Computes all available metrics.

        Parameters
        ----------
        iou_thresholds : list[float], default=[0.1, 0.5, 0.75]
            A list of IOU thresholds to compute metrics over.
        score_thresholds : list[float], default=[0.5]
            A list of score thresholds to compute metrics over.
        filter_ : Filter, optional
            A collection of filter parameters and masks.

        Returns
        -------
        dict[MetricType, list[Metric]]
            Lists of metrics organized by metric type.
        """
        metrics = self.compute_precision_recall(
            iou_thresholds=iou_thresholds,
            score_thresholds=score_thresholds,
            filter_=filter_,
        )
        metrics[MetricType.ConfusionMatrix] = self.compute_confusion_matrix(
            iou_thresholds=iou_thresholds,
            score_thresholds=score_thresholds,
            filter_=filter_,
        )
        return metrics

ignored_prediction_labels property

Prediction labels that are not present in the ground truth set.

metadata property

Evaluation metadata.

missing_prediction_labels property

Ground truth labels that are not present in the prediction set.

compute_confusion_matrix(iou_thresholds, score_thresholds, filter_=None)

Computes confusion matrices at various thresholds.

Parameters:

Name	Type	Description	Default
iou_thresholds	list[float]	A list of IOU thresholds to compute metrics over.	required
score_thresholds	list[float]	A list of score thresholds to compute metrics over.	required
filter_	Filter	A collection of filter parameters and masks.	None

Returns:

Type	Description
list[Metric]	List of confusion matrices per threshold pair.

Source code in valor_lite/object_detection/manager.py

def compute_confusion_matrix(
    self,
    iou_thresholds: list[float],
    score_thresholds: list[float],
    filter_: Filter | None = None,
) -> list[Metric]:
    """
    Computes confusion matrices at various thresholds.

    Parameters
    ----------
    iou_thresholds : list[float]
        A list of IOU thresholds to compute metrics over.
    score_thresholds : list[float]
        A list of score thresholds to compute metrics over.
    filter_ : Filter, optional
        A collection of filter parameters and masks.

    Returns
    -------
    list[Metric]
        List of confusion matrices per threshold pair.
    """
    if not iou_thresholds:
        raise ValueError("At least one IOU threshold must be passed.")
    elif not score_thresholds:
        raise ValueError("At least one score threshold must be passed.")

    if filter_ is not None:
        detailed_pairs, _, _ = self.filter(filter_=filter_)
    else:
        detailed_pairs = self._detailed_pairs

    if detailed_pairs.size == 0:
        return []

    results = compute_confusion_matrix(
        detailed_pairs=detailed_pairs,
        iou_thresholds=np.array(iou_thresholds),
        score_thresholds=np.array(score_thresholds),
    )
    return unpack_confusion_matrix_into_metric_list(
        results=results,
        detailed_pairs=detailed_pairs,
        iou_thresholds=iou_thresholds,
        score_thresholds=score_thresholds,
        index_to_datum_id=self.index_to_datum_id,
        index_to_groundtruth_id=self.index_to_groundtruth_id,
        index_to_prediction_id=self.index_to_prediction_id,
        index_to_label=self.index_to_label,
    )

compute_precision_recall(iou_thresholds, score_thresholds, filter_=None)

Computes all metrics except for ConfusionMatrix

Parameters:

Name	Type	Description	Default
iou_thresholds	list[float]	A list of IOU thresholds to compute metrics over.	required
score_thresholds	list[float]	A list of score thresholds to compute metrics over.	required
filter_	Filter	A collection of filter parameters and masks.	None

Returns:

Type	Description
dict[MetricType, list]	A dictionary mapping MetricType enumerations to lists of computed metrics.

Source code in valor_lite/object_detection/manager.py

def compute_precision_recall(
    self,
    iou_thresholds: list[float],
    score_thresholds: list[float],
    filter_: Filter | None = None,
) -> dict[MetricType, list[Metric]]:
    """
    Computes all metrics except for ConfusionMatrix

    Parameters
    ----------
    iou_thresholds : list[float]
        A list of IOU thresholds to compute metrics over.
    score_thresholds : list[float]
        A list of score thresholds to compute metrics over.
    filter_ : Filter, optional
        A collection of filter parameters and masks.

    Returns
    -------
    dict[MetricType, list]
        A dictionary mapping MetricType enumerations to lists of computed metrics.
    """
    if not iou_thresholds:
        raise ValueError("At least one IOU threshold must be passed.")
    elif not score_thresholds:
        raise ValueError("At least one score threshold must be passed.")

    if filter_ is not None:
        _, ranked_pairs, label_metadata = self.filter(filter_=filter_)
    else:
        ranked_pairs = self._ranked_pairs
        label_metadata = self._label_metadata

    results = compute_precion_recall(
        ranked_pairs=ranked_pairs,
        label_metadata=label_metadata,
        iou_thresholds=np.array(iou_thresholds),
        score_thresholds=np.array(score_thresholds),
    )
    return unpack_precision_recall_into_metric_lists(
        results=results,
        label_metadata=label_metadata,
        iou_thresholds=iou_thresholds,
        score_thresholds=score_thresholds,
        index_to_label=self.index_to_label,
    )

create_filter(datums=None, groundtruths=None, predictions=None, labels=None)

Creates a filter object.

Parameters:

Name	Type	Description	Default
datum	list[str] | NDArray[int32]	An optional list of string ids or indices representing datums to keep.	required
groundtruth	list[str] | NDArray[int32]	An optional list of string ids or indices representing ground truth annotations to keep.	required
prediction	list[str] | NDArray[int32]	An optional list of string ids or indices representing prediction annotations to keep.	required
labels	list[str] | NDArray[int32]	An optional list of labels or indices to keep.	None

Source code in valor_lite/object_detection/manager.py

def create_filter(
    self,
    datums: list[str] | NDArray[np.int32] | None = None,
    groundtruths: list[str] | NDArray[np.int32] | None = None,
    predictions: list[str] | NDArray[np.int32] | None = None,
    labels: list[str] | NDArray[np.int32] | None = None,
) -> Filter:
    """
    Creates a filter object.

    Parameters
    ----------
    datum : list[str] | NDArray[int32], optional
        An optional list of string ids or indices representing datums to keep.
    groundtruth : list[str] | NDArray[int32], optional
        An optional list of string ids or indices representing ground truth annotations to keep.
    prediction : list[str] | NDArray[int32], optional
        An optional list of string ids or indices representing prediction annotations to keep.
    labels : list[str] | NDArray[int32], optional
        An optional list of labels or indices to keep.
    """
    mask_datums = np.ones(self._detailed_pairs.shape[0], dtype=np.bool_)

    # filter datums
    if datums is not None:
        # convert to indices
        if isinstance(datums, list):
            datums = np.array(
                [self.datum_id_to_index[uid] for uid in datums],
                dtype=np.int32,
            )

        # validate indices
        if datums.size == 0:
            raise EmptyFilterError(
                "filter removes all datums"
            )  # validate indices
        elif datums.min() < 0:
            raise ValueError(
                f"datum index cannot be negative '{datums.min()}'"
            )
        elif datums.max() >= len(self.index_to_datum_id):
            raise ValueError(
                f"datum index cannot exceed total number of datums '{datums.max()}'"
            )

        # apply to mask
        mask_datums = np.isin(self._detailed_pairs[:, 0], datums)

    filtered_detailed_pairs = self._detailed_pairs[mask_datums]
    n_pairs = self._detailed_pairs[mask_datums].shape[0]
    mask_groundtruths = np.zeros(n_pairs, dtype=np.bool_)
    mask_predictions = np.zeros_like(mask_groundtruths)

    # filter by ground truth annotation ids
    if groundtruths is not None:
        # convert to indices
        if isinstance(groundtruths, list):
            groundtruths = np.array(
                [
                    self.groundtruth_id_to_index[uid]
                    for uid in groundtruths
                ],
                dtype=np.int32,
            )

        # validate indices
        if groundtruths.size == 0:
            warnings.warn("filter removes all ground truths")
        elif groundtruths.min() < 0:
            raise ValueError(
                f"groundtruth annotation index cannot be negative '{groundtruths.min()}'"
            )
        elif groundtruths.max() >= len(self.index_to_groundtruth_id):
            raise ValueError(
                f"groundtruth annotation index cannot exceed total number of groundtruths '{groundtruths.max()}'"
            )

        # apply to mask
        mask_groundtruths[
            ~np.isin(
                filtered_detailed_pairs[:, 1],
                groundtruths,
            )
        ] = True

    # filter by prediction annotation ids
    if predictions is not None:
        # convert to indices
        if isinstance(predictions, list):
            predictions = np.array(
                [self.prediction_id_to_index[uid] for uid in predictions],
                dtype=np.int32,
            )

        # validate indices
        if predictions.size == 0:
            warnings.warn("filter removes all predictions")
        elif predictions.min() < 0:
            raise ValueError(
                f"prediction annotation index cannot be negative '{predictions.min()}'"
            )
        elif predictions.max() >= len(self.index_to_prediction_id):
            raise ValueError(
                f"prediction annotation index cannot exceed total number of predictions '{predictions.max()}'"
            )

        # apply to mask
        mask_predictions[
            ~np.isin(
                filtered_detailed_pairs[:, 2],
                predictions,
            )
        ] = True

    # filter by labels
    if labels is not None:
        # convert to indices
        if isinstance(labels, list):
            labels = np.array(
                [self.label_to_index[label] for label in labels]
            )

        # validate indices
        if labels.size == 0:
            raise EmptyFilterError("filter removes all labels")
        elif labels.min() < 0:
            raise ValueError(
                f"label index cannot be negative '{labels.min()}'"
            )
        elif labels.max() >= len(self.index_to_label):
            raise ValueError(
                f"label index cannot exceed total number of labels '{labels.max()}'"
            )

        # apply to mask
        labels = np.concatenate([labels, np.array([-1])])  # add null label
        mask_groundtruths[
            ~np.isin(filtered_detailed_pairs[:, 3], labels)
        ] = True
        mask_predictions[
            ~np.isin(filtered_detailed_pairs[:, 4], labels)
        ] = True

    filtered_detailed_pairs, _, _ = filter_cache(
        self._detailed_pairs,
        mask_datums=mask_datums,
        mask_ground_truths=mask_groundtruths,
        mask_predictions=mask_predictions,
        n_labels=len(self.index_to_label),
    )

    number_of_datums = (
        datums.size
        if datums is not None
        else np.unique(filtered_detailed_pairs[:, 0]).size
    )

    return Filter(
        mask_datums=mask_datums,
        mask_groundtruths=mask_groundtruths,
        mask_predictions=mask_predictions,
        metadata=Metadata.create(
            detailed_pairs=filtered_detailed_pairs,
            number_of_datums=number_of_datums,
            number_of_labels=len(self.index_to_label),
        ),
    )

evaluate(iou_thresholds=[0.1, 0.5, 0.75], score_thresholds=[0.5], filter_=None)

Computes all available metrics.

Parameters:

Name	Type	Description	Default
iou_thresholds	list[float]	A list of IOU thresholds to compute metrics over.	[0.1, 0.5, 0.75]
score_thresholds	list[float]	A list of score thresholds to compute metrics over.	[0.5]
filter_	Filter	A collection of filter parameters and masks.	None

Returns:

Type	Description
dict[MetricType, list[Metric]]	Lists of metrics organized by metric type.

Source code in valor_lite/object_detection/manager.py

def evaluate(
    self,
    iou_thresholds: list[float] = [0.1, 0.5, 0.75],
    score_thresholds: list[float] = [0.5],
    filter_: Filter | None = None,
) -> dict[MetricType, list[Metric]]:
    """
    Computes all available metrics.

    Parameters
    ----------
    iou_thresholds : list[float], default=[0.1, 0.5, 0.75]
        A list of IOU thresholds to compute metrics over.
    score_thresholds : list[float], default=[0.5]
        A list of score thresholds to compute metrics over.
    filter_ : Filter, optional
        A collection of filter parameters and masks.

    Returns
    -------
    dict[MetricType, list[Metric]]
        Lists of metrics organized by metric type.
    """
    metrics = self.compute_precision_recall(
        iou_thresholds=iou_thresholds,
        score_thresholds=score_thresholds,
        filter_=filter_,
    )
    metrics[MetricType.ConfusionMatrix] = self.compute_confusion_matrix(
        iou_thresholds=iou_thresholds,
        score_thresholds=score_thresholds,
        filter_=filter_,
    )
    return metrics

filter(filter_)

Performs filtering over the internal cache.

Parameters:

Name	Type	Description	Default
filter_	Filter	The filter parameterization.	required

Returns:

Type	Description
NDArray[float64]	Filtered detailed pairs.
NDArray[float64]	Filtered ranked pairs.
NDArray[int32]	Label metadata.

Source code in valor_lite/object_detection/manager.py

def filter(
    self, filter_: Filter
) -> tuple[NDArray[np.float64], NDArray[np.float64], NDArray[np.int32],]:
    """
    Performs filtering over the internal cache.

    Parameters
    ----------
    filter_ : Filter
        The filter parameterization.

    Returns
    -------
    NDArray[float64]
        Filtered detailed pairs.
    NDArray[float64]
        Filtered ranked pairs.
    NDArray[int32]
        Label metadata.
    """
    return filter_cache(
        detailed_pairs=self._detailed_pairs,
        mask_datums=filter_.mask_datums,
        mask_ground_truths=filter_.mask_groundtruths,
        mask_predictions=filter_.mask_predictions,
        n_labels=len(self.index_to_label),
    )