Add application file

README.md

@@ -1,50 +1,153 @@
 ---
-title: isco_hierachical_accuracy_v2
+title: ISCO-08 Hierarchical Accuracy Measure v2
 datasets:
--  
+-  ICILS/multilingual_parental_occupations
 tags:
 - evaluate
 - metric
-description: "TODO: add a description here"
+description: "The ISCO-08 Hierarchical Accuracy Measure is an implementation of the measure described in [Functional Annotation of Genes Using Hierarchical Text Categorization](https://www.researchgate.net/publication/44046343_Functional_Annotation_of_Genes_Using_Hierarchical_Text_Categorization) (Kiritchenko, Svetlana and Famili, Fazel. 2005) applied to the ISCO-08 classification scheme by the International Labour Organization."
 sdk: gradio
 sdk_version: 3.19.1
 app_file: app.py
 pinned: false
 ---
 
-# Metric Card for isco_hierachical_accuracy_v2
-
-***Module Card Instructions:*** *Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples.*
+# Metric Card for ISCO-08 Hierarchical Accuracy Measure
 
 ## Metric Description
-*Give a brief overview of this metric, including what task(s) it is usually used for, if any.*
+
+ISCO‑08 is a four‑level taxonomy (Table 1). Correctly locating an occupation at the appropriate level—Major, Sub‑major, Minor or Unit—is essential, yet many practical systems make small but still useful errors (e.g. confusing two Unit groups within the same Minor group). The present metric extends classical precision, recall and *F*‑measure to this hierarchical setting, thereby providing a more nuanced assessment than flat accuracy.
+
+|Digits|Group level  |
+|--|--|
+| 1-digit | Major groups |
+| 2-digits | Sub-major groups |
+| 3-digits | Minor groups |
+| 4-digits | Unit groups |
+
+![Figure 1 – An excerpt of the ISCO‑08 hierarchy showing the ancestors of node 2211](https://huggingface.co/spaces/danieldux/isco_hierachical_accuracy_v2/resolve/main/figure_1.png)
+
+In this context, the hierarchical accuracy measure is specifically designed to evaluate classifications within this structured framework. It emphasizes the importance of precision in classifying occupations at the correct level of specificity:
+
+1. **Major Groups (1-digit codes):** The broadest category that encompasses a wide range of occupations grouped based on their fundamental characteristic and role in the job market. Misclassification among different Major groups implies a fundamental misunderstanding of the occupational role.
+
+2. **Sub-major Groups (2-digit codes):** Provide a more detailed classification within each Major group, delineating categories that share specific occupational characteristics. Errors within a Major group but across different Sub-major groups are less severe than those across Major groups but are still significant due to the deviation from the correct occupational category.
+
+3. **Minor Groups (3-digit codes):** Offer further granularity within Sub-major groups, classifying occupations that require similar skill sets and qualifications. Misclassifications within a Sub-major group but across different Minor groups are penalized, yet to a lesser extent than errors across Sub-major or Major groups, as they still fall within a broader correct occupational category.
+
+4. **Unit Groups (4-digit codes):** The most specific level, identifying precise occupations. Misclassification within a Minor group, between different Unit groups, represents the least severe error, as it occurs within the context of closely related occupations.
+
+The hierarchical accuracy measure rewards more precise classifications that correctly identify an occupation's placement down to the specific Unit group level and applies penalties for misclassifications based on the hierarchical distance between the correct and assigned categories. This approach allows for a refined evaluation of classification systems, acknowledging partially correct classifications while penalizing inaccuracies more severely as they deviate further from the correct level in the hierarchical structure.
+
+For example, a misclassification into Unit group 2211 (Generalist Medical Practitioners) when the correct category is Unit group 2212 (Specialist Medical Practitioners) should incur a lesser penalty than misclassification into Unit group 2352 (Special Needs Teachers), as Unit groups 2211 and 2212 are within the same Minor group 221 (Medical doctors) and share a closer hierarchical relationship compared to Minor group 235 (Other teaching professionals) which is in a different Sub-major group (23 Teaching professionals).
+
+The measure applies a higher penalty for errors that occur between more distant categories within the hierarchy:
+
+1. Correct classifications at a more specific level (e.g., Minor group 2211) are evaluated more favorably than classifications at a more general level within the same hierarchy (e.g., Sub-major group 22), since the former indicates a closer match to the correct category.
+2. Conversely, incorrect classifications at a more specific level are penalized more heavily than those at a more general level. For instance, classifying into Minor group 222 (Nursing and Midwifery Professionals) is considered worse than classifying into its parent category, Sub-major group 22, if the correct classification is Minor group 221, because the incorrect specific classification indicates a farther deviation from the correct category.
+
+Misclassification among sibling categories (e.g., between different Minor groups within the same Sub-major group) is less severe than misclassification at a higher hierarchical level (e.g., between different Major groups).
+
+The measure extends the concepts of precision and recall into a hierarchical context, introducing hierarchical precision (ℎ𝑃) and hierarchical recall (ℎ𝑅). In this framework, each sample belongs not only to its designated class but also to all ancestor categories in the hierarchy, excluding the root (we exclude the root of the graph, since all samples belong to the root by default). This adjustment allows the measure to account for the hierarchical structure of the classification scheme, rewarding more accurate location of a sample within the hierarchy and penalizing errors based on their hierarchical significance.
+
+To calculate the hierarchical measure, extend the set of real classes
+
+$$C_i = \{G\}$$
+with all ancestors of 𝐺:
+$$\vec{C}_i = \{B, C, E, G\}$$
+
+Similarly, extend the set of predicted classes
+
+$$C^′_i = \{F\}$$
+with all ancestors of 𝐹:
+$$\vec{C}^′_i = \{C, F\}$$
+
+Class 𝐶 is the only correctly assigned label from the extended sets:
+
+$$| \vec{C}_i ∩ \vec{C}^′_i| = 1$$
+
+There are
+
+$$| \vec{C}^′_i| = 2$$
+predicted labels and
+$$| \vec{C}_i| = 4$$
+
+real classes.
+
+Therefore,
+
+$$hP = \frac{| \vec{C}_i ∩ \vec{C}^′_i|}  {|\vec{C}^′_i |} = \frac{1}{2}$$
+$$hR = \frac{| \vec{C}_i ∩ \vec{C}^′_i|}  {|\vec{C}_i |} = \frac{1}{2}$$
+
+Finally, combine ℎ𝑃 and ℎ𝑅 into the hierarchical 𝐹-measure:
+
+$$hF_β = \frac{(β^2 + 1) · hP · hR}{(β^2 · hP + hR)}, β ∈ [0, +∞)$$
+The metric **rewards depth**: predicting *221* instead of *22* yields higher *hF* because more ancestors overlap. It **penalises distance**: predicting a Unit in a different Sub‑major group incurs a larger loss than confusing two Units under the same Minor group.
 
 ## How to Use
-*Give general statement of how to use the metric*
 
-*Provide simplest possible example for using the metric*
+- **Model evaluation** Assess neural or rule‑based classifiers that map free‑text occupation descriptions to ISCO‑08 codes.
+- **Inter‑rater agreement** Quantify consistency between human coders when full agreement at the Unit level is not always expected.
 
 ### Inputs
-*List all input arguments in the format below*
-- **input_field** *(type): Definition of input, with explanation if necessary. State any default value(s).*
+
+- **references** *(List[str])*: List of reference ISCO-08 codes (true labels or coder 1 codes).
+- **predictions** *(List[str])*: List of predicted ISCO-08 codes (predicted labels) or coder 2 codes. This is the predicted classification or classification to compare against the reference codes.
 
 ### Output Values
 
-*Explain what this metric outputs and provide an example of what the metric output looks like. Modules should return a dictionary with one or multiple key-value pairs, e.g. {"bleu" : 6.02}*
+Values are decimal numbers between 0 and 1. Higher scores are better.
 
-*State the range of possible values that the metric's output can take, as well as what in that range is considered good. For example: "This metric can take on any value between 0 and 100, inclusive. Higher scores are better."*
+**Example output**:
+
+```python
+{
+    "accuracy": 0.25,
+    "hierarchical_precision": 1.0,
+    "hierarchical_recall": 0.4,
+    "hierarchical_fmeasure": 0.5714285714285715,
+}
+```
 
 #### Values from Popular Papers
-*Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.*
+
+The following table is from the paper [Improving Parental Occupation Coding Procedures with AI](https://www.iea.nl/sites/default/files/2024-09/Improving-Parental-Occupation-Coding-Procedures-AI.pdf).
+
+**Summary of multilingual model overall accuracies and hF-measure scores**
+
+| Model name | Training dataset (training & validation splits) | Evaluation dataset (test split) | Accuracy | hFβ |
+|------------|--------------------------------------------------|---------------------------------|----------|-----|
+| Model 1    | ICILS                                            | ICILS                           | 63%      | 0.89|
+| Model 2    | ILO                                              | ILO                             | 92%      | 0.99|
+| Model 2    | ILO                                              | ICILS                           | 36%      | 0.94|
+| Model 3    | ICILS+ILO                                        | ICILS+ILO                       | 80%      | 0.93|
+| Model 3    | ICILS+ILO                                        | ICILS                           | 62%      | 0.95|
 
 ### Examples
-*Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.*
+
+```python
+def compute_metrics(p: EvalPrediction):
+    preds = p.predictions[0] if isinstance(p.predictions, tuple) else p.predictions
+    preds = np.argmax(preds, axis=1)
+    # Use labels instead of ids for hierarchical ISCO-08 classification
+    preds = [model.config.id2label[p] for p in preds]
+    refs = [model.config.id2label[r] for r in p.label_ids]
+    result = metric.compute(predictions=preds, references=refs)
+    return result
+```
 
 ## Limitations and Bias
-*Note any known limitations or biases that the metric has, with links and references if possible.*
+
+No known limitations or bias.
 
 ## Citation
-*Cite the source where this metric was introduced.*
+
+This metric was developed as part of an [IEA R&D project](https://www.iea.nl/publications/other/rd-outcomes), [Improving Parental Occupation Coding Procedures with AI](https://www.iea.nl/sites/default/files/2024-09/Improving-Parental-Occupation-Coding-Procedures-AI.pdf) (Duckworth et al, 2024). The metric was used to evaluate the multilingual [ICILS XLM-R ISCO](https://huggingface.co/ICILS/xlm-r-icils-ilo) classification model.
+
+Note: This is version 2 of the ISCO Hierarchical Accuracy Measure.
 
 ## Further References
-*Add any useful further references.*
+
+- [The International Standard Classification of Occupations- ISCO-08](https://isco.ilo.org/en/isco-08/)
+- [Improving Parental Occupation Coding Procedures with AI](https://www.iea.nl/sites/default/files/2024-09/Improving-Parental-Occupation-Coding-Procedures-AI.pdf) (Duckworth et al, 2024)
+- [ICILS XLM-R ISCO model](https://huggingface.co/ICILS/xlm-r-icils-ilo)

app.py

@@ -1,6 +1,71 @@
+from tests import test_cases
+from evaluate.utils.logging import get_logger
+from evaluate.utils import (
+    infer_gradio_input_types,
+    parse_gradio_data,
+    json_to_string_type,
+    parse_readme,
+    parse_test_cases,
+)
+from pathlib import Path
+
 import evaluate
-from evaluate.utils import launch_gradio_widget
+import sys
+
+logger = get_logger(__name__)
+
+
+def launch_gradio_widget(metric, test_cases):
+    """Launches `metric` widget with Gradio."""
+
+    try:
+        import gradio as gr
+    except ImportError as error:
+        logger.error(
+            "To create a metric widget with Gradio make sure gradio is installed."
+        )
+        raise error
+
+    local_path = Path(sys.path[0])
+    # if there are several input types, use first as default.
+    if isinstance(metric.features, list):
+        (feature_names, feature_types) = zip(*metric.features[0].items())
+    else:
+        (feature_names, feature_types) = zip(*metric.features.items())
+    gradio_input_types = infer_gradio_input_types(feature_types)
+
+    parsed_test_cases = parse_test_cases(test_cases, feature_names, gradio_input_types)
+
+    def compute(data):
+        return metric.compute(**parse_gradio_data(data, gradio_input_types))
+
+    demo = gr.Interface(
+        fn=compute,
+        inputs=gr.Dataframe(
+            value=parsed_test_cases[0],
+            headers=feature_names,
+            col_count=len(feature_names),
+            datatype=json_to_string_type(gradio_input_types),
+            # row_count=5,
+        ),
+        outputs=gr.Textbox(label=metric.name),
+        description=(
+            metric.info.description
+            + "\nISCO codes must be wrapped in double quotes."
+            # " Alternatively you can use a JSON-formatted list as input."
+        ),
+        title=f"Metric: {metric.name}",
+        article=parse_readme(local_path / "README.md"),
+        # TODO: load test cases and use them to populate examples
+        examples=[parsed_test_cases],
+    )
+    if __name__ == "__main__":
+        demo.launch()
+    else:
+        return demo
+
 
+module = evaluate.load("danieldux/isco_hierarchical_accuracy_v2")
 
-module = evaluate.load("danieldux/isco_hierachical_accuracy_v2")
-launch_gradio_widget(module)
+if __name__ == "__main__":
+    launch_gradio_widget(module, test_cases)

ham.py

@@ -0,0 +1,170 @@
+"""
+Compute hierarchical precision, recall, and F_beta for a single instance.
+
+Returns: Tuple[float, float, float]: (hierarchical precision, hierarchical recall, F_beta score)
+"""
+
+# Copyright 2025 Daniel Duckworth
+# Licensed under the Apache License, Version 2.0
+
+from typing import Iterable, List, Tuple, Optional, Dict, Any
+
+
+def _normalize(code: Optional[str]) -> Optional[str]:
+    """
+    Normalize an ISCO-08 code to a digit string of length 1..4.
+    Returns None if the input is empty/invalid.
+    Preserves leading zeros if they were present in the original string.
+    """
+    if code is None:
+        return None
+    s = str(code).strip()
+
+    # If it's purely digits already, keep as-is to preserve leading zeros
+    if s.isdigit():
+        if 1 <= len(s) <= 4:
+            return s
+        return None
+
+    # Otherwise strip non-digits while preserving any leading 0s present
+    digits = "".join(ch for ch in s if ch.isdigit())
+    if 1 <= len(digits) <= 4:
+        return digits
+    return None
+
+
+def ancestors(code: Optional[str]) -> List[str]:
+    """
+    Ancestor-closure (excluding the artificial root): all non-empty prefixes.
+    For '2211' -> ['2','22','221','2211'].
+    """
+    norm = _normalize(code)
+    if norm is None:
+        return []
+    return [norm[:k] for k in range(1, len(norm) + 1)]
+
+
+def hp_hr_hfbeta(
+    true_code: Optional[str], pred_code: Optional[str], beta: float = 1.0
+) -> Tuple[float, float, float]:
+    """
+    Per-instance hierarchical precision, recall, and F_beta.
+    """
+    C = set(ancestors(true_code))
+    Cp = set(ancestors(pred_code))
+
+    if not C or not Cp:
+        return 0.0, 0.0, 0.0
+
+    m = len(C & Cp)
+    hp = m / len(Cp)
+    hr = m / len(C)
+
+    if hp == 0.0 and hr == 0.0:
+        return 0.0, 0.0, 0.0
+
+    b2 = beta * beta
+    hf = (1.0 + b2) * hp * hr / (b2 * hp + hr)
+    return hp, hr, hf
+
+
+def hierarchical_scores(
+    y_true: Iterable[Optional[str]],
+    y_pred: Iterable[Optional[str]],
+    beta: float = 1.0,
+    average: str = "both",  # "micro", "macro", or "both"
+    return_per_instance: bool = False,
+) -> Dict[str, Any]:
+    """
+    Compute micro/macro aggregated hierarchical P/R/F_beta.
+    """
+    y_true = list(y_true)
+    y_pred = list(y_pred)
+    if len(y_true) != len(y_pred):
+        raise ValueError("y_true and y_pred must have the same length")
+
+    inst_hp, inst_hr, inst_hf = [], [], []
+    per_instance = []
+
+    M = 0  # total intersection
+    P = 0  # total predicted path length
+    T = 0  # total true path length
+
+    for g, p in zip(y_true, y_pred):
+        C = set(ancestors(g))
+        Cp = set(ancestors(p))
+
+        if C and Cp:
+            m = len(C & Cp)
+            hp = m / len(Cp)
+            hr = m / len(C)
+            if hp == 0.0 and hr == 0.0:
+                hf = 0.0
+            else:
+                b2 = beta * beta
+                hf = (1.0 + b2) * hp * hr / (b2 * hp + hr)
+
+            inst_hp.append(hp)
+            inst_hr.append(hr)
+            inst_hf.append(hf)
+
+            M += m
+            P += len(Cp)
+            T += len(C)
+        else:
+            hp = hr = hf = 0.0
+            inst_hp.append(hp)
+            inst_hr.append(hr)
+            inst_hf.append(hf)
+
+        if return_per_instance:
+            per_instance.append(
+                {
+                    "hP": hp,
+                    "hR": hr,
+                    "hF_beta": hf,
+                }
+            )
+
+    out: Dict[str, Any] = {}
+
+    if average in ("macro", "both"):
+        macro_hp = sum(inst_hp) / len(inst_hp) if inst_hp else 0.0
+        macro_hr = sum(inst_hr) / len(inst_hr) if inst_hr else 0.0
+        macro_hf_mean = sum(inst_hf) / len(inst_hf) if inst_hf else 0.0
+        b2 = beta * beta
+        macro_hf_from_pr = (
+            (1.0 + b2) * macro_hp * macro_hr / (b2 * macro_hp + macro_hr)
+            if (macro_hp + macro_hr) > 0
+            else 0.0
+        )
+        out.update(
+            {
+                "macro_hP": macro_hp,
+                "macro_hR": macro_hr,
+                "macro_hF_beta_mean": macro_hf_mean,
+                "macro_hF_beta_from_macroPR": macro_hf_from_pr,
+            }
+        )
+
+    if average in ("micro", "both"):
+        micro_hp = (M / P) if P > 0 else 0.0
+        micro_hr = (M / T) if T > 0 else 0.0
+        b2 = beta * beta
+        micro_hf = (
+            (1.0 + b2) * micro_hp * micro_hr / (b2 * micro_hp + micro_hr)
+            if (micro_hp + micro_hr) > 0
+            else 0.0
+        )
+        out.update(
+            {
+                "micro_hP": micro_hp,
+                "micro_hR": micro_hr,
+                "micro_hF_beta": micro_hf,
+            }
+        )
+
+    if return_per_instance:
+        out["per_instance"] = per_instance
+
+    return out

isco_hierachical_accuracy_v2.py

@@ -1,95 +1,125 @@
-# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
+# Copyright 2020 The HuggingFace Datasets Authors
+# and 2025 Daniel Duckworth (metric integration)
 #
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed under the Apache License, Version 2.0
 #
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""TODO: Add a description here."""
+# Metric: ISCO-08 Hierarchical Precision/Recall/Fβ with micro/macro aggregation.
+# The metric treats each code as belonging to all ancestor prefixes (excluding the root).
 
 import evaluate
 import datasets
 
+# Import the implementation from a sibling module file.
+# If packaging as an evaluate module, place both files in the same module directory.
+from ham import hierarchical_scores
+
 
-# TODO: Add BibTeX citation
 _CITATION = """\
-@InProceedings{huggingface:module,
-title = {A great new module},
-authors={huggingface, Inc.},
-year={2020}
+@inproceedings{kosmopoulos2015evaluation,
+  title={Evaluation Measures for Hierarchical Classification: A Unified View and Novel Proposals},
+  author={Kosmopoulos, Aris and Partalas, Ioannis and Gaussier, Eric and Paliouras, George and Androutsopoulos, Ion},
+  booktitle={Neurocomputing},
+  year={2015}
+}
+@misc{isco08,
+  title={International Standard Classification of Occupations (ISCO-08)},
+  howpublished={International Labour Organization},
+  year={2008}
 }
 """
 
-# TODO: Add description of the module here
 _DESCRIPTION = """\
-This new module is designed to solve this great ML task and is crafted with a lot of care.
+Hierarchical precision (hP), recall (hR), and Fβ (hFβ) for ISCO-08 codes.
+Each code is expanded to its ancestor-closure (all non-empty prefixes), and
+overlap between predicted and reference closures determines hP/hR. This rewards
+correct depth and penalizes distance in the hierarchy.
 """
 
-
-# TODO: Add description of the arguments of the module here
 _KWARGS_DESCRIPTION = """
-Calculates how good are predictions given some references, using certain scores
 Args:
-    predictions: list of predictions to score. Each predictions
-        should be a string with tokens separated by spaces.
-    references: list of reference for each prediction. Each
-        reference should be a string with tokens separated by spaces.
-Returns:
-    accuracy: description of the first score,
-    another_score: description of the second score,
-Examples:
-    Examples should be written in doctest format, and should illustrate how
-    to use the function.
+    predictions (List[str] | List[int]): Predicted ISCO-08 codes (length 1..4).
+        Strings are recommended to preserve leading zeros.
+    references (List[str] | List[int]): Reference ISCO-08 codes (length 1..4).
+    beta (float, optional): F-measure beta parameter. Default 1.0.
+    average (str, optional): "micro", "macro", or "both". Default "both".
+    return_per_instance (bool, optional): If True, returns a list of per-instance
+        dicts with hP/hR/hFβ. Default False.
+
+Returns (dict):
+    If average includes "macro":
+        - macro_hP
+        - macro_hR
+        - macro_hF_beta_mean              # mean of per-instance hFβ
+        - macro_hF_beta_from_macroPR      # Fβ computed from macro hP/hR
+    If average includes "micro":
+        - micro_hP
+        - micro_hR
+        - micro_hF_beta
+    If return_per_instance:
+        - per_instance: List[{"hP": float, "hR": float, "hF_beta": float}, ...]
 
-    >>> my_new_module = evaluate.load("my_new_module")
-    >>> results = my_new_module.compute(references=[0, 1], predictions=[0, 1])
-    >>> print(results)
-    {'accuracy': 1.0}
+Examples:
+    >>> import evaluate
+    >>> metric = evaluate.load("path/to/isco_hierachical_accuracy_v2.py")
+    >>> refs = ["2211", "22", "3112"]
+    >>> preds = ["22",   "2213", "2211"]
+    >>> metric.compute(references=refs, predictions=preds, beta=1.0, average="both")
+    {'macro_hP': 0.5833333333333334, 'macro_hR': 0.5, 'macro_hF_beta_mean': 0.5, 'macro_hF_beta_from_macroPR': 0.5384615384615384, 'micro_hP': 0.6, 'micro_hR': 0.5, 'micro_hF_beta': 0.5454545454545454}
 """
 
-# TODO: Define external resources urls if needed
-BAD_WORDS_URL = "http://url/to/external/resource/bad_words.txt"
+# Optional external resources
+ISCO_CODES_URL = (
+    "https://www.ilo.org/ilostat-files/ISCO/newdocs-08-2021/ISCO-08/ISCO-08%20EN.csv"
+)
 
 
 @evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
-class isco_hierachical_accuracy_v2(evaluate.Metric):
-    """TODO: Short description of my evaluation module."""
+class isco_hierachical_accuracy_v2(evaluate.Metric):  # keep class name as requested
+    """Hierarchical ISCO-08 evaluation metric for hP/hR/hFβ."""
 
     def _info(self):
-        # TODO: Specifies the evaluate.EvaluationModuleInfo object
+        # The features describe how inputs are structured when used with a Dataset;
+        # compute(...) can still accept raw Python lists.
         return evaluate.MetricInfo(
-            # This is the description that will appear on the modules page.
             module_type="metric",
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
-            # This defines the format of each prediction and reference
-            features=datasets.Features({
-                'predictions': datasets.Value('int64'),
-                'references': datasets.Value('int64'),
-            }),
-            # Homepage of the module for documentation
-            homepage="http://module.homepage",
-            # Additional links to the codebase or references
-            codebase_urls=["http://github.com/path/to/codebase/of/new_module"],
-            reference_urls=["http://path.to.reference.url/new_module"]
+            features=datasets.Features(
+                {
+                    "predictions": datasets.Sequence(datasets.Value("string")),
+                    "references": datasets.Sequence(datasets.Value("string")),
+                }
+            ),
+            homepage="https://isco.ilo.org/en/isco-08/",
+            codebase_urls=["https://github.com/huggingface/evaluate"],
+            reference_urls=[
+                "https://www.ilo.org/ilostat-files/ISCO/newdocs-08-2021/ISCO-08/ISCO-08%20EN%20Vol%201.pdf",
+                "https://www.ilo.org/ilostat-files/ISCO/newdocs-08-2021/ISCO-08/ISCO-08%20EN%20Structure%20and%20definitions.xlsx",
+                "https://www.ilo.org/ilostat-files/ISCO/newdocs-08-2021/ISCO-08/ISCO-08%20-88%20EN%20Index.xlsx",
+            ],
         )
 
     def _download_and_prepare(self, dl_manager):
-        """Optional: download external resources useful to compute the scores"""
-        # TODO: Download external resources if needed
+        # No external assets are required.
         pass
 
-    def _compute(self, predictions, references):
-        """Returns the scores"""
-        # TODO: Compute the different scores of the module
-        accuracy = sum(i == j for i, j in zip(predictions, references)) / len(predictions)
-        return {
-            "accuracy": accuracy,
-        }
+    def _compute(
+        self,
+        predictions,
+        references,
+        beta: float = 1.0,
+        average: str = "both",
+        return_per_instance: bool = False,
+    ):
+        """
+        Returns hierarchical precision/recall/Fβ (micro/macro).
+        """
+        results = hierarchical_scores(
+            y_true=references,
+            y_pred=predictions,
+            beta=beta,
+            average=average,
+            return_per_instance=return_per_instance,
+        )
+        return results

tests.py

@@ -1,17 +1,134 @@
 test_cases = [
     {
-        "predictions": [0, 0],
-        "references": [1, 1],
-        "result": {"metric_score": 0}
+        "references": [
+            "1111",
+            "1111",
+            "1111",
+            "1111",
+            "1111",
+            "1111",
+            "1111",
+            "1111",
+            "1111",
+            "1111",
+        ],
+        "predictions": [
+            "1111",
+            "1112",
+            "1120",
+            "1211",
+            "1311",
+            "2111",
+            "111",
+            "11",
+            "1",
+            "9999",
+        ],
+        "result": {
+            "accuracy": 0.1111111111111111,
+            "hierarchical_precision": 0.26666666666666666,
+            "hierarchical_recall": 1.0,
+            "hierarchical_fmeasure": 0.4210526315789474,
+        },
     },
     {
-        "predictions": [1, 1],
-        "references": [1, 1],
-        "result": {"metric_score": 1}
+        "references": ["1111"],
+        "predictions": ["1111"],
+        "result": {
+            "accuracy": 1.0,
+            "hierarchical_precision": 1.0,
+            "hierarchical_recall": 1.0,
+            "hierarchical_fmeasure": 1.0,
+        },
     },
     {
-        "predictions": [1, 0],
-        "references": [1, 1],
-        "result": {"metric_score": 0.5}
-    }
-]
+        "references": ["1111"],
+        "predictions": ["1112"],
+        "result": {
+            "accuracy": 0.0,
+            "hierarchical_precision": 0.75,
+            "hierarchical_recall": 0.75,
+            "hierarchical_fmeasure": 0.75,
+        },
+    },
+    {
+        "references": ["1111"],
+        "predictions": ["1120"],
+        "result": {
+            "accuracy": 0.0,
+            "hierarchical_precision": 0.5,
+            "hierarchical_recall": 0.5,
+            "hierarchical_fmeasure": 0.5,
+        },
+    },
+    {
+        "references": ["1111"],
+        "predictions": ["1211"],
+        "result": {
+            "accuracy": 0.0,
+            "hierarchical_precision": 0.25,
+            "hierarchical_recall": 0.25,
+            "hierarchical_fmeasure": 0.25,
+        },
+    },
+    {
+        "references": ["1111"],
+        "predictions": ["1311"],
+        "result": {
+            "accuracy": 0.0,
+            "hierarchical_precision": 0.25,
+            "hierarchical_recall": 0.25,
+            "hierarchical_fmeasure": 0.25,
+        },
+    },
+    {
+        "references": ["1111"],
+        "predictions": ["2111"],
+        "result": {
+            "accuracy": 0.0,
+            "hierarchical_precision": 0.0,
+            "hierarchical_recall": 0.0,
+            "hierarchical_fmeasure": 0,
+        },
+    },
+    {
+        "references": ["1111"],
+        "predictions": ["111"],
+        "result": {
+            "accuracy": 0.0,
+            "hierarchical_precision": 1.0,
+            "hierarchical_recall": 0.25,
+            "hierarchical_fmeasure": 0.4,
+        },
+    },
+    {
+        "references": ["1111"],
+        "predictions": ["11"],
+        "result": {
+            "accuracy": 0.0,
+            "hierarchical_precision": 1.0,
+            "hierarchical_recall": 0.25,
+            "hierarchical_fmeasure": 0.4,
+        },
+    },
+    {
+        "references": ["1111"],
+        "predictions": ["1"],
+        "result": {
+            "accuracy": 0.0,
+            "hierarchical_precision": 1.0,
+            "hierarchical_recall": 0.25,
+            "hierarchical_fmeasure": 0.4,
+        },
+    },
+    {
+        "references": ["1111"],
+        "predictions": ["9999"],
+        "result": {
+            "accuracy": 0.0,
+            "hierarchical_precision": 0.0,
+            "hierarchical_recall": 0.0,
+            "hierarchical_fmeasure": 0,
+        },
+    },
+]