Auditing GDPval Coverage and Related Hub Models
Source:vignettes/openai-gdpval-benchmark.Rmd
openai-gdpval-benchmark.RmdQuestion and scope
GDPval is a public benchmark dataset of professional task prompts, reference files, deliverables, and rubrics. This case study audits the dataset’s coverage from R and shows how to discover Hub models that mention GDPval in their metadata.
This article does not run benchmark evaluations or infer model capability. A model tag, download count, or search result is metadata, not a GDPval score.
To render the live outputs, set
HF_CASE_STUDY_RUN_PUBLIC_DATA=true before building the
article.
Load the public Parquet file
GDPval contains list columns for reference and deliverable files. Reading the public Parquet file preserves those columns cleanly.
library(dplyr)
library(stringr)
library(tibble)
if (!requireNamespace("arrow", quietly = TRUE)) {
stop("Install the 'arrow' package to build this article.", call. = FALSE)
}
gdpval_url <- paste0(
"https://huggingface.co/datasets/openai/gdpval/resolve/main/",
"data/train-00000-of-00001.parquet"
)
gdpval <- arrow::read_parquet(gdpval_url, as_data_frame = TRUE)
required_cols <- c(
"task_id", "sector", "occupation", "prompt",
"reference_files", "reference_file_urls", "reference_file_hf_uris",
"deliverable_files", "deliverable_file_urls", "deliverable_file_hf_uris",
"rubric_pretty", "rubric_json"
)
stopifnot(all(required_cols %in% names(gdpval)))
gdpval <- gdpval |>
mutate(
prompt_chars = nchar(prompt),
n_reference_files = lengths(reference_files),
n_deliverable_files = lengths(deliverable_files),
rubric_chars = nchar(rubric_pretty)
)
print(tibble(
source_dataset = "openai/gdpval",
source_file = "data/train-00000-of-00001.parquet",
rows = nrow(gdpval),
columns = ncol(gdpval) - 4,
sectors = n_distinct(gdpval$sector),
occupations = n_distinct(gdpval$occupation),
median_prompt_chars = median(gdpval$prompt_chars),
zero_reference_tasks = sum(gdpval$n_reference_files == 0),
max_reference_files = max(gdpval$n_reference_files)
), width = Inf)
#> # A tibble: 1 × 9
#> source_dataset source_file rows columns sectors
#> <chr> <chr> <int> <dbl> <int>
#> 1 openai/gdpval data/train-00000-of-00001.parquet 220 12 9
#> occupations median_prompt_chars zero_reference_tasks max_reference_files
#> <int> <dbl> <int> <int>
#> 1 44 2024. 95 17The first summary establishes the basic benchmark shape: 220 tasks, 12 source columns, 9 sectors, and 44 occupations. Prompt length and reference-file counts describe benchmark packaging, not task difficulty or model performance.
Inspect the source schema
print(tibble(
column = required_cols,
present = required_cols %in% names(gdpval),
role = c(
"unique task identifier",
"sector label",
"occupation label",
"task instructions",
"supporting file names",
"supporting file URLs",
"supporting file Hugging Face URIs",
"expected deliverable file names",
"expected deliverable file URLs",
"expected deliverable Hugging Face URIs",
"human-readable rubric",
"machine-readable rubric JSON"
)
), n = Inf, width = Inf)
#> # A tibble: 12 × 3
#> column present role
#> <chr> <lgl> <chr>
#> 1 task_id TRUE unique task identifier
#> 2 sector TRUE sector label
#> 3 occupation TRUE occupation label
#> 4 prompt TRUE task instructions
#> 5 reference_files TRUE supporting file names
#> 6 reference_file_urls TRUE supporting file URLs
#> 7 reference_file_hf_uris TRUE supporting file Hugging Face URIs
#> 8 deliverable_files TRUE expected deliverable file names
#> 9 deliverable_file_urls TRUE expected deliverable file URLs
#> 10 deliverable_file_hf_uris TRUE expected deliverable Hugging Face URIs
#> 11 rubric_pretty TRUE human-readable rubric
#> 12 rubric_json TRUE machine-readable rubric JSONThe deliverable and rubric fields are central to GDPval. They are the evidence needed for an actual evaluation, so a metadata audit should confirm they are present before any model claims are made.
Audit sector and occupation coverage
sector_counts <- gdpval |>
count(sector, sort = TRUE)
occupation_summary <- gdpval |>
count(occupation, sort = TRUE) |>
summarise(
occupations = n(),
min_tasks_per_occupation = min(n),
median_tasks_per_occupation = median(n),
max_tasks_per_occupation = max(n)
)
print(sector_counts, n = Inf, width = Inf)
#> # A tibble: 9 × 2
#> sector n
#> <chr> <int>
#> 1 Finance and Insurance 25
#> 2 Government 25
#> 3 Health Care and Social Assistance 25
#> 4 Information 25
#> 5 Manufacturing 25
#> 6 Professional, Scientific, and Technical Services 25
#> 7 Real Estate and Rental and Leasing 25
#> 8 Wholesale Trade 25
#> 9 Retail Trade 20
print(occupation_summary, width = Inf)
#> # A tibble: 1 × 4
#> occupations min_tasks_per_occupation median_tasks_per_occupation
#> <int> <int> <dbl>
#> 1 44 5 5
#> max_tasks_per_occupation
#> <int>
#> 1 5These counts describe benchmark coverage. They should not be read as the distribution of activity in the broader economy or as sector-level AI exposure.
Audit prompt and file burden
prompt_summary <- gdpval |>
summarise(
min_prompt_chars = min(prompt_chars),
median_prompt_chars = median(prompt_chars),
max_prompt_chars = max(prompt_chars),
median_rubric_chars = median(rubric_chars)
)
reference_distribution <- gdpval |>
count(n_reference_files, name = "tasks") |>
arrange(n_reference_files)
deliverable_distribution <- gdpval |>
count(n_deliverable_files, name = "tasks") |>
arrange(n_deliverable_files)
print(prompt_summary, width = Inf)
#> # A tibble: 1 × 4
#> min_prompt_chars median_prompt_chars max_prompt_chars median_rubric_chars
#> <int> <dbl> <int> <dbl>
#> 1 617 2024. 6618 5344.
print(reference_distribution, n = Inf, width = Inf)
#> # A tibble: 11 × 2
#> n_reference_files tasks
#> <int> <int>
#> 1 0 95
#> 2 1 70
#> 3 2 26
#> 4 3 15
#> 5 4 5
#> 6 5 3
#> 7 6 2
#> 8 7 1
#> 9 8 1
#> 10 15 1
#> 11 17 1
print(deliverable_distribution, n = Inf, width = Inf)
#> # A tibble: 7 × 2
#> n_deliverable_files tasks
#> <int> <int>
#> 1 0 35
#> 2 1 141
#> 3 2 32
#> 4 3 8
#> 5 4 2
#> 6 5 1
#> 7 6 1The reference-file distribution shows whether a task is pure prompt reading or requires supporting documents. That matters for evaluation logistics, but it is not by itself a measure of task difficulty.
Check list-column integrity
For reference and deliverable metadata, the file names, URLs, and Hugging Face URIs should have matching lengths within each task.
integrity <- gdpval |>
transmute(
reference_lengths_match =
lengths(reference_files) == lengths(reference_file_urls) &
lengths(reference_files) == lengths(reference_file_hf_uris),
deliverable_lengths_match =
lengths(deliverable_files) == lengths(deliverable_file_urls) &
lengths(deliverable_files) == lengths(deliverable_file_hf_uris),
has_rubric_pretty = !is.na(rubric_pretty) & rubric_chars > 0,
has_rubric_json = !is.na(rubric_json) & nchar(rubric_json) > 0
)
integrity_summary <- integrity |>
summarise(
reference_lengths_all_match = all(reference_lengths_match),
deliverable_lengths_all_match = all(deliverable_lengths_match),
tasks_with_rubric_pretty = sum(has_rubric_pretty),
tasks_with_rubric_json = sum(has_rubric_json)
)
print(integrity_summary, width = Inf)
#> # A tibble: 1 × 4
#> reference_lengths_all_match deliverable_lengths_all_match
#> <lgl> <lgl>
#> 1 TRUE TRUE
#> tasks_with_rubric_pretty tasks_with_rubric_json
#> <int> <int>
#> 1 220 220This check is a guardrail before downloading files or scoring model outputs. If file-name and URI lengths do not align, downstream evaluation code could silently use the wrong supporting materials.
Inspect reference-heavy examples
reference_heavy_examples <- gdpval |>
arrange(desc(n_reference_files), desc(prompt_chars)) |>
transmute(
task_id,
sector,
occupation,
prompt_chars,
n_reference_files,
n_deliverable_files,
prompt_preview = str_trunc(str_squish(prompt), 90)
) |>
slice_head(n = 8)
print(reference_heavy_examples, n = Inf, width = Inf)
#> # A tibble: 8 × 7
#> task_id
#> <chr>
#> 1 ee09d943-5a11-430a-b7a2-971b4e9b01b5
#> 2 43dc9778-450b-4b46-b77e-b6d82b202035
#> 3 4d1a8410-e9c5-4be5-ab43-cc55563c594c
#> 4 6974adea-8326-43fa-8187-2724b15d9546
#> 5 7d7fc9a7-21a7-4b83-906f-416dea5ad04f
#> 6 4b894ae3-1f23-4560-b13d-07ed1132074e
#> 7 90edba97-74f0-425a-8ff6-8b93182eb7cb
#> 8 6a900a40-8d2b-4064-a5b1-13a60bc173d8
#> sector
#> <chr>
#> 1 Professional, Scientific, and Technical Services
#> 2 Professional, Scientific, and Technical Services
#> 3 Health Care and Social Assistance
#> 4 Information
#> 5 Professional, Scientific, and Technical Services
#> 6 Information
#> 7 Health Care and Social Assistance
#> 8 Wholesale Trade
#> occupation
#> <chr>
#> 1 Accountants and Auditors
#> 2 Accountants and Auditors
#> 3 First-Line Supervisors of Office and Administrative Support Workers
#> 4 News Analysts, Reporters, and Journalists
#> 5 Accountants and Auditors
#> 6 Audio and Video Technicians
#> 7 Registered Nurses
#> 8 Sales Representatives, Wholesale and Manufacturing, Technical and Scientific …
#> prompt_chars n_reference_files n_deliverable_files
#> <int> <int> <int>
#> 1 2987 17 1
#> 2 714 15 2
#> 3 3434 8 3
#> 4 2803 7 0
#> 5 2403 6 1
#> 6 2064 6 0
#> 7 3364 5 1
#> 8 2975 5 1
#> prompt_preview
#> <chr>
#> 1 As our Senior Staff Accountant in Financial Reporting & Assembly, you’ve been…
#> 2 You are a mid-level Tax Preparer at an accounting firm. You have been given t…
#> 3 Every year, Nu Arc Medical Center (NAMC) interviews applicants for a position…
#> 4 You are a technology journalist at a respected online news publisher, working…
#> 5 You are a Senior Staff Accountant at Aurisic. You have been tasked with prepa…
#> 6 You’re an audio mix engineer working at a reputable recording studio. A new a…
#> 7 You are a registered nurse at a dialysis facility. At your dialysis facility,…
#> 8 You are an account manager for an international medical wholesaler, Danish Wh…These examples are useful for planning an evaluation run. Reference-heavy tasks may require file parsing, retrieval, or multimodal handling in addition to text generation.
Discover Hub model metadata
The Hub can be searched for model metadata that mentions GDPval. Treat these results as leads, not benchmark scores.
model_mentions <- tryCatch(
{
huggingfaceR::hf_search_models(search = "gdpval", limit = 10) |>
transmute(
model_id,
task,
downloads,
likes,
evidence = "Hub model search result for 'gdpval'"
)
},
error = function(e) {
tibble(
model_id = paste("Hub search failed:", conditionMessage(e)),
task = NA_character_,
downloads = NA_integer_,
likes = NA_integer_,
evidence = "Search did not complete"
)
}
)
print(model_mentions, n = Inf, width = Inf)
#> # A tibble: 1 × 5
#> model_id task downloads likes
#> <chr> <chr> <int> <int>
#> 1 dtometzki/Qwen2.5-Coder-7B-gdpval text-generation 6 0
#> evidence
#> <chr>
#> 1 Hub model search result for 'gdpval'A search result means the model card or metadata references GDPval. It does not mean the model is official, that it was fine-tuned on GDPval, or that it achieved any particular benchmark score.
Evidence ladder for model claims
print(tibble(
evidence_level = c(
"Official GDPval result",
"Reproducible evaluation run",
"Model card self-report",
"Hub metadata or search hit"
),
safe_wording = c(
"The official source reports this result.",
"This exact model/version was run on GDPval with documented prompts, files, and scoring.",
"The model card reports a GDPval-related result; verify the method before comparing.",
"The model metadata mentions GDPval; this is only a discovery lead."
)
), n = Inf, width = Inf)
#> # A tibble: 4 × 2
#> evidence_level
#> <chr>
#> 1 Official GDPval result
#> 2 Reproducible evaluation run
#> 3 Model card self-report
#> 4 Hub metadata or search hit
#> safe_wording
#> <chr>
#> 1 The official source reports this result.
#> 2 This exact model/version was run on GDPval with documented prompts, files, an…
#> 3 The model card reports a GDPval-related result; verify the method before comp…
#> 4 The model metadata mentions GDPval; this is only a discovery lead.The key distinction is evidence. Metadata discovery helps find candidate models or cards to read next, but it should not be turned into a leaderboard.
Takeaways
This case study audited GDPval as a public benchmark dataset, not as a model-performance claim. The reusable workflow is: load the Parquet file, verify the schema, summarize task and file coverage, inspect rubrics and deliverables, and treat Hub model search results as metadata leads only.