library(dplyr)
library(ggplot2)
library(grf)
library(knitr)
library(lmtest)
library(sandwich)
library(scales)
library(tidyr)Design-Based Supervised Learning: A Tutorial
From Imperfect ML/LLM Labels to Valid Downstream Inference
This tutorial introduces Design-Based Supervised Learning (DSL), a framework proposed by Egami et al. in Using Imperfect Surrogates for Downstream Inference: Design-Based Supervised Learning for Social Science Applications of Large Language Models [2023]. DSL uses imperfect ML/LLM labels together with a small expert-labeled subsample to recover valid downstream statistical analysis. The tutorial explains the core statistical idea and implements a reproducible simulation in R.
Suppose a researcher uses an LLM to label the editorial tone of 50,000 news articles and then studies how tone is associated with later stock returns. If the LLM makes different kinds of errors for different articles, a naive regression using the LLM labels can be biased even when the labels look accurate overall; DSL shows how to use a smaller expert-labeled subsample to correct the downstream analysis.
Table of Contents
1 Getting Started
This tutorial is fully reproducible. It uses synthetic data, so no external data files are required. The code requires the following R packages: dplyr, ggplot2, grf, knitr, tidyr, sandwich, lmtest, and scales.
The HTML version includes a simple table of contents at the top of the page. Each item links to a major section of the tutorial.
2 What is DSL?
The basic idea of DSL is:
- ML/LLM labels for documents (or any other multimodal data) are useful but imperfect – inaccurate and biased; expert labels are more precise and less biased – but expensive.
- Imperfect labels become a problem when they are used as if they were expert labels in downstream statistical analysis, especially when prediction errors are not harmless random noise.
- DSL uses a known-probability expert-labeled subsample to construct an augmented inverse-probability-weighted pseudo-outcome. Because the expert-labeling probabilities are known by design, the correction remains centered on the oracle expert-label target even if the supervised prediction model is imperfect.
2.1 Objective
Suppose we study a large collection of documents. Each document has a covariate \(X_i\), and our target is a regression using an expert-coded binary label \(Y_i\):
\[ Y_i = \alpha + \beta X_i + \varepsilon_i. \]
If every document had an expert label, we could run this regression directly. That full expert-label regression is the oracle analysis. (We will use this as benchmark.)
However, in practice, we usually observe:
- a surrogate label \(Q_i\), such as an ML or LLM label, for every document;
- the expert label \(Y_i\) only for a smaller subsample.
The goal of DSL is not to perfectly predict every missing \(Y_i\). The goal is to recover the oracle downstream quantity, such as the slope \(\beta\), as closely as possible while reporting valid uncertainty.
2.2 Notations and Assumptions
We use the following notation:
| Symbol | Meaning |
|---|---|
| \(Y_i\) | expert label, observed only for labeled units |
| \(Q_i\) | AI or surrogate label, observed for all units |
| \(X_i\) | covariate used in the downstream analysis |
| \(W_i\) | additional feature useful for predicting \(Y_i\), but not the target regressor |
| \(R_i\) | indicator that unit \(i\) is expert labeled |
| \(\pi_i\) | probability that unit \(i\) is expert labeled |
| \(\hat g_i\) | supervised prediction of \(Y_i\), often using \(Q_i\), \(X_i\), and other features |
Following Egami et al. (2023), this tutorial uses \(W_i\) for an auxiliary feature; note that this differs from the common causal-inference convention where \(W_i\) often denotes a treatment indicator.
The central design assumption is:
\[ E(R_i \mid Q_i, X_i, W_i) = \pi_i, \quad 0 < \pi_i \leq 1, \]
and \(\pi_i\) is known to the analyst. This known-probability design is where the design-based guarantee comes from.
This assumption is plausible when researchers draw and record the expert-labeling sample themselves. For example, simple random sampling gives \(\pi_i = n/N\), and stratified sampling gives one known probability per stratum.
DSL deliberately does not assume:
- the surrogate is highly accurate;
- prediction errors are random;
- prediction errors are mean-zero;
- the supervised prediction model \(\hat g_i\) is correctly specified.
2.3 What DSL Does and Does Not Identify
DSL corrects a measurement or labeling problem in the outcome used for downstream analysis. It does not by itself solve confounding or identify a causal effect. If the downstream analysis is causal, causal identification must come from a separate design, such as random assignment, unconfoundedness, difference-in-differences, or an instrumental-variable design.
In this tutorial, the downstream target is the oracle regression slope that would be obtained if every unit had an expert label. In a causal application, the same DSL correction could be used when the outcome is a text-coded or LLM-coded variable, but the causal interpretation would still require separate identifying assumptions.
2.4 A Small Distinction: Inaccurate vs Harmful Bias
It is useful to separate two ideas.
First, labels can be inaccurate at the unit level. This means \(Q_i \neq Y_i\) for some observations.
Second, errors can be harmful for downstream inference. For a regression on \(X_i\), the key condition for naive surrogate-only regression is much stronger than “some labels are randomly wrong.” It is closer to:
\[ E(Q_i - Y_i \mid X_i) = 0. \]
If this condition held exactly, replacing \(Y_i\) with \(Q_i\) might not bias the downstream slope. But the DSL papers emphasize that this condition is usually hard to justify. Prediction errors can depend on \(X_i\), the true label, subgroups, prompts for LLM, or features omitted from the downstream regression.
3 Roadmap of the Method
Before writing code, it helps to separate the method into four stages.
| Stage | What we do | Why it matters |
|---|---|---|
| 1. Define the target | Specify the oracle downstream analysis, such as regressing the expert label \(Y_i\) on \(X_i\). | DSL targets this oracle analysis, not perfect unit-level prediction. |
| 2. Design expert labeling | Draw an expert-labeled subsample with known positive probability \(\pi_i\). | This known design makes the correction valid. |
| 3. Build pseudo-outcomes | Start with a prediction \(\hat g_i\) and add the correction \(R_i(Y_i-\hat g_i)/\pi_i\). | The correction recenters the outcome on the expert-label target. |
| 4. Run and evaluate | Regress the pseudo-outcome on \(X_i\), then check bias, RMSE, coverage, and sensitivity. | This shows whether DSL recovers the oracle analysis with useful precision. |
In code, we first set \(\hat g_i=Q_i\) to show the raw correction. We then learn a cross-fitted \(\hat g_i=\hat g(Q_i,X_i,W_i)\) with GRF. The target is the same in both cases; the supervised learner can improve efficiency if it makes the residual \(Y_i-\hat g_i\) smaller.
4 Step-by-Step Implementation
Now we simulate a synthetic dataset to show the implementation of DSL. The simulation is intentionally simple:
- \(X_i\) is the covariate in the downstream regression.
- \(W_i\) is an extra full-sample feature that helps predict the expert label.
- \(Y_i\) is the expert label.
- \(Q_i\) is a biased and imprecise surrogate label observed for everyone.
The simulation generates \(Y_i\) for every observation so that we can benchmark each method against the oracle. In real applications, \(Y_i\) is observed only for the expert-labeled subsample.
4.1 Step 1: Simulate a Biased Surrogate and a Useful Feature
The first block creates a population with three important objects:
x: the covariate in the downstream regression. This is the variable whose slope we ultimately care about.w: an extra feature that helps predict the expert label but is not itself the target of the downstream analysis.y_expert: the expert label we wish we had for every observation.
Then we construct y_surrogate, an imperfect AI-style label. Its error rates depend on x, so the surrogate can have high accuracy while still biasing the downstream slope.
set.seed(20260506)
make_population <- function(n = 6000) {
x <- rnorm(n)
w <- 0.5 * x + rnorm(n)
p_y <- plogis(-0.25 + 0.75 * x + 0.55 * w)
y <- rbinom(n, size = 1, prob = p_y)
false_negative <- plogis(-2.1 + 0.8 * x)
false_positive <- plogis(-2.3 - 0.5 * x)
q <- ifelse(
y == 1,
rbinom(n, size = 1, prob = 1 - false_negative),
rbinom(n, size = 1, prob = false_positive)
)
tibble(
id = seq_len(n),
x = x,
w = w,
y_expert = y,
y_surrogate = q,
surrogate_error = y_expert != y_surrogate
)
}
population <- make_population()
oracle_fit <- lm(y_expert ~ x, data = population)
oracle_slope <- unname(coef(oracle_fit)[2])
surrogate_fit <- lm(y_surrogate ~ x, data = population)
label_quality <- tibble(
quantity = c(
"Surrogate accuracy",
"Mean expert label",
"Mean surrogate label",
"Oracle slope",
"Surrogate-only slope",
"Surrogate-only slope bias"
),
value = c(
mean(population$y_surrogate == population$y_expert),
mean(population$y_expert),
mean(population$y_surrogate),
oracle_slope,
unname(coef(surrogate_fit)[2]),
unname(coef(surrogate_fit)[2]) - oracle_slope
)
)
kable(label_quality, digits = 3)| quantity | value |
|---|---|
| Surrogate accuracy | 0.861 |
| Mean expert label | 0.459 |
| Mean surrogate label | 0.450 |
| Oracle slope | 0.205 |
| Surrogate-only slope | 0.078 |
| Surrogate-only slope bias | -0.127 |
The table compares the oracle expert-label analysis with the naive surrogate-only analysis. The surrogate has reasonably high accuracy, but its downstream slope is biased because errors vary with x.
error_by_x <- population %>%
mutate(
x_bin = cut(
x,
breaks = quantile(x, probs = seq(0, 1, 0.1)),
include.lowest = TRUE
)
) %>%
group_by(x_bin) %>%
summarize(
mean_x = mean(x),
error_rate = mean(surrogate_error),
.groups = "drop"
)
ggplot(error_by_x, aes(x = mean_x, y = error_rate)) +
geom_line(linewidth = 0.8, color = "#2C7FB8") +
geom_point(size = 2.4, color = "#2C7FB8") +
scale_y_continuous(labels = scales::percent_format(accuracy = 1)) +
labs(
x = "Mean X within decile",
y = "Surrogate error rate",
title = "Surrogate error is not evenly distributed"
) +
theme_minimal(base_size = 12)The plot shows why accuracy is not enough. Error rates are not flat across x, so a regression using y_surrogate can estimate the wrong relationship between the label and x.
4.2 Step 2: Draw the Expert-Labeled Sample
We draw a simple random sample of expert-labeled observations, giving each unit the known probability \(\pi_i = n/N\). The function records:
labeled: the indicator \(R_i\), equal to 1 if the unit is expert-labeled;sample_prob: the known sampling probability \(\pi_i = n/N\);y_expert_obs: the expert label as the analyst observes it, withNAfor unlabeled units.
In real data, y_expert would be missing for unlabeled rows. The simulation keeps it only for evaluation.
label_sample <- function(data, n_labeled = 1000) {
n <- nrow(data)
labeled_ids <- sample(data$id, size = n_labeled)
data %>%
mutate(
labeled = as.integer(id %in% labeled_ids),
sample_prob = n_labeled / n,
y_expert_obs = ifelse(labeled == 1, y_expert, NA_real_)
)
}
set.seed(99)
demo_data <- label_sample(population, n_labeled = 1000)
demo_data %>%
summarize(
n = n(),
expert_labeled = sum(labeled),
sample_prob = first(sample_prob),
labeled_surrogate_accuracy =
mean(y_surrogate[labeled == 1] == y_expert[labeled == 1])
) %>%
kable(digits = 3)| n | expert_labeled | sample_prob | labeled_surrogate_accuracy |
|---|---|---|---|
| 6000 | 1000 | 0.167 | 0.87 |
The summary confirms the annotation design. The known probability is what lets the expert-labeled subsample represent the full population in the correction.
4.3 Step 3: Build Raw DSL
This is the simplest teaching version: use \(Q_i\) itself as \(\hat g_i\).
\[ \tilde Y_i^{raw} = Q_i + \frac{R_i}{\pi_i}(Y_i - Q_i). \]
4.3.1 Why Raw DSL Is Unbiased for the Downstream Regression
The raw version separates validity from efficiency. Conditional on the full population values \((Y_i,Q_i,X_i,W_i)\), the only randomness is whether unit \(i\) is selected for expert labeling. Since the design gives \(E_R(R_i \mid Y_i,Q_i,X_i,W_i)=\pi_i\),
\[ \begin{aligned} E_R(\tilde Y_i^{raw} \mid Y_i, Q_i, X_i, W_i) &= Q_i + \frac{E_R(R_i \mid Y_i,Q_i,X_i,W_i)}{\pi_i}(Y_i-Q_i) \\ &= Q_i + (Y_i-Q_i) = Y_i. \end{aligned} \]
So \(Q_i\) can be biased or inaccurate, but the pseudo-outcome is still centered on the expert label when \(\pi_i\) is correct. In this linear regression example, OLS is linear in the outcome, so using \(\tilde Y_i^{raw}\) gives a slope centered on the oracle slope. Raw DSL can still be noisy; the fuller DSL version replaces \(Q_i\) with a learned, cross-fitted \(\hat g_i\) to reduce the residual \(Y_i-\hat g_i\).
In real data, \(Y_i\) is missing for unlabeled observations. The code therefore uses y_expert_obs and sets the correction term to zero when labeled == 0.
The code constructs two new variables:
raw_correction: the inverse-probability weighted correction. For labeled observations, it compares the expert label to the surrogate label and upweights that difference by \(1 / \pi_i\). For unlabeled observations, the correction is zero because no expert label is available.y_tilde_raw: the raw DSL pseudo-outcome. It starts from the surrogate label and then adds the expert-label correction.
After constructing this pseudo-outcome, we run the same downstream regression as before: outcome on x. The only change is which outcome we use.
demo_data <- demo_data %>%
mutate(
raw_correction = ifelse(
labeled == 1,
(y_expert_obs - y_surrogate) / sample_prob,
0
),
y_tilde_raw = y_surrogate + raw_correction
)
expert_only_fit <- lm(y_expert_obs ~ x, data = filter(demo_data, labeled == 1))
raw_dsl_fit <- lm(y_tilde_raw ~ x, data = demo_data)
baseline_comparison <- tibble(
method = c("Oracle", "Surrogate-only"),
slope = c(coef(oracle_fit)[2], coef(surrogate_fit)[2]),
slope_bias = slope - oracle_slope
)
kable(baseline_comparison, digits = 3)| method | slope | slope_bias |
|---|---|---|
| Oracle | 0.205 | 0.000 |
| Surrogate-only | 0.078 | -0.127 |
This small table only sets the baseline: the surrogate-only regression is different from the oracle regression. Raw DSL has now been constructed, but the full comparison among expert-only, raw DSL, and cross-fitted DSL is easier to interpret after all methods are available. The full Algorithm 1 version from the NeurIPS paper keeps the same correction and first learns a better prediction \(\hat g_i\); its potential advantage is efficiency, not a different target. This is why the next step is not just “use a better model,” but “use a better model out of fold.”
4.4 Step 4: Learn Cross-Fitted \(\hat g_i\) with GRF
Now replace the raw surrogate \(Q_i\) with a supervised prediction \(\hat g_i\). We split observations into 5 folds. For each fold, we train a generalized random forest on expert-labeled observations outside the fold, then predict \(Y_i\) inside the held-out fold.
The learner uses the surrogate \(Q_i\), the downstream covariate \(X_i\), and the extra feature \(W_i\):
\[ \hat g_k(Q_i, X_i, W_i) \approx E(Y_i \mid Q_i, X_i, W_i). \]
The learner \(\hat g_i\) is used for efficiency: a better prediction makes the correction residual \(Y_i-\hat g_i\) smaller. Cross-fitting is used for separation: the expert label used in the correction term should not also train the prediction for that same observation. Without cross-fitting, a flexible learner could overfit the expert-labeled observations and make the residual correction look artificially small. Cross-fitting keeps each prediction out-of-fold while targeting the same oracle downstream analysis.
The cross-fitting function does three things:
- split all observations into folds;
- train
grf::regression_forest()on expert-labeled observations outside the held-out fold; - predict \(\hat g_i\) for every observation in the held-out fold.
This matters because the same expert label should not be used both to train \(\hat g_i\) and to evaluate its own correction residual.
crossfit_grf_prediction <- function(data, features, outcome, labeled,
k = 5, num.trees = 200,
seed = 1234) {
n <- nrow(data)
set.seed(seed)
fold_id <- sample(rep(seq_len(k), length.out = n))
g_hat <- rep(NA_real_, n)
for (fold in seq_len(k)) {
train_id <- fold_id != fold & data[[labeled]] == 1
test_id <- fold_id == fold
forest_fit <- grf::regression_forest(
X = as.matrix(data[train_id, features, drop = FALSE]),
Y = data[[outcome]][train_id],
num.trees = num.trees,
seed = seed + fold
)
g_hat[test_id] <- predict(
forest_fit,
newdata = as.matrix(data[test_id, features, drop = FALSE])
)$predictions
}
pmin(pmax(g_hat, 0), 1)
}The output is an out-of-fold prediction for every observation. We then build the fuller DSL pseudo-outcome:
\[ \tilde Y_i^{GRF} = \hat g_i + \frac{R_i}{\pi_i}(Y_i - \hat g_i). \]
The target is unchanged. The gain comes from replacing \(Q_i\) with a better prediction, which can make the residual \(Y_i - \hat g_i\) smaller.
grf_features <- c("y_surrogate", "x", "w")
demo_data <- demo_data %>%
mutate(
g_hat_grf = crossfit_grf_prediction(
data = demo_data,
features = grf_features,
outcome = "y_expert_obs",
labeled = "labeled",
k = 5,
num.trees = 200,
seed = 1234
),
grf_correction = ifelse(
labeled == 1,
(y_expert_obs - g_hat_grf) / sample_prob,
0
),
y_tilde_grf = g_hat_grf + grf_correction
)
grf_dsl_fit <- lm(y_tilde_grf ~ x, data = demo_data)
prediction_diagnostic <- tibble(
prediction = c("Raw surrogate Q", "Cross-fitted GRF prediction"),
mse_against_expert_label = c(
mean((demo_data$y_surrogate - demo_data$y_expert)^2),
mean((demo_data$g_hat_grf - demo_data$y_expert)^2)
)
)
kable(prediction_diagnostic, digits = 3)| prediction | mse_against_expert_label |
|---|---|
| Raw surrogate Q | 0.139 |
| Cross-fitted GRF prediction | 0.093 |
The MSE diagnostic uses the simulated full y_expert; in real data, it would have to be estimated on labeled data or by cross-validation. The point is the efficiency intuition: DSL validity comes from the design correction, while efficiency improves when \(\hat g_i\) predicts \(Y_i\) well.
5 Evaluation: Did DSL Recover the Oracle Analysis?
The central question is whether the feasible analyses move us closer to the oracle downstream regression. In a real application, the full expert-label oracle is unobserved. In this simulated tutorial, however, we kept the full expert label \(Y_i\) so that we can compare each feasible estimator to the oracle slope.
The diagnostic is simple:
\[ \text{error}(\hat\beta_m) = \hat\beta_m - \hat\beta_{\text{oracle}}, \]
where \(m\) indexes the method. We also report robust confidence intervals and whether each interval contains the oracle slope in this simulated draw. This is not a full Monte Carlo coverage study; it is a compact check that links the code back to the target.
slope_summary <- function(fit, method) {
robust_table <- lmtest::coeftest(
fit,
vcov. = sandwich::vcovHC(fit, type = "HC1")
)
tibble(
method = method,
slope = robust_table["x", "Estimate"],
std_error = robust_table["x", "Std. Error"]
)
}
evaluation_table <- bind_rows(
slope_summary(surrogate_fit, "Surrogate-only"),
slope_summary(expert_only_fit, "Expert-only"),
slope_summary(raw_dsl_fit, "Raw DSL: g = Q"),
slope_summary(grf_dsl_fit, "Cross-fitted DSL with GRF")
) %>%
mutate(
slope_bias = slope - oracle_slope,
conf_low = slope - 1.96 * std_error,
conf_high = slope + 1.96 * std_error,
contains_oracle = conf_low <= oracle_slope & conf_high >= oracle_slope
)
kable(evaluation_table, digits = 3)| method | slope | std_error | slope_bias | conf_low | conf_high | contains_oracle |
|---|---|---|---|---|---|---|
| Surrogate-only | 0.078 | 0.006 | -0.127 | 0.066 | 0.091 | FALSE |
| Expert-only | 0.215 | 0.012 | 0.010 | 0.191 | 0.239 | TRUE |
| Raw DSL: g = Q | 0.187 | 0.014 | -0.018 | 0.160 | 0.213 | TRUE |
| Cross-fitted DSL with GRF | 0.199 | 0.010 | -0.006 | 0.180 | 0.219 | TRUE |
Read the table as a downstream analysis diagnostic rather than a label-prediction score. Surrogate-only uses all observations but can target the wrong regression. Expert-only targets the oracle analysis but uses only the expert-labeled sample. DSL uses all observations and then uses the expert-labeled sample to correct the downstream target.
plot_table <- evaluation_table %>%
mutate(method = factor(method, levels = rev(method)))
ggplot(plot_table, aes(y = method, x = slope)) +
geom_vline(xintercept = oracle_slope, linetype = "dashed",
color = "firebrick") +
geom_segment(aes(x = conf_low, xend = conf_high,
y = method, yend = method),
linewidth = 0.7) +
geom_point(size = 2.4, color = "steelblue") +
labs(
x = "Estimated slope on X",
y = NULL,
title = "DSL targets the full expert-label oracle analysis"
) +
theme_minimal(base_size = 12)The dashed line is the oracle slope. The tutorial lesson is not that DSL perfectly predicts every expert label; it is that the correction can move the downstream estimate away from a biased surrogate-only analysis and back toward the oracle target while retaining uncertainty from the expert-labeling design.
6 Sensitivity: What If Sampling Probabilities Are Approximate?
The clean DSL guarantee relies on known sampling probabilities. In the pseudo-outcome,
\[ \tilde Y_i = \hat g_i + \frac{R_i}{\pi_i}(Y_i - \hat g_i), \]
the factor \(R_i / \pi_i\) tells each expert-labeled observation how much of the full population it represents. If the true probability is \(\pi_i\) but the analyst uses \(\tilde \pi_i\), then the expected pseudo-outcome becomes:
\[ E_R(\tilde Y_i \mid Y_i, \hat g_i, X_i) = \hat g_i + \frac{\pi_i}{\tilde \pi_i}(Y_i - \hat g_i). \]
This equals \(Y_i\) only when \(\tilde \pi_i = \pi_i\). Otherwise, the remaining bias is related to both probability misspecification and the prediction residual \(Y_i - \hat g_i\).
In simple random sampling, this problem is usually avoidable because \(\pi_i = n/N\) is known by design. In stratified sampling, it is also avoidable if each stratum’s sampling probability is recorded. The risky cases are convenience labeling, undocumented active learning, nonresponse, or expert labels inherited from another project.
The simple sensitivity check below keeps the same expert-labeled sample and artificially changes the sampling probabilities used in the correction. The true design is simple random sampling, so delta = 0 is correct. Positive or negative delta makes the assumed probability vary with x, mimicking a situation where the analyst uses incorrect relative sampling weights.
prob_sensitivity <- tibble(delta = seq(-0.7, 0.7, by = 0.05)) %>%
rowwise() %>%
mutate(
raw_dsl_slope = {
assumed_prob <- demo_data$sample_prob *
exp(delta * as.numeric(scale(demo_data$x)))
assumed_prob <- assumed_prob *
mean(demo_data$sample_prob) / mean(assumed_prob)
assumed_prob <- pmin(pmax(assumed_prob, 1e-6), 1)
y_tilde_assumed <- demo_data$y_surrogate + ifelse(
demo_data$labeled == 1,
(demo_data$y_expert_obs - demo_data$y_surrogate) / assumed_prob,
0
)
coef(lm(y_tilde_assumed ~ demo_data$x))[2]
},
grf_dsl_slope = {
assumed_prob <- demo_data$sample_prob *
exp(delta * as.numeric(scale(demo_data$x)))
assumed_prob <- assumed_prob *
mean(demo_data$sample_prob) / mean(assumed_prob)
assumed_prob <- pmin(pmax(assumed_prob, 1e-6), 1)
y_tilde_assumed <- demo_data$g_hat_grf + ifelse(
demo_data$labeled == 1,
(demo_data$y_expert_obs - demo_data$g_hat_grf) / assumed_prob,
0
)
coef(lm(y_tilde_assumed ~ demo_data$x))[2]
}
) %>%
ungroup() %>%
pivot_longer(
cols = c(raw_dsl_slope, grf_dsl_slope),
names_to = "method",
values_to = "slope"
) %>%
mutate(
method = recode(
method,
raw_dsl_slope = "Raw DSL: g = Q",
grf_dsl_slope = "Cross-fitted DSL with GRF"
)
)
ggplot(prob_sensitivity, aes(x = delta, y = slope, color = method)) +
geom_hline(yintercept = oracle_slope, linetype = "dashed",
color = "firebrick") +
geom_vline(xintercept = 0, linetype = "dotted") +
geom_line(linewidth = 0.8) +
labs(
x = "Degree of relative sampling-probability misspecification",
y = "Estimated slope on X",
color = NULL,
title = "Known sampling probabilities are part of the design"
) +
theme_minimal(base_size = 12)This is a small sensitivity illustration rather than a full sensitivity analysis. DSL is strongest when the expert-labeling design is planned in advance and the sampling probabilities are recorded. If probabilities are only approximate, DSL may still help, but the clean design-based inference guarantee no longer follows automatically.
7 Takeaways and Limitations
The main lesson is that DSL targets the downstream analysis that would be run under a chosen benchmark annotation procedure. Surrogate labels can be useful even when they are biased, but they should not be analyzed as if they were expert labels. The design-based correction gives DSL its validity because the expert-labeling probabilities are known and positive; cross-fitted ML is primarily for precision, not for changing the target.
The main limitations follow from the same logic:
- DSL needs a planned annotation design. The “design” in DSL is the sampling design for expert annotation, not a causal design. DSL can be used in an observational causal study only if the causal assumptions are justified separately and the expert-labeled subset was sampled with known positive probabilities. A convenience set of expert labels, or labels collected through an undocumented annotation process, cannot be treated as DSL without extra assumptions.
- The benchmark labels still matter. DSL estimates the analysis we would have run if every unit had been labeled by the chosen benchmark procedure. It does not prove that the “expert” label is neutral or objectively correct. If the expert procedure is biased, contested, or unreliable, DSL reproduces that benchmark, so researchers should use clear codebooks, multiple coders, adjudication, audits, or uncertainty procedures when the benchmark itself is questionable.
- Precision gains are not automatic. The design correction gives validity even when the surrogate is biased, but smaller standard errors require a useful prediction \(\hat g_i\). If another method can credibly model the prediction errors directly, for example by assuming a parametric measurement-error model or a known error-rate structure, that more specialized method might be more efficient for that task. DSL trades some possible efficiency for weaker assumptions about prediction errors.
8 AI Disclosure
I used ChatGPT Codex to help improve the readability of the prose and code, check formatting, and keep the code style consistent. All errors remain my own, and the original ideas, organization, and final responsibility for this tutorial are mine.
9 References
Egami, Naoki, Musashi Hinck, Brandon M. Stewart, and Hanying Wei. 2023. “Using Imperfect Surrogates for Downstream Inference: Design-Based Supervised Learning for Social Science Applications of Large Language Models.” Advances in Neural Information Processing Systems 36.
Egami, Naoki, Musashi Hinck, Brandon M. Stewart, and Hanying Wei. 2024. “Using Large Language Model Annotations for the Social Sciences: A General Framework of Using Predicted Variables in Downstream Analyses.” American Journal of Political Science (conditionally accepted).
Maranca, Alessandra Rister Portinari, Jihoon Chung, Musashi Hinck, Adam D. Wolsky, Naoki Egami, and Brandon M. Stewart. 2025. “Correcting the Measurement Errors of AI-Assisted Labeling in Image Analysis Using Design-Based Supervised Learning.” Sociological Methods & Research 54(3): 984-1016. https://doi.org/10.1177/00491241251333372