Designing Customer Feedback for Subscription Boxes

1 Introduction

• The design problem balances scarce responses across candidate items and questionnaire formats, which affect both response counts and information per response.The platform chooses which items to ask about, how many questions to ask each customer, and how fine a rating scale to use.Source: §1 · p. 2
• Feedback is most valuable for items whose position around the selection cutoff is uncertain, rather than simply for the most uncertain items.Uncertainty-only sampling does no better than an even spread in the experiments.Source: §1 · p. 3
• 15 to 81 percent: with a correct prior, the envelope rule lowers decision loss relative to an even spread, chiefly by favoring items near the cutoff.The rule values responses in blocks, addressing the slow initial growth in value for items far from the cutoff.Source: §1 · p. 3
• Within about 2 percent of the best on average: choosing formats by expected information per customer asked performs nearly as well in the experiments.Under the proxy and optimal response allocation, formats are ranked exactly by this single measure.Source: §1 · p. 3
• Within about 7 percent: measuring rating information at the cutoff reproduces ordinal decision loss for 1 to 100 responses per item.A prior-averaged measure can misstate decision loss by about half.Source: §1 · p. 3

1.1 Related Literature

• The paper extends decision-focused information measurement to top-K feedback design and combines questionnaire length, completion, and rating information in format comparisons.The related literature addresses sampling or measurement precision, while this work studies feedback collection for a group-level selection.Source: §1.1 · pp. 4–5

2 Model and Optimization Problem

• The model represents one round in which the platform sends a questionnaire and uses returned ratings to fill a box with candidate items.The box contains K of L candidates for a target customer population.Source: §2 · p. 5, §2.1 · p. 5

2.1 Item values

• Each candidate item's value is its average utility across the target population, and the platform begins with Gaussian prior beliefs that can capture relationships among items.Prior means can reflect product attributes or experience with similar items; covariance captures how ratings of related items update beliefs.Source: §2.1 · p. 5

2.2 Ordinal customer feedback

• Customers report ordered categories rather than utilities, with an ordered-probit model assigning ratings according to thresholds.The thresholds are treated as known and can be estimated from past ratings on the same scale; rating information varies with the item's value relative to those thresholds.Source: §2.2 · p. 6

2.3 Feedback collection and allocation

• A feedback format specifies the number of items each customer rates and the response scale, while the allocation determines how many completed responses each item receives.Expected responses depend on the number of customers shown the questionnaire, its length, and its completion rate; the design is fixed before answers arrive.Source: §2.3 · pp. 6–7

2.4 The top-K decision and decision loss

• The decision loss is the expected value the selected K-item set falls short of the oracle set chosen with knowledge of true item values.After observing ratings, the platform selects the K items with the highest estimated values.Source: §2.4 · p. 7

2.5 The joint feedback-design problem

• For a fixed customer pool, the platform jointly chooses a format and response allocation to minimize decision loss, subject to the format's response budget.Because ordinal loss has no closed form, the paper uses a Gaussian proxy to plan allocations for each format, then compares formats.Source: §2.5 · pp. 7–8

3.1 The proxy

• The proxy replaces each ordinal rating with an information-equivalent Gaussian signal, making decision loss depend on the design through a covariance matrix.The signal is centered at the item value, with variance set to the inverse of its information; the resulting proxy loss depends on the design through A(N; S).Source: §3 · p. 8, §3.1 · p. 8

3.2 Where to measure the information

• Information should be measured at the selection cutoff because errors matter when they move an item across it; under the stated independent-prior conditions, the loss ratio tends to one exactly when proxy information equals κ(S).The limiting result assumes equal responses per item and the prior-variance condition specified in Proposition 1.Source: §3.2 · pp. 9–10

3.3 How well the proxy reproduces the ordinal model

• The per-item prior average tracks estimation error better in most settings, but both proxy choices understate that error with many responses because items far from thresholds dominate the average.Ratings of those far-from-threshold items provide almost no information.Source: §3.3 · pp. 10–11
• The cutoff-information proxy’s decision-loss ratio is within 6 percent of one for 10 to 100 responses per item and ranges from 0.92 to 1.13 across six budgets up to 3,000.At 10 responses per item, the ratio is 0.97 to 1.04; at 1, 2, or 5 responses per item, it is 0.93 to 1.01.Source: Table 2 · p. 11, §3.3 · p. 11
• Using a per-item prior average can understate loss by about 50 percent with binary ratings or 20 percent with five-point ratings when cutoff values lie above the thresholds, and overstate it by 11 to 57 percent when cutoffs lie near thresholds.The direction and size of the error depend on the cutoff’s position relative to rating thresholds.Source: §3.3 · pp. 10–11
• The proxy’s format rankings diverge most when cutoffs lie far from rating thresholds, although planning rankings under proxy and ordinal losses agree about as closely as independent ordinal simulations.Within a single replication, the proxy is weaker: for binary ratings and 10 responses per item, the models share 80 percent of selected items on average even though expected losses agree.Source: §3.3 · p. 12

4 Allocating Feedback Across Items

• The format fixes the total response budget and information per response before the platform plans an allocation from its prior.The questionnaire is sent once, although the analysis also allows information per response to differ across items.Source: §4 · p. 12

4.1 Why the top-K boundary matters

• Items whose estimated-value intervals straddle the prior cutoff are contested, and errors far from that boundary can leave the top-K selection unchanged.If every estimation error is smaller than half the gap between the Kth and (K+1)th true values, the selected set is stable.Source: §4.1 · pp. 12–13, Figure 1 · p. 13

4.2 How the decision loss splits across items

• Near misses around an item’s cutoff create expected decision costs proportional to the estimate’s cutoff density and residual uncertainty.The cutoff weight captures how likely an item is to lie near the selection boundary.Source: §4.2 · pp. 13–14

4.3 The value of one more response

• A response’s marginal value is the local cutoff weight times the residual variance it removes, so ranking items by uncertainty alone can misallocate feedback.The uncertainty-only rule does about as well as uniform allocation in the reported uneven-prior and MovieLens experiments.Source: §4.3 · p. 14

4.4 The value of a block of responses

• For an item facing a fixed cutoff, marginal response value rises while κN_j < τ_j x(z_j), then falls, making its total response value S-shaped.The peak occurs later for items farther from the cutoff; for distances of one and two prior standard deviations, the peak information ratios are about 0.5 and 1.4 times prior information.Source: §4.4 · p. 15

4.5 Planning the allocation

• The marginal-value rule can under-explore distant items, leaving most unanswered and performing about 10 percent worse than uniform allocation in one reported setting.This occurs with prior variance 0.3 and 300 responses per item.Source: §4.5 · p. 16
• The envelope rule allocates responses by each item’s concave-envelope gains and, with fixed cutoffs, falls short of the best plan by at most one item’s total response gain.The rule values distant items in blocks, avoiding the myopic rule’s tendency to leave them unexplored.Source: §4.5 · pp. 16–17

4.6 What planning can gain

• With a large budget, optimal response shares are proportional to the square roots of items’ cutoff weights, and planning gains most when few items are contested.The effective number of contested items is computed from the prior and approximates the envelope rule’s loss ratio within 0.03 at 300 responses per item.Source: §4.6 · p. 18

5 Choosing the Feedback Format

• EIS can rank formats without planning a separate allocation for each, and the envelope rule’s loss also depends on format only through EIS.Source: §5 · p. 18, T EIS(Q, S) · p. 20

5.1 Factorized information across response scales

• Changing rating scales multiplies information by the same factor for every item, allowing formats to be compared apart from their allocation.This factorization would not support separate format and allocation choices if it differed across items.Source: §5.1 · p. 19

5.2 Effective information supply

• EIS is expected completed responses per customer asked, Q α(Q, S), multiplied by information per response relative to the baseline, β(S).It measures information at the cutoff between selected and unselected items.Source: §5.2 · p. 19

5.3 When EIS ranks formats

• Under the Gaussian proxy and continuous relaxation, the format with the largest EIS is optimal when allocation is optimized.Source: §5.3 · pp. 19–20, T EIS(Q, S) · p. 20
• Under the ordinal model, choosing by EIS comes within 3 percent of the best format with whole responses and a per-item response limit.Source: T EIS(Q, S) · p. 20

5.4 Calibrating candidate formats

• To estimate EIS, the platform estimates completion from randomized format exposure and submission logs, estimates relative information from fitted ordinal models, then checks leading formats by simulation.Source: §5.4 · p. 20
• On the MovieLens calibration, a five-point rating carries 1.73 times the information of a like/dislike answer at the stated cutoff.The estimate ranges from 1.83 for 5 movies to 1.63 for 20; scale information depends on threshold placement relative to the cutoff as well as category count.Source: §5.4 · p. 20
• The five-point formats have higher EIS in the survey scenario (ratio 1.38) but lower EIS in the in-product scenario (ratio 0.58).These comparisons use assumed completion rates; the authors report no public randomized experiment varying both questionnaire length and scale.Source: §5.4 · p. 21, Table 3 · p. 21

6.1 How well the proxy ranks designs

• For allocations, proxy and ordinal rankings have correlation 0.89 to 0.93 and regret below 2 percent; for formats, ranking correlation falls from 0.92 under mild to 0.74 under steep completion loss.The proxy is within 0.02 of the ordinal-simulation benchmark in rank correlation and within 0.7 percentage points in regret.Source: §6.1 · p. 22, Table 4 · p. 23

6.2 Planning the questionnaire

• The envelope rule has the lowest or tied-lowest loss in all 72 settings, reducing loss by 56 to 81 percent at prior variance 0.1, about half at 0.3, and 15 to 43 percent at 1.Its advantage over the square-root rule is at most 4 percent of uniform loss with 10 responses per item and is not significant with 300.Source: §6.2 · p. 23, Figure 3 · p. 24

6.3 Which items each plan asks about

• The envelope rule asks about more items than the marginal-value rule in the illustrated instance: 34 versus 16 of 60, with proxy losses of 0.026 versus 0.068.Across instances, it leaves 15 items unanswered on average, compared with 35 for the marginal-value rule; the unanswered items are those whose place is already clear from the prior.Source: §6.3 · p. 25

6.4 Choosing the format

• For format choice, the format with the largest expected information supply is the best in 67 to 82 percent of settings and stays within 3 percent of the best when the one-response-per-customer limit is imposed.Without that limit, choosing by expected information supply comes within 2 percent of the best format; other rules have 14 to 88 percent higher loss.Source: §6.4 · pp. 25–26

6.5 Relaxing the assumptions

• With correlated items, the envelope rule lowers loss relative to uniform allocation by 42 to 58 percent at prior variance 0.3 and 15 to 35 percent at 1.Planning with the full covariance changes loss relative to uniform allocation by −0.01 to +0.13; under uneven prior information, the marginal-value rule does up to 87 percent worse than uniform allocation.Source: §6.5 · pp. 26–27
• With a correct prior, two questionnaire waves lower decision loss by 38 to 54 percent and four waves by 49 to 69 percent relative to one wave.With a wrong prior and 300 responses per item, the envelope rule’s loss reaches up to 7.7 times uniform loss; widening the prior and sending waves reduce this risk.Source: §6.5 · p. 27

6.6 MovieLens ratings

• 6 to 28 percent: On MovieLens ratings, the envelope, minimum-share, and marginal-value rules lower decision loss relative to uniform allocation.The envelope rule outperforms the square-root rule by 4 to 8 percent of uniform loss at every budget.Source: Table 6 · p. 28, §6.6 · p. 28

7 Conclusion

• Responses matter most for uncertain items likely to be near the selection cutoff; the envelope rule values responses in blocks because their value grows slowly at first and faster later.A plan built on an overconfident prior may omit items it wrongly places far from the cutoff.Source: §7 · p. 28
• Formats can be compared by expected information at the cutoff per customer asked, and a finer scale is better only when its additional information outweighs its lower completion rate.The completion rates in the format comparison are assumed; a randomized experiment would measure them.Source: §7 · pp. 28–29

Conditions of Proposition 1

• The proof assumes independent item priors and that the k least precise items’ prior precisions sum to more than one.This condition keeps the ordinal limit finite; bounded prior variances alone do not ensure finiteness under correlated priors.Source: A Details and Proofs for Section 3 · p. 33

Lemma 9 (Loss decomposition by item)

• Under independent priors, expected selection loss decomposes into item-level expected near-miss costs at cutoffs independent of each item’s value and estimate.The same decomposition applies to the Gaussian proxy, using its proxy estimates, cutoffs, and near-miss costs.Source: A Details and Proofs for Section 3 · p. 34

Lemma 10 (Monotonicity of the estimate)

• Posterior means are nondecreasing in each response and, with deviations fixed, in the item value; consequently, the probabilities of estimates falling below or above a cutoff move monotonically with value.Source: A Details and Proofs for Section 3 · pp. 34–35

Lemmas 11–12

• At a fixed cutoff c, n times the ordinal near-miss cost tends to f(c)/(2I_S(c)), while the proxy limit is f(c)/(2κ); the proxy costs are uniformly bounded after multiplication by n.The ordinal estimate also has uniform local normality on bounded intervals, which supports the fixed-cutoff limit.Source: A Details and Proofs for Section 3 · p. 36

Proof of Proposition 1

• Under (A1)–(A2), n times the ordinal and proxy losses converge respectively to Σ_i E[ψ_i/(2I_S(ζ_i))] and Σ_i E[ψ_i]/(2κ_i); their ratio tends to one exactly when these limits match.For a common proxy information κ0, the limits match if and only if κ0 equals the corresponding expected-information expression.Source: A Details and Proofs for Section 3 · p. 38

Lemma 13 and Proof of Proposition 1

• For the proxy, n E[˜gi,n(˜ζi)] → E[ψi]/(2κi); finiteness follows from n gi,n ≤ Hi and fi/(2IS) ≤ Hi in the limit, with Step 3’s tail bound also holding for ζi.Source: A Details and Proofs for Section 3 · p. 38

Allocations with fixed shares

• With fixed positive response-budget shares per item, the proxy-to-ordinal loss ratio tends to a weighted average of item-specific limits, bounded by their smallest and largest values relative to common κ(S).The result follows as each item's response count grows without bound while its share of the total budget converges to a positive value.Source: A Details and Proofs for Section 3 · pp. 38–39

Other budgets

• At 30, 300, and 3,000 responses per item, the proxy-to-ordinal loss ratio is 0.92–1.11 using cutoff information, versus 0.41–1.57 using per-item prior averages.Standard errors are at most 0.11 and 0.14, respectively.Source: A Details and Proofs for Section 3 · p. 39

Correlated items

• With item correlation ρ = 0.6, ∆/∆ord ranges from 0.89–1.10 using cutoff information and 0.44–1.60 using per-item prior averages.These ranges hold from 10 to 1,000 responses per item across every configuration and scale.Source: A Details and Proofs for Section 3 · p. 39

Item-level information at the cutoff

• Across Section 6 prior instances, item-level and pooled cutoff information differ on average by at most 3.3% for binary ratings and 1.5% for five-point ratings.The difference is at most 1.8% for the five-point-to-binary value ratio used in Section 5; similar cutoffs across items explain the close values.Source: A Details and Proofs for Section 3 · p. 39

• At zero responses, the local cutoff weight is bj = 1/2 qj(µj), and is small when the prior mean lies far from likely cutoff values.Source: B Details and Proofs for Section 4 · p. 42
• For zj > 0, bj tends to zero as information u tends to zero, then its percentage growth rate initially outpaces the decline in the other marginal-value factor.As responses grow, bj tends to half the prior density at the cutoff; with a random cutoff, fixed-cutoff values are averaged over that cutoff.Source: B Details and Proofs for Section 4 · p. 42
• The envelope rule assigns responses by descending increments of concave-envelope value curves, with at most one item ending strictly inside a linear piece.The actual value curve equals its envelope at each piece’s endpoints.Source: B Details and Proofs for Section 4 · p. 43
• As Ntot grows, the square-root rule’s decision loss divided by the optimal loss tends to 1.The rule uses one number per item and requires no value curves.Source: B Details and Proofs for Section 4 · p. 43
• Decision loss falls like 1/Ntot because prior-averaged loss is dominated by cases where the Kth- and (K + 1)th-ranked items nearly tie.For fixed true values, loss would fall exponentially.Source: B Details and Proofs for Section 4 · p. 44

• The mapping N → β(S)N is a bijection to baseline-scale allocations, and corresponding allocations have the same decision loss under formats S and S0.Source: C Proofs for Section 5 · p. 45
• Rescaling response counts by β(S) lets the format comparison be expressed through baseline-scale allocations.Source: C Proofs for Section 5 · p. 45
• The proof’s equivalence is allocation-specific: it compares (N, S) with the mapped allocation under S0.Source: C Proofs for Section 5 · p. 45

• The function Ψ is format-independent and nonincreasing as its baseline-scale allocation budget grows.Source: T EIS(Q, S) · p. 45
• In the continuous relaxation, the envelope rule traces the same baseline-equivalent allocation path under every scale and stops at T EIS(Q, S).Source: T EIS(Q, S) · p. 45
• Under the continuous relaxation, decision loss depends on the format only through T EIS(Q, S).Source: T EIS(Q, S) · p. 45

D Additional Results for Section 6

• In 17 of 18 configuration, scale, and budget combinations, the envelope rule had the lowest ordinal loss among twelve designs.For envelope-rule allocation, proxy decision loss divided by ordinal loss ranged from 0.99 to 1.04; for the square-root rule, it ranged from 0.98 to 1.02.Source: D Additional Results for Section 6 · p. 45
• For uniform allocation, the standard error of a paired difference was at most 0.07 times the uniform loss.This bound is reported for Figure 3; the quoted 15 to 81 percent range averages across three configurations and two scales.Source: D Additional Results for Section 6 · p. 45
• None of the envelope rule’s loss differences versus the square-root, marginal-value, minimum-share, or OCBA-m rules was significantly above zero.Table entries report each difference divided by uniform loss; the passage does not provide individual values.Source: D Additional Results for Section 6 · p. 45, Table 7 · p. 46

0.3 (random configuration, binary ratings, 30 and 100 responses per item, 20 instances with 10

• With prior variance 1, the envelope rule’s two-wave loss under four wrong priors was 0.46 to 0.78 relative to uniform allocation.The comparison uses uniform allocation with the prior as given as its baseline.Source: D Additional Results for Section 6 · p. 46

E Retention Associations in an Industrial Curation Log

• In about 52,000 orders placed between July and November 2025, separate logistic regressions estimated retention associations using three outcome labels.The analysis used standardized continuous features and reported odds ratios per one standard deviation.Source: E Retention Associations in an… · p. 47, Table 9 · p. 47
• The cross-order repeat rate measures the share of box items also present in the customer’s previous four orders, calculated overall and separately for recipe and occasional items.Other defined measures include premium intensity, within-box duplication density, and a duplication spike relative to the customer’s trailing mean.Source: E Retention Associations in an… · p. 47
• Repeat-rate associations varied by customer segment, and unmeasured engagement may still correlate with repeat rates and box economics despite conditioning on tenure.The authors use the observational estimates only as motivation.Source: E Retention Associations in an… · p. 47
• Repeated items were associated with lower retention, most strongly for repeated recipes; larger boxes were associated with higher retention and premium intensity with lower retention.The conclusions were the same under the three retention labels; these associations concern box contents, not item values.Source: E Retention Associations in an… · p. 47, Table 9 · p. 47