predictset

CRAN: predictset citation info Coverdale C (2026). predictset: Model-Agnostic Conformal Prediction for Classification and Regression . R package version 0.3.0, https://github.com/charlescoverdale/predictset . Corresponding BibTeX entry: @Manual{, title = {predictset: Model-Agnostic Conformal Prediction for Classification and Regression}, author = {Charles Coverdale}, year = {2026}, note = {R package version 0.3.0}, url = {https://github.com/charlescoverdale/predictset}, }

NEWS code{white-space: pre-wrap;} span.smallcaps{font-variant: small-caps;} span.underline{text-decoration: underline;} div.column{display: inline-block; vertical-align: top; width: 50%;} div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;} ul.task-list{list-style: none;} predictset 0.3.0 Documentation Documented Jackknife+ and CV+ theoretical coverage guarantee (1-2alpha) per Barber et al. (2021) Documented ACI asymptotic (not finite-sample) coverage guarantee per Gibbs and Candes (2021) Documented CQR dependence on quantile model quality Documented deterministic vs randomized APS variants Added coverage guarantee footnotes to README and vignette method tables Internal Added graphics and grDevices to DESCRIPTION Imports Added missing test dependencies to Suggests predictset 0.2.0 New features conformal_mondrian() and conformal_mondrian_class() for group-conditional (Mondrian) conformal prediction conformal_weighted() for weighted conformal prediction under covariate shift conformal_aci() for adaptive conformal inference (sequential prediction) conformal_pvalue() for conformal p-values conformal_compare() for benchmarking multiple methods side-by-side coverage_by_group() and coverage_by_bin() for conditional coverage diagnostics Progress bars via verbose = TRUE for conformal_jackknife() and conformal_cv() Improvements NA/NaN/Inf input validation with informative error messages Column dimension checks between training and test data Probability matrix column validation for classification methods Graceful handling of unseen factor levels in APS/RAPS/LAC scoring Negative scale model prediction warnings for normalized conformal predictset 0.1.0 Initial release with split conformal, CV+, Jackknife+, CQR (regression) and split, APS, RAPS, LAC (classification)

README code{white-space: pre-wrap;} span.smallcaps{font-variant: small-caps;} span.underline{text-decoration: underline;} div.column{display: inline-block; vertical-align: top; width: 50%;} div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;} ul.task-list{list-style: none;} pre > code.sourceCode { white-space: pre; position: relative; } pre > code.sourceCode > span { display: inline-block; line-height: 1.25; } pre > code.sourceCode > span:empty { height: 1.2em; } .sourceCode { overflow: visible; } code.sourceCode > span { color: inherit; text-decoration: inherit; } div.sourceCode { margin: 1em 0; } pre.sourceCode { margin: 0; } @media screen { div.sourceCode { overflow: auto; } } @media print { pre > code.sourceCode { white-space: pre-wrap; } pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; } } pre.numberSource code { counter-reset: source-line 0; } pre.numberSource code > span { position: relative; left: -4em; counter-increment: source-line; } pre.numberSource code > span > a:first-child::before { content: counter(source-line); position: relative; left: -1em; text-align: right; vertical-align: baseline; border: none; display: inline-block; -webkit-touch-callout: none; -webkit-user-select: none; -khtml-user-select: none; -moz-user-select: none; -ms-user-select: none; user-select: none; padding: 0 4px; width: 4em; color: #aaaaaa; } pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; } div.sourceCode { } @media screen { pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; } } code span.al { color: #ff0000; font-weight: bold; } /* Alert */ code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */ code span.at { color: #7d9029; } /* Attribute */ code span.bn { color: #40a070; } /* BaseN */ code span.bu { color: #008000; } /* BuiltIn */ code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */ code span.ch { color: #4070a0; } /* Char */ code span.cn { color: #880000; } /* Constant */ code span.co { color: #60a0b0; font-style: italic; } /* Comment */ code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */ code span.do { color: #ba2121; font-style: italic; } /* Documentation */ code span.dt { color: #902000; } /* DataType */ code span.dv { color: #40a070; } /* DecVal */ code span.er { color: #ff0000; font-weight: bold; } /* Error */ code span.ex { } /* Extension */ code span.fl { color: #40a070; } /* Float */ code span.fu { color: #06287e; } /* Function */ code span.im { color: #008000; font-weight: bold; } /* Import */ code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */ code span.kw { color: #007020; font-weight: bold; } /* Keyword */ code span.op { color: #666666; } /* Operator */ code span.ot { color: #007020; } /* Other */ code span.pp { color: #bc7a00; } /* Preprocessor */ code span.sc { color: #4070a0; } /* SpecialChar */ code span.ss { color: #bb6688; } /* SpecialString */ code span.st { color: #4070a0; } /* String */ code span.va { color: #19177c; } /* Variable */ code span.vs { color: #4070a0; } /* VerbatimString */ code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */ predictset predictset is an R package for model-agnostic conformal prediction and distribution-free uncertainty quantification. It constructs prediction intervals (regression) and prediction sets (classification) with finite-sample coverage guarantees - no distributional assumptions required. Works with any model: lm , glm , ranger , xgboost , or custom user-defined models via make_model() . Installation # Install from CRAN install.packages ( "predictset" ) # Or install the development version from GitHub # install.packages("devtools") devtools :: install_github ( "charlescoverdale/predictset" ) library (predictset) # Get 90% prediction intervals around any model - 3 lines of code result <- conformal_split (x, y, model = y ~ ., x_new = x_new, alpha = 0.10 ) result $ lower # lower bounds result $ upper # upper bounds What is conformal prediction? Standard machine learning models produce point predictions: a single number for regression, a single class for classification. But in practice, you almost always need to know how uncertain that prediction is. Conformal prediction is a framework for wrapping any model in a layer of calibrated uncertainty quantification. Given a target coverage level (say 90%), it produces prediction intervals or prediction sets that are guaranteed to contain the true value at least 90% of the time, regardless of the underlying model or data distribution. The key property is that this guarantee holds in finite samples. It’s not asymptotic, and it doesn’t require distributional assumptions. The only requirement is that the calibration data and test data are exchangeable (roughly: drawn from the same distribution). This makes conformal prediction fundamentally different from parametric confidence intervals, bootstrap intervals, or Bayesian credible intervals, all of which depend on modelling assumptions that may not hold. predictset implements the main conformal methods from the recent literature (split conformal, Jackknife+, CV+, conformalized quantile regression for regression, and APS, RAPS, and LAC for classification) in a lightweight package with only two dependencies ( cli and stats ). How does predictset compare to other packages? Feature predictset probably conformalInference MAPIE Language R R R Python Regression Yes Yes Yes Yes Classification Yes No No Yes Model-agnostic Yes tidymodels only Yes scikit-learn only On CRAN Pending Yes No (GitHub only) N/A Jackknife+ / CV+ Yes No Yes Yes CQR Yes Yes Yes Yes APS / RAPS Yes No No Yes Mondrian CP Yes No No Yes Weighted CP Yes No No Yes Adaptive CI Yes No No No Conditional diagnostics Yes No No Partial Dependencies 2 14+ 5 N/A Last updated 2026 2024 2019 2024 predictset is designed to complement rather than compete with probably . If you’re working in the tidymodels ecosystem and only need regression intervals, probably integrates neatly with your workflow. predictset fills the gaps: classification methods (APS, RAPS, LAC), Jackknife+/CV+ for regression, and a model-agnostic interface that works with any model, not just tidymodels workflows. conformalInference by Ryan Tibshirani was foundational research code, but it hasn’t been updated since 2019, isn’t on CRAN, and doesn’t cover classification. Quick start Regression: prediction intervals with coverage verification library (predictset) set.seed ( 42 ) n <- 500 x <- matrix ( rnorm (n * 5 ), ncol = 5 ) y <- x[, 1 ] * 2 + x[, 2 ] + rnorm (n) x_new <- matrix ( rnorm ( 100 * 5 ), ncol = 5 ) y_new <- x_new[, 1 ] * 2 + x_new[, 2 ] + rnorm ( 100 ) # true values (for evaluation) # Fit conformal intervals result <- conformal_split (x, y, model = y ~ ., x_new = x_new, alpha = 0.10 ) print (result) #> ── Conformal Prediction Intervals (Split Conformal) ── #> • Coverage target: "90%" #> • Training: 250 | Calibration: 250 | Predictions: 100 #> • Conformal quantile: 1.876 #> • Median interval width: 3.7519 # Extract intervals as a data frame head ( data.frame ( pred = result $ pred, lower = result $ lower, upper = result $ upper)) # Verify coverage: should be ≥ 90% coverage (result, y_new) # Predict on new data later (fit once, predict many times) future_data <- matrix ( rnorm ( 50 * 5 ), ncol = 5 ) predict (result, newdata = future_data) Classification: prediction sets set.seed ( 42 ) n <- 400 x <- matrix ( rnorm (n * 4 ), ncol = 4 ) y <- factor ( ifelse (x[, 1 ] + x[, 2 ] > 0 , "A" , "B" )) x_new <- matrix ( rnorm ( 50 * 4 ), ncol = 4 ) clf <- make_model ( train_fun = function (x, y) glm (y ~ ., data = data.frame ( y = y, x), family = "binomial" ), predict_fun = function (object, x_new) { df <- as.data.frame (x_new) names (df) <- paste0 ( "X" , seq_len ( ncol (x_new))) p <- predict (object, newdata = df,

Help for package predictset const macros = { "\\R": "\\textsf{R}", "\\mbox": "\\text", "\\code": "\\texttt"}; function processMathHTML() { var l = document.getElementsByClassName('reqn'); for (let e of l) { katex.render(e.textContent, e, { throwOnError: false, macros }); } return; } Package {predictset} Contents predictset-package conformal_aci conformal_aps conformal_class_split conformal_compare conformal_cqr conformal_cv conformal_jackknife conformal_lac conformal_mondrian conformal_mondrian_class conformal_pvalue conformal_raps conformal_split conformal_weighted coverage coverage_by_bin coverage_by_group interval_width make_model plot.predictset_aci plot.predictset_class plot.predictset_reg predict.predictset_class predict.predictset_reg print.predictset_aci print.predictset_class print.predictset_model print.predictset_reg set_size summary.predictset_class summary.predictset_reg Title: Conformal Prediction and Uncertainty Quantification Version: 0.3.0 Description: Implements conformal prediction methods for constructing prediction intervals (regression) and prediction sets (classification) with finite-sample coverage guarantees. Methods include split conformal, 'CV+' and 'Jackknife+' (Barber et al. 2021) < doi:10.1214/20-AOS1965 >, 'Conformalized Quantile Regression' (Romano et al. 2019) < doi:10.48550/arXiv.1905.03222 >, 'Adaptive Prediction Sets' (Romano, Sesia, Candes 2020) < doi:10.48550/arXiv.2006.02544 >, 'Regularized Adaptive Prediction Sets' (Angelopoulos et al. 2021) < doi:10.48550/arXiv.2009.14193 >, Mondrian conformal prediction for group-conditional coverage (Vovk et al. 2005), weighted conformal prediction for covariate shift (Tibshirani et al. 2019), and adaptive conformal inference for sequential prediction (Gibbs and Candes 2021). All methods are distribution-free and provide calibrated uncertainty quantification without parametric assumptions. Works with any model that can produce predictions from new data, including 'lm', 'glm', 'ranger', 'xgboost', and custom user-defined models. License: MIT + file LICENSE Encoding: UTF-8 Language: en-US URL: https://github.com/charlescoverdale/predictset BugReports: https://github.com/charlescoverdale/predictset/issues RoxygenNote: 7.3.3 Depends: R (≥ 4.1.0) Imports: cli (≥ 3.6.0), grDevices, graphics, stats Suggests: testthat (≥ 3.0.0), ranger, ggplot2, knitr, rmarkdown, parsnip (≥ 1.0.0), probably, rsample, workflows VignetteBuilder: knitr Config/testthat/edition: 3 NeedsCompilation: no Packaged: 2026-03-14 17:10:13 UTC; charlescoverdale Author: Charles Coverdale [aut, cre, cph] Maintainer: Charles Coverdale <charlesfcoverdale@gmail.com> Repository: CRAN Date/Publication: 2026-03-19 14:50:15 UTC predictset: Conformal Prediction and Uncertainty Quantification Description Implements conformal prediction methods for constructing prediction intervals (regression) and prediction sets (classification) with finite-sample coverage guarantees. Methods include split conformal, 'CV+' and 'Jackknife+' (Barber et al. 2021) doi:10.1214/20-AOS1965 , 'Conformalized Quantile Regression' (Romano et al. 2019) doi:10.48550/arXiv.1905.03222 , 'Adaptive Prediction Sets' (Romano, Sesia, Candes 2020) doi:10.48550/arXiv.2006.02544 , 'Regularized Adaptive Prediction Sets' (Angelopoulos et al. 2021) doi:10.48550/arXiv.2009.14193 , Mondrian conformal prediction for group-conditional coverage (Vovk et al. 2005), weighted conformal prediction for covariate shift (Tibshirani et al. 2019), and adaptive conformal inference for sequential prediction (Gibbs and Candes 2021). All methods are distribution-free and provide calibrated uncertainty quantification without parametric assumptions. Works with any model that can produce predictions from new data, including 'lm', 'glm', 'ranger', 'xgboost', and custom user-defined models. Author(s) Maintainer : Charles Coverdale charlesfcoverdale@gmail.com [copyright holder] See Also Useful links: https://github.com/charlescoverdale/predictset Report bugs at https://github.com/charlescoverdale/predictset/issues Adaptive Conformal Inference Description Implements basic Adaptive Conformal Inference (ACI) for sequential prediction. The miscoverage level alpha is adjusted online based on whether previous predictions covered the true values, maintaining long-run coverage even under distribution shift. Usage conformal_aci(y_pred, y_true, alpha = 0.1, gamma = 0.005) Arguments y_pred A numeric vector of point predictions (sequential). y_true A numeric vector of true values (sequential). alpha Target miscoverage level. Default 0.10 . gamma Learning rate for alpha adjustment. Default 0.005 . Larger values adapt faster but are less stable. Details ACI provides asymptotic coverage guarantees under distribution drift, not the finite-sample guarantees of split conformal prediction. The long-run average coverage converges to 1 - \alpha as the sequence length grows (Gibbs and Candes, 2021). Value A list with components: lower Numeric vector of lower bounds. upper Numeric vector of upper bounds. covered Logical vector indicating whether each interval covered the true value. alphas Numeric vector of the adapted alpha values at each step. coverage Overall empirical coverage. References Gibbs, I. and Candes, E. (2021). Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems , 34. See Also Other regression methods: conformal_cqr () , conformal_cv () , conformal_jackknife () , conformal_mondrian () , conformal_split () , conformal_weighted () Examples set.seed(42) n <- 200 y_true <- cumsum(rnorm(n, sd = 0.1)) + rnorm(n) y_pred <- c(0, y_true[-n]) # naive lag-1 prediction result <- conformal_aci(y_pred, y_true, alpha = 0.10, gamma = 0.01) print(result$coverage) Adaptive Prediction Sets Description Constructs prediction sets using the Adaptive Prediction Sets (APS) method of Romano, Sesia, and Candes (2020). Classes are included in order of decreasing predicted probability until the cumulative probability exceeds the conformal threshold. Usage conformal_aps( x, y, model, x_new, alpha = 0.1, cal_fraction = 0.5, randomize = FALSE, seed = NULL ) Arguments x A numeric matrix or data frame of predictor variables. y A factor (or character/integer vector coerced to factor) of class labels. model A make_model() specification with type = "classification" , or a fitted model object that produces class probabilities. x_new A numeric matrix or data frame of new predictor variables. alpha Miscoverage level. Default 0.10 gives 90 percent prediction sets. cal_fraction Fraction of data used for calibration. Default 0.5 . randomize Logical. If TRUE , uses randomized scores for exact coverage (but prediction sets become stochastic). Default FALSE . seed Optional random seed. Details When randomize = FALSE (the default), this implementation uses the deterministic variant of APS, which provides conservative coverage (at least 1 - \alpha ). The randomized variant ( randomize = TRUE ) achieves exact 1 - \alpha coverage but produces non-reproducible prediction sets. Value A predictset_class object. See conformal_lac() for details. The method component is "aps" . References Romano, Y., Sesia, M. and Candes, E.J. (2020). Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems , 33. doi:10.48550/arXiv.2006.02544 See Also Other classification methods: conformal_lac () , conformal_mondrian_class () , conformal_raps () Examples set.seed(42) n <- 300 x <- matrix(rnorm(n * 4), ncol = 4) y <- factor(sample(c("A", "B", "C"), n, replace = TRUE)) x_new <- matrix(rnorm(50 * 4), ncol = 4) clf <- make_model( train_fun = function(x, y) glm(y ~ ., data = data.frame(y = y, x), family = "binomial"), predict_fun = function(object, x_new) { df <- as.data.frame(x_new) names(df) <- paste0("X", seq_len(ncol(x_new))) p <- predict(object, newdata = df, type = "response") cbind(A = p / 2, B = p / 2, C = 1 - p) }, type = "classification" ) result <- conformal