Calculate confidence intervals (CIs) for MICE and associated metrics using bootstrap sampling and the percentile method.

Confidence intervals are estimated for overall accuracy, MICE, and all class-aggregated, macro-averaged metrics produced by mice() or miceCM(). Returns metric name, mean metric value, median metric value, lower confidence interval bounds (low.ci), and upper confidence interval bounds (upper.ci) as a dataframe object.

Usage

miceCI(
  reps = 200,
  frac = 0.7,
  lowPercentile,
  highPercentile,
  reference,
  prediction,
  mappings = levels(as.factor(reference)),
  multiclass = TRUE,
  positiveIndex = 1
)

Arguments

reps: number of bootstrap replicates to use. Default is 200.
frac: proportion of samples to include in each bootstrap sample. Default is 0.7.
lowPercentile: lower percentile for confidence interval. Default is 0.025 for a 95% CI.
highPercentile: upper percentile for confidence interval. Default is 0.975 for a 95% CI.
reference: column of reference labels as factor data type.
prediction: column of predicted labels as factor data type.
mappings: names of classes (if not provided, factor levels are used).
multiclass: TRUE or FALSE. If TRUE, treats classification as multiclass. If FALSE, treats classification as binary. Default is TRUE.
positiveIndex: index for positive case for binary classification. Ignored for multiclass classification. Default is 1 or first factor level

Value

dataframe object of metric name and estimated mean value, median value, and lower and upper CIs.

Examples

#Multiclass example
data(mcData)
ciResultsMC <- miceCI(rep=1000,
frac=.7,
mcData$ref,
mcData$pred,
lowPercentile=0.025,
highPercentile=0.975,
mappings=c("Barren", "Forest", "Impervious", "Low Vegetation", "Mixed Dev", "Water"),
multiclass=TRUE)

print(ciResultsMC)
#>            metric      mean    median    low.ci   high.ci
#> 1 overallAccuracy 0.9367192 0.9366989 0.9333093 0.9403142
#> 2            MICE 0.7650484 0.7649787 0.7519007 0.7784874
#> 3         macroPA 0.7390639 0.7387886 0.7187456 0.7601615
#> 4     macroRTBUCE 0.6968798 0.6965937 0.6759870 0.7178261
#> 5         macroUA 0.6626562 0.6623159 0.6415482 0.6843171
#> 6     macroCTBICE 0.6443924 0.6441264 0.6234891 0.6669481
#> 7         macroF1 0.6987446 0.6986543 0.6797916 0.7169005
#> 8 macroF1Efficacy 0.6695788 0.6695257 0.6503167 0.6886411

#Binary example
data(biData)
ciResultsBi <- miceCI(rep=1000,
frac=.7,
biData$ref,
biData$pred,
lowPercentile=0.025,
highPercentile=0.975,
mappings = c("Mined", "Not Mined"),
multiclass=FALSE,
positiveIndex=1)

print(ciResultsBi)
#>                   metric      mean    median    low.ci   high.ci
#> 1        overallAccuracy 0.9955643 0.9957143 0.9934286 0.9974286
#> 2                   MICE 0.9282719 0.9297299 0.8885995 0.9608437
#> 3              Precision 0.8874233 0.8879999 0.8299764 0.9355010
#> 4      precisionEfficacy 0.8837033 0.8843115 0.8252965 0.9332810
#> 5                    NPV 0.8874233 0.8879999 0.8299764 0.9355010
#> 6            npvEfficacy 0.8837033 0.8843115 0.8252965 0.9332810
#> 7                 Recall 0.9875464 0.9906541 0.9618955 0.9999999
#> 8         recallEfficacy 0.9871329 0.9903593 0.9607169 0.9999999
#> 9            Specificity 0.9875464 0.9906541 0.9618955 0.9999999
#> 10 specificityEfficicacy 0.9871329 0.9903593 0.9607169 0.9999999
#> 11               f1Score 0.9345581 0.9356222 0.9010832 0.9632712
#> 12       f1ScoreEfficacy 0.9322891 0.9334281 0.8974229 0.9619442