Calculate confidence intervals (CIs) for MICE and associated metrics using bootstrap sampling and the percentile method.
Arguments
- reps
number of bootstrap replicates to use. Default is 200.
- lowPercentile
lower percentile for confidence interval. Default is 0.025 for a 95% CI.
- highPercentile
upper percentile for confidence interval. Default is 0.975 for a 95% CI.
- reference
column of reference labels as factor data type.
- prediction
column of predicted labels as factor data type.
- mappings
names of classes (if not provided, factor levels are used).
- multiclass
TRUE or FALSE. If TRUE, treats classification as multiclass. If FALSE, treats classification as binary. Default is TRUE.
- positiveIndex
index for positive case for binary classification. Ignored for multiclass classification. Default is 1 or first factor level
Value
dataframe object of metric name and estimated mean value, median value, and lower and upper CIs.
Details
Confidence intervals are estimated for overall accuracy, MICE, and all class-aggregated, macro-averaged metrics produced by mice() or miceCM(). Returns metric name, mean metric value, median metric value, lower confidence interval bounds (low.ci), and upper confidence interval bounds (upper.ci) as a dataframe object.
Examples
#Multiclass example
data(mcData)
ciResultsMC <- miceCI(rep=100,
mcData$ref,
mcData$pred,
lowPercentile=0.025,
highPercentile=0.975,
mappings=c("Barren", "Forest", "Impervious", "Low Vegetation", "Mixed Dev", "Water"),
multiclass=TRUE)
print(ciResultsMC)
#> metric mean median low.ci high.ci
#> 1 overallAccuracy 0.9364207 0.9366551 0.9341430 0.9385327
#> 2 MICE 0.7931729 0.7939025 0.7859287 0.8004718
#> 3 macroPA 0.6619488 0.6617183 0.6453769 0.6799644
#> 4 macroRTBUCE 0.6437077 0.6432955 0.6273163 0.6619592
#> 5 macroUA 0.7386979 0.7390663 0.7231347 0.7544900
#> 6 macroCTBICE 0.7018734 0.7023724 0.6860901 0.7174588
#> 7 macroF1 0.6981951 0.6984726 0.6836253 0.7140613
#> 8 macroF1Efficacy 0.6715089 0.6716198 0.6569860 0.6879459
#Binary example
data(biData)
ciResultsBi <- miceCI(rep=100,
biData$ref,
biData$pred,
lowPercentile=0.025,
highPercentile=0.975,
mappings = c("Mined", "Not Mined"),
multiclass=FALSE,
positiveIndex=1)
print(ciResultsBi)
#> metric mean median low.ci high.ci
#> 1 overallAccuracy 0.9954000 0.9956000 0.9932950 0.9971050
#> 2 MICE 0.9329272 0.9345119 0.9059960 0.9560698
#> 3 Precision 0.9873428 0.9874606 0.9649008 0.9999999
#> 4 precisionEfficacy 0.9868788 0.9870071 0.9635442 0.9999999
#> 5 NPV 0.9873428 0.9874606 0.9649008 0.9999999
#> 6 npvEfficacy 0.9868788 0.9870071 0.9635442 0.9999999
#> 7 Recall 0.8820378 0.8861780 0.8279788 0.9237998
#> 8 recallEfficacy 0.8776828 0.8818158 0.8216868 0.9211071
#> 9 Specificity 0.8820378 0.8861780 0.8279788 0.9237998
#> 10 specificityEfficicacy 0.8776828 0.8818158 0.8216868 0.9211071
#> 11 f1Score 0.9315243 0.9337262 0.9019917 0.9561630
#> 12 f1ScoreEfficacy 0.9288655 0.9311267 0.8976934 0.9545300