Standardised Classification of Effects Based on Their Uncertainty

A Simple, Stable and Transparent Classification of Effects

The effectclass package provides functions to classify effects in a stable and transparent manner. We base the classification on the lower and upper bounds of a confidence (or credible) interval, focussing on the uncertainty rather than the point estimates. First, we compare the confidence interval with a reference. This reference typically refers to the null hypothesis. An effect with confidence interval above the reference is a significant (important) effect. Let’s call this a positive effect. Likewise, a negative effect has a confidence interval below the reference.

Comparing confidence intervals to the reference.

But what if the confidence interval contains the reference? Then the width of the confidence interval plays an important role. Suppose the confidence interval is wide. The value of the effect can range from a strong negative effect over no effect to a strong positive effect. Basically, we have no information on the effect. On the other hand, suppose we have a narrow confidence interval. The effect still is either positive or negative, but we know that it won’t be a strong positive nor strong negative effect. We could define a lower¹ and upper² threshold and use them to separate confidence intervals that contain the reference. If the confidence interval is between both the upper and lower threshold, we can state that the effect does not exceed these thresholds. We set the thresholds at a level where we can consider the effect to be not relevant so we can call this no effect. As soon as the confidence interval contains the reference and at least one of the thresholds, we consider it an unknown effect.

Comparing confidence intervals to the reference and thresholds.

Comparing the confidence interval with a reference and two thresholds, gives us four distinct categories. The table below lists the rules for each category and suggests a symbol and standardised wording for effects and trends.

A Detailed Classification

The upper threshold divides the zone of a positive effect. Suppose the confidence interval is between the reference and the upper threshold. The confidence interval is above the reference so there is a positive effect. At the same time, the confidence interval is also below the upper threshold which would lead to no effect if the confidence interval contains the reference. We suggest the term moderate positive effect for this situation. A confidence interval above the upper threshold is outside of the no effect zone. We suggest the term strong positive effect for this situation. We keep the term positive effect when the confidence interval is above the reference and contains the upper threshold.

Comparing confidence intervals to the reference and thresholds for positive effects.

We split the negative effect in a strong negative effect, moderate negative effect and negative effect using analogous rules as for the positve effect. Except that we use the lower threshold instead of the upper threshold. The no effect class remains as is.

When the unknown effect case contains the reference and upper threshold but not the lower threshold, we known that we can exclude a strong negative effect. We classify such an effect as a potential positive effect. Similarly, when the confidence interval contains the reference and the lower threshold but not the upper threshold, we call it a potential negative effect. If the confidence interval contains both the lower and upper threshold, we call it an unknown effect.

Comparing confidence intervals to the reference and thresholds for unknown effects.

The table below gives an overview of the ten classes that we can distinct based on the reference, lower and upper threshold. Note that four classes get the same symbol and terminology as in the four class classification. Those classes have wide confidence intervals and we can’t refine the classification. The six other classes get a two digit symbol. They inherit their first symbol from the four class classification. The second digit refines the classification, using the same symbols as a kind of modifier. The terminology of these six classes uses in principle their four class classification plus a modifying adjective (strong or moderate). The unknown class is an exception. Here we use the potential adjective in combination with positive effect or negative effect.

Graphical overview of the ten class classification split by the number of thresholds in the confidence interval.

Classification Using effectclass

The effectclass package provides classification() for classifying confidence intervals. The function requires a vector of lower bounds of the confidence intervals (ucl) and a vector of upper bounds (ucl). It also requires the lower and upper thresholds (threshold) as a two-element vector. In case the thresholds are symmetric around the reference, you can provide a single value which is the absolute difference between the reference and both thresholds. The default reference is zero. classification() returns a ten level factor using the two digits symbols mentioned above. The levels range from strong positive effect over no effect to strong negative effect. The potential and unknown levels end the row.

ds <- data.frame(
  lcl =        c(-5, -5, -5, -5, -2, -2, -2, 1, 1, 4),
  ucl =         c(5,  2, -1, -4,  5,  2, -1, 5, 2, 5)
)
library(effectclass)
ds$effect <- classification(lcl = ds$lcl, ucl = ds$ucl, threshold = 3)
ds <- ds[order(ds$effect), ]
kable(ds, row.names = FALSE)

coarse_classification() converts the ten classes into the corresponding four classes. The factor has the four single digit levels in a similar order as the ten class factor.

ds$coarse <- coarse_classification(ds$effect)
kable(ds, row.names = FALSE)

In some cases we are not interested in the sign of the effect. remove_sign() recodes the ten classes into six and the four classes into three. It replaces + and - by *.

ds$unsigned_10 <- remove_sign(ds$effect)
ds$unsigned_4 <- remove_sign(ds$coarse)
kable(ds, row.names = FALSE)

Suggestion for Displaying Effects in a Tabular Format

We recommend to publish at least the confidence interval and its classification in a table. You could add the point estimate too, but never without the confidence interval. Think about a relevant sorting order of the rows in the table. The example below displays the effect in decreasing order with uncertain effects at the bottom. Note that we first sort on the classification and then on the point estimate. We added a format() function for your convenience. A double hyphen -- is rendered in Markdown as a so-called em-dash ‘–’. Use the type = "markdown" argument of format() to convert the classification as verbatim text, which preserves the double hyphen when rendering the document.

set.seed(20191126)
n <- 50
ds <- data.frame(
  effect = paste0(
    sample(LETTERS, n, replace = TRUE),
    sample(LETTERS, n, replace = TRUE),
    sample(LETTERS, n, replace = TRUE)
  ),
  point_estimate = runif(n, min = -2, max = 2),
  se = runif(n)
)
ds$lcl <- qnorm(p = 0.025, mean = ds$point_estimate, sd = ds$se)
ds$ucl <- qnorm(p = 0.975, mean = ds$point_estimate, sd = ds$se)
ds$estimate <- sprintf("%.2f (%.2f; %.2f)", ds$point_estimate, ds$lcl, ds$ucl)
ds$class <- classification(lcl = ds$lcl, ucl = ds$ucl, threshold = 1)
tab <- ds[order(ds$class, -ds$point_estimate), c("effect", "estimate", "class")]
tab$markdown <- format(tab$class, type = "markdown")
kable(
  tab,
  caption = "Example table with effects, their confidences interval and
  classification using different formatting styles.",
  row.names = FALSE,
  align = "c"
)