- Cronbach’s Alpha - Quick Definition
- SPSS Cronbach’s Alpha Output
- Increase Cronbach’s Alpha by Removing Items
- Cronbach’s Alpha is Negative
- There are too Few Cases (N = 0) for the Analysis
- APA Reporting Cronbach’s Alpha

## Introduction

A psychology faculty wants to examine the reliability of a personality test. They therefore have a sample of N = 90 students fill it out. The data thus gathered are in big-5.sav, partly shown below.

As suggested by the variable names, our test attempts to measure the “big 5” personality traits. For other data files, a factor analysis is often used to find out which variables measure which subscales.

Anyway. Our main research question is: what are the reliabilities for these 5 subscales as indicated by Cronbach’s alpha? But first off: what's Cronbach’s alpha anyway?

## Cronbach’s Alpha - Quick Definition

Cronbach’s alpha is the extent to which the sum over 2(+)

variables measures a single underlying trait.
More precisely, Cronbach’s alpha is the proportion of variance of such a sum score that can be accounted for by a single trait. That is, it is the extent to which a sum score *reliably* measures something and (thus) the extent to which a set of items *consistently* measure “the same thing”.

Cronbach’s alpha is therefore known as a measure of **reliability** or **internal consistency**. The most common rules of thumb for it are that

- Cronbach’s alpha ≥
**0.80**is**good**and - Cronbach’s alpha ≈
**0.70**may or may not be just**acceptable**.

## SPSS Reliability Dialogs

In SPSS, we get Cronbach’s alpha from as shown below.

For analyzing the first subscale, agreeableness, we fill out the dialogs as shown below.

Clicking Paste results in the syntax below. Let's run it.

***CRONBACH AGREEABLENESS.**

RELIABILITY

/VARIABLES=agree01 agree02 agree03 agree04 agree05

/SCALE('Agreeableness') ALL

/MODEL=ALPHA

/STATISTICS=CORR

/SUMMARY=TOTAL.

## SPSS Cronbach’s Alpha Output I

For reliability, SPSS only offers listwise exclusion of missing values: all results are based only on N = 85 cases having zero missing values on our 5 analysis variables or “items”.

Cronbach’s alpha = 0.894. You can usually ignore Cronbach’s Alpha Based on Standardized Items: standardizing variables into z-scores prior to computing scale scores is rarely -if ever- done.

Finally, excluding a variable from a (sub)scale may increase Cronbach’s Alpha. That's not the case in this table: for each item, Cronbach’s Alpha if Item Deleted is lower than the α = 0.894 based on all 5 items.

We'll now run the exact same analysis for our second subscale, conscientiousness. Doing so results in the syntax below.

***CRONBACH CONSCIENTIOUSNESS.**

RELIABILITY

/VARIABLES=consc01 consc02 consc03 consc04 consc05

/SCALE('Conscientiousness') ALL

/MODEL=ALPHA

/STATISTICS=CORR

/SUMMARY=TOTAL.

## Increase Cronbach’s Alpha by Removing Items

For the conscientiousness subscale, Cronbach’s alpha = 0.658, which is pretty poor. However, note that Cronbach’s Alpha if Item Deleted = 0.726 for both consc02 and consc04.

Since removing either item should result in α ≈ 0.726, we're not sure which should be removed first. Two ways to find out are

- increasing the decimal places or (better)
- sorting the table by its last column.

As you probably saw, we already did both with the following OUTPUT MODIFY commands:

***Increase decimal places for Cronbach's alpha if item deleted.**

output modify

/select tables

/tablecells select = ['Cronbach''s Alpha if Item Deleted'] format = 'f10.8'.

***Sort item-total statistics by Cronbach's alpha if item deleted.**

output modify

/select tables

/table sort = collabel('Cronbach''s Alpha if Item Deleted').

It turns out that removing consc04 increases alpha slightly more than consc02. The preferred way for doing so is to simply copy-paste the previous RELIABILITY command, remove consc04 from it and rerun it.

***REMOVE CONSC04.**

RELIABILITY

/VARIABLES=consc01 consc02 consc03 consc05

/SCALE('Conscientiousness') ALL

/MODEL=ALPHA

/STATISTICS=CORR

/SUMMARY=TOTAL.

After doing so, Cronbach’s alpha = 0.724. It's not *exactly* the predicted 0.726 because removing consc04 increases the sample size to N = 84. Note that we can increase α even further to 0.814 by removing consc02 as well. The syntax below does just that.

***REMOVE CONSC02.**

RELIABILITY

/VARIABLES=consc01 consc03 consc05

/SCALE('Conscientiousness') ALL

/MODEL=ALPHA

/STATISTICS=CORR

/SUMMARY=TOTAL.

Note that Cronbach’s alpha = 0.814 if we compute our conscientiousness subscale as the sum or mean over consc01, consc03 and consc05. Since that's fine, we're done with this subscale.

Let's proceed with the next subscale: extraversion. We do so by running the exact same analysis on extra01 to extra05, which results in the syntax below.

***CRONBACH EXTRAVERSION.**

RELIABILITY

/VARIABLES=extra01 extra02 extra03 extra04 extra05

/SCALE('Extraversion') ALL

/MODEL=ALPHA

/STATISTICS=CORR

/SUMMARY=TOTAL.

## Cronbach’s Alpha is Negative

As shown below, Cronbach’s alpha = -0.663 for the extraversion subscale. This implies that some correlations among items are negative (second table, below).

All extraversion items are coded similarly: they have identical value labels so that's not the problem. The problem is that some items measure the opposite of the other items as shown below.

The solution is to simply reverse code such “negative items”: we RECODE these 2 items and adjust their value/variable labels with the syntax below.

***REVERSE CODE EXTRA01 AND EXTRA02.**

RECODE extra01 extra02 (1.0 = 5.0)(2.0 = 4.0)(3.0 = 3.0)(4.0 = 2.0)(5.0 = 1.0).

EXECUTE.

VALUE LABELS

/extra01 5.0 'Disagree strongly' 4.0 'Disagree a little' 3.0 'Neither agree nor disagree' 2.0 'Agree a little' 1.0 'Agree strongly' 6 'No answer'

/extra02 5.0 'Disagree strongly' 4.0 'Disagree a little' 3.0 'Neither agree nor disagree' 2.0 'Agree a little' 1.0 'Agree strongly' 6 'No answer'.

VARIABLE LABELS

extra01 'Tends to be quiet (R)'

extra02 'Is sometimes shy, inhibited (R)'.

Rerunning the exact same reliability analysis as previous now results in Cronbach’s alpha = 0.857 for the extraversion subscale.

So let's proceed with the neuroticism subscale. The syntax below runs our default reliability analysis on neur01 to neur05.

***CRONBACH NEUROTICISM.**

RELIABILITY

/VARIABLES=neur01 neur02 neur03 neur04 neur05

/SCALE('ALL VARIABLES') ALL

/MODEL=ALPHA

/STATISTICS=CORR

/SUMMARY=TOTAL.

## There are too Few Cases (N = 0) for the Analysis

Note that our last command doesn't result in any useful tables. We only get the warning shown below.

The 3 most likely **causes** for this problem are that

- one or more variables contains only missing values;
- an incorrect FILTER filters out
*all*cases in the data; - missing values are scattered over numerous analysis variables.

A very quick way to find out is running a minimal DESCRIPTIVES command as in descriptives neur01 to neur05. Upon doing so, we learn that each variable has N ≥ 67 but valid N (listwise) = 0.

So what we really want here, is to use pairwise exclusion of missing values. For some dumb reason, that's not included in SPSS. However, doing it manually isn't as hard as it seems.

## Cronbach’s Alpha with Pairwise Exclusion of Missing Values

We'll start off with the formula for Cronbach’s alpha, which is

$$Cronbach’s\;\alpha = \frac{k^2 \overline{S_{xy}}}{\Sigma S^2_x + 2 \Sigma S_{xy}}$$

where

- \(k\) denotes the number of items;
- \(S_{xy}\) denotes the covariance between each pair of different items;
- \(S^2_x\) denotes the sample variance for each item.

Note that a pairwise covariance matrix contains all statistics used by this formula. It is easily obtained via the regression syntax below:

***COMPUTE COVARIANCE MATRIX FOR NEUR01 THROUGH NEUR05.**

regression

/missing

**pairwise**

/dependent neur01

/method enter neur02 to neur05

/descriptives n

**cov**.

Next, we copy the result into this Googlesheet. Finally, a handful of very simple formulas tell us that α = 0.889.

Now, which sample size should we report for this subscale? I propose you follow the conventions for pairwise regression here and
report the smallest *pairwise* N
which results in N = 44 for this analysis. Again, note that the formula for finding this minimum over a block of cells is utterly simple.

## APA Reporting Cronbach’s Alpha

The table below shows how to report Cronbach’s alpha in APA style for all subscales.

This table contains the actual results from big-5.sav so you can verify that your analyses correspond to mine. The easiest way to create this table is to manually copy-paste your final results into Excel. This makes it easy to adjust things such as decimal places and styling.

Thanks for reading.

## THIS TUTORIAL HAS 14 COMMENTS:

## By Ruben Geert van den Berg on November 22nd, 2022

Frankly, I'm not too sure about that: if missing values are scattered over variables, listwise exclusion can cause a huge drop in sample size.

Since missingness tends to correlate with basically everything, the cases that are left for procedures can't even remotely be considered a simple random sample from all respondents.

Now, in real life, most nett samples are not quite representative of general populations. Perhaps this problem is exacerbated by using a non random subset (without any missings) from such a sample?

My fear is that this may eventually result in more bias than that caused by pairwise exclusion: with listwise exclusion, you lose more data points in a non random fashion than with pairwise exclusion.

I'm not sure if both phenomena have been equally well studied and documented for real life data.

Curious about your opinion on my considerations.

## By Jon K Peck on November 22nd, 2022

Missing data is the curse of data analysis.

I am not an expert on reliability, but the issues are pretty much the same with regression.

The best way to handle it is to minimize it in the first place with careful design, but that, of course, isn't always possible, and forcing respondents to answer in a survey context may cause them to provide inaccurate answers anyway.

In the survey context, users may just drop out of the survey, which is the ultimate listwise deletion.

Neither listwise nor pairwise treatment can really cure the problem in the sense that results

may not generalize to the sample as a whole or the population. Listwise may cause a lot of data

loss that pairwise could reduce, and it may, as you say, leave an estimation sample that is itself

reduced in size and nonrepresentative unless the data are MCAR.

But pairwise has the additional problem that the statistical properties of the results are not what they would be using the usual formulas. In the case of regression and for some of the reliability calculations, the pairwise covariance matrix might not even be positive definite, and even if it is, the estimated standard errors would not be correct, because the elements are estimated with different data. That's a big problem with pairwise.

If individual variables have a lot of missing data, it might be best to omit them completely. Missing value imputation might be applied, but that has its own problems.

One cool way to handle missing data in regression and classification problems is with a random forest. The SPSS random forest extension command (SPSSINC RANFOR) provides two methods: (1) Rough, which works for all variables, imputes missing values in scale variables as the variable's median value. For categorical variables, the modal value is used with ties broken at random. (2) RF impute, which is only available for predictors, imputes missing values based on a random forest.

Maybe this could be adapted for reliability.

## By Ruben Geert van den Berg on November 23rd, 2022

I think these considerations are nicely balanced and I roughly agree with these points.

A quick remark is that -in real life- I usually exclude entire cases with high percentages of missing values. Over all variables within some analysis, I tend to allow for some 10% of all data points missing and then use pairwise exclusion.

Random forests are basically classification trees, iteratively run on random subsets of data, right?

What I understood is that each iteration is some kind of (CH)AID. I'm not a fan of CHAID at all because I never saw it outperform OLS regression (possibly with dummified variables). It also tends to fail to detect interaction effects if main effects are absent.

I think multiple imputation may be a much more promising approach for handling missing values.

It's an interesting discussion anyway, especially for real analysts working with real data as opposed to the academic armchair philosophers who can afford to sell "no" for an answer ;-)

## By Jon K Peck on November 23rd, 2022

Random forests are often very effective in both classification and regression problems without much tuning. (Since it handles both classification and regression, I placed it awkwardly on the Analyze menu rather than one of its submenus.) For missing values, the rfimpute choice is a bit like an EM algorithm. From the underlying library doc...

The algorithm starts by imputing NAs using [median or mode]. Then randomForest is called with the completed data. The proximity matrix from the randomForest is used to update the imputation of the NAs. For continuous predictors, the imputed value is the weighted average of the non-missing observations, where the weights are the proximities. For categorical predictors, the imputed value is the category with the largest average proximity. This process is iterated.

It is a bit frustrating that the output is not a model like regression but rather an ensemble, but it does help with variable selection as well as prediction. And it evades a lot of the inference problems with either listwise or pairwise missing value treatments, because it does not produce statistical significance results - only error rates and variable importance. There is a lot more output available than the current RANFOR extension provides. Maybe time to work on that further.

As for trees, they really need cross validation, whether CHAID or C&RT, but they are often useful as a starting point in modeling.