SPSS – Batch Process Files with Python

Running syntax over several SPSS data files in one go is fairly easy. If we use SPSS with Python we don't even have to type in the file names. The Python os (for operating system) module will do it for us.
Try it for yourself by downloading spssfiles.zip. Unzip these files into d:\spssfiles as shown below and you're good to go.

Find All Files and Folders in Root Directory

The syntax below creates a Python list of files and folders in rDir, our root directory. Prefixing it with an r as in r'D:\spssfiles' ensures that the backslash doesn't do anything weird.

Result

Filter Out All .Sav Files

As we see, os.listdir() creates a list of all files and folders in rDir but we only want SPSS data files. For filtering them out, we first create and empty list with savs = []. Next, we'll add each file to this list if it endswith(".sav").

Using Full Paths for SPSS Files

For doing anything whatsoever with our data files, we probably want to open them. For doing so, SPSS needs to know in which folder they are located. We could simply set a default directory in SPSS with CD as in CD "d:\spssfiles". However, having Python create full paths to our files with os.path.join() is a more fool proof approach for this.

Result

Have SPSS Open Each Data File

Generally, we open a data file in SPSS with something like GET FILE "d:\spssfiles\mydata.sav". If we replace the file name with each of the paths in our Python list, we'll open each data file, one by one. We could then add some syntax we'd like to run on each file. Finally, we could save our edits with SAVE OUTFILE "...". and that'll batch process multiple files. In this example, however, we'll simply look up which variables each file contains with spssaux.GetVariableNamesList().

Result

Inspect which Files Contain “Salary”

Now suppose we'd like to know which of our files contain some variable “salary”. We'll simply check if it's present in our variable names list and -if so- print back the name of the data file.

Result

Circumvent Python’s Case Sensitivity

There's one more point I'd like to cover: since we search for “salary”, Python won't detect “Salary” or “SALARY” because it's fully case sensitive. I you don't like that, the simple solution is to convert all variable names for all files to lower()case.
A basic way to change all items in a Python list is [i... for i in list] where i... is a modified version of i, in our case i.lower(). This technique is known as a Python list comprehension and the syntax below uses it to lowercase all variable names (line 13).

Note: since I usually avoid all uppercasing in SPSS variable names, the result is identical to our case sensitive search.

Thanks for reading.

SPSS with Python – Looping over Scatterplots

The right way for looping over tables, charts and other procedures in SPSS is with Python. We'll show how to do so on some real world examples. We'll use alcotest.sav throughout, part of which is shown below.
Note that you need to have the SPSS Python Essentials properly installed for running these examples on your own computer.

Example 1: Simple Loop over Bar Charts

We'd like to visualize how mean reaction times are related to the order in which people went through the 3 alcohol conditions. We'll start by generating the syntax for the first chart from the menu as shown below.

As a rule of thumb, try to use Legacy Dialogs for generating charts. The interface and resulting syntax are wonderfully simple and often result in the exact same charts as the much more complex Chart Builder.

We'll remove all line breaks from the pasted syntax, resulting in GRAPH /BAR(SIMPLE)=MEAN(no_1) BY order. Running this line results the first desired bar chart. For running similar charts over different reaction times, we could copy-paste the line and replace no_1 by no_2 and so on. However, a cleaner way to go is with the Python syntax below.

SPSS Python Loop Syntax 1

Note

You'll probably recognize the bar chart syntax near the end of the second block. The only difference is that the variable name has been replaced by %s. This is a Python string placeholder and it'll be replaced by a different variable name in each iteration.

Result

Example 2: Look Up Variable Names from Data

One thing we don't like about the first example is spelling out the variable names. Python can retrieve them from your data in many ways. An approach that always works is specifying variable names with the SPSS TO and ALL keywords. As shown below, the specification can be expanded into a Python list over which you can loop as desired.

Example 3: Parallel Looping

We'd now like to inspect scatterplots of reaction times of no alcohol versus medium alcohol over each of the 5 trials. Like previously, we'll first generate syntax for just one scatterplot as shown below.

After removing all line breaks, these steps result in GRAPH /SCATTERPLOT(BIVAR)=med_1 WITH no_1 /MISSING=LISTWISE.

Retrieving Variable Names by Pattern

The syntax below sets up two empty Python lists and loops over all variable names in our data. Variable names starting with “no_” are added to one list and those that start with “med_” go into the other. Finally, we'll loop over both lists in parallel for generating our scatterplots.

Note

The second block loops over list indices (“listInd”) that refer to the first, second, ... element in either list. Python then retrieves the first, second, ... variable name from either list with noVars[listInd].

Example 4: Create Variable Names with Concatenation

We'll now show an easier option for our scatterplots that'll work if variable names end in simple numeric suffixes. We'll simply loop over a list holding numbers 1 through 5 (generated by range(1,6)) and concatenate these numbers to the variable name roots.

Note

In Python, %d is a general integer placeholder. It's replaced by some integer number that's specified later.
Alternatively, %(varSuffix)d is replaced by the integer number in varSuffix if %locals() is specified at the end. Using %locals() makes your code more readable and shorter, especially with multiple (text or number) placeholders.

Example 5: Lower Triangular Loop

Our final example creates all possible different scatterplots among a set of variables. That is, if we'd run a correlation matrix of these variables, each cell underneath the main diagonal (hence “lower triangle”) is visualized in a scatterplot. This time we'll look up the variable names by their indices under variable view as shown below.

Syntax

Final Note

Explaining every single line of Python code was way beyond the scope of this tutorial. However, with a bit of trial and error (and Google), you can adapt and reuse these examples in your own projects. Or so we hope anyway. Give it a shot. You'll get there.
Thank you for reading.

Regression over Many Dependent Variables

"I have a data file on which I'd like to carry out several regression analyses. I have four dependent variables, v1 through v4. The independent variables (v5 through v14) are the same for all analyses. How can I carry out these four analyses in an efficient way that would also work for 100 dependent variables?"

SPSS Python Syntax Example

Description

• That this syntax uses Python so you need to have the SPSS Python Essentials installed in order to run it;
• The syntax will simply run a standard SPSS regression analysis analysis over different dependent variables one-by-one;
• Except for the occurrence of %s, Python will submit to SPSS a textbook example of regression syntax generated by the GUI. It can be modified as desired.
• The TO and ALL keywords may be used for specifying the dependent and independent variables. The entire specification is enclosed in quotes.
• As a test file for this solution, you could use supermarket.sav.

Apply Dictionary Information from Excel

Question

“I have an Excel workbook whose three sheets contain data values, variable labels and value labels. How can I apply the dictionary information from these last two sheets to the SPSS dataset after importing the data values?”

Option A: Python

A nice and clean option is to have Python read the dictionary information from the Excel sheets. The cell contents can then be inserted into standard VARIABLE LABELS and ADD VALUE LABELS commands. Running these commands applies the variable labels and value labels to the data values. We'll use data_and_labels.xls for demonstrating this approach.

1. Read the Data Values

Reading Excel data values into SPSS is straightforward. We usually paste the required syntax from File Open Data. The screenshot below shows which options to select.

Importing Excel Data into SPSS

SPSS Syntax for Reading Excel Data

2. Create Variable Labels Command

Let's first open our workbook and take a look at how the second sheet is structured. As shown in the screenshot below, the first column holds variable names and the second variable labels.

SPSS Variable Labels in Excel

Now we'll read this second sheet with Python instead of SPSS. Note that you need to have the SPSS Python Essentials as well as the xlrd module installed first. The syntax below shows how to create the VARIABLE LABELS commands as a single (multi line) string. For now we'll just print it for inspection.

SPSS Python Syntax Example

3. Create Value Labels Command

SPSS Value Labels in Excel

Remember that Python objects persist over program blocks. We can therefore leave out the first lines of syntax from the previous example. The Excel sheet holding value labels has the same basic structure as the one with variable labels (see screenshot). The main difference is that we'll now insert three pieces of information (variable name, value, value label) into each line. We'll generate our ADD VALUE LABELS commands as shown below.

SPSS Python Syntax Example

Running the Python Generated Syntax

If neither of the generated commands require any further tweaking, the only thing left to do is just run them by using spss.Submit. The syntax below does so and thus finishes this job.

Option B: Syntax Generating Syntax

Before Python was introduced to SPSS, a different approach was needed for this situation. It comes down to declaring a new (long) string variable and using CONCAT to create lines of syntax as string values. Next, we save the contents of this string variable as a .txt file with an .sps extension and INSERT it.
We don't usually recommend taking this approach but we'll present it anyway for the sake of the demonstration. Some of the commands used by the syntax below are explained in SPSS Datasets Tutorial 1 - Basics and SPSS String Variables Tutorial.

SPSS Syntax Generating Syntax

Remove Value Label from Multiple Variables

Question

"I'd like to completely remove the value label from a value for many variables at once. Is there an easy way to accomplish that?"

SPSS Python Syntax Example

Description

• Since this syntax uses Python, make sure you have the SPSS Python Essentials installed.
• The two things you'll want to modify for using the example on other data are the variables and the value from which the label should be removed. Both are boldfaced in the syntax example.
• Note that the variables can be specified using the TO and ALL keywords.
• One could use supermarket.sav for testing purposes.

SPSS – Creating a Dictionary Dataset

An often requested feature is to export variable and value labels to Excel. This handy tool creates an SPSS Dataset containing these labels. It can either be saved as an Excel sheet or further edited in SPSS.

SPSS Create Dictionary Dataset Tool - How To Use

• Make sure you have the SPSS Python Essentials installed.
• Next, download and install the Dictionary Dataset Tool. Note that this is an SPSS custom dialog.
• Click Utilities Create Dictionary Dataset.
• Click Paste and run the pasted syntax.
• This creates a new dataset called Dictionary_Overview holding all value labels and variable labels.
• Note that the value for all variable labels is (the lowest value found in the dictionary -1). It merely serves as a placeholder for the value label "Variable Label" and should not be taken literally.
• To avoid confusion, display value labels rather than values by clicking the value labels icon (see screenshot below).
• Clicking the tool's Help button will take you to this tutorial. We very much appreciate your feedback on it.

SPSS Dictionary Dataset Tool - Result

Saving the dictionary overview as Excel sheet

Creating a single sheet Excel workbook holding the dictionary information is demonstrated below. Note that it saves value labels rather than values. For more on setting your working directory see Change Your Working Directory.

Final Note

We've had some doubts regarding the optimal output format before we finally went with a single dataset holding all value and variable labels. An alternative we considered was to directly create an Excel workbook with separate sheets for value labels and variable labels. We may offer this as a second version at some point.

Search Syntax Files for Expression

Question

"I found a variable "v_4" in an old data file and I can't remember how exactly I created it. The syntax I used got a bit messy, I have different files and they're in different folders. Is there an easy way to find out which syntax files contain the expression "v_4"?"

SPSS Search Syntax Files Tool

SPSS Search Syntax Files Tool

• Make sure you have the SPSS Python Essentials installed.
• Next, download and install the SPSS Search Syntax Files Tool. Note that this is an SPSS custom dialog.
• Click Utilities Search Syntax Files.
• Specify a folder, a search expression and a file extension.
• Optionally, specify the first character(s) of the file names you'd like to search.
• Click Paste and run the pasted syntax.
• The full paths to all files containing the search expression are printed back in the SPSS output window.
• Clicking the tool's Help button will take you to this tutorial. We very much appreciate your feedback on it.

Notes

• All files containing plain text can be scanned with this tool. This includes .txt, .html, .csv and .py files.
• Note that MS Word (.doc) and Excel (.xls) files do not contain plain text and therefore can't be scanned by this tool.
• Note that you can copy-paste paths from the SPSS output window into Windows Explorer in order to open files.

Move all Files from Subfolders to Main Folder

Question

"I'd like to work with a number of .sav files but they are scattered over different folders. All file names are unique. Is there any easy way to search through a number of folders for .sav files and move these into some root directory?"

SPSS Python Syntax Example

Description

• Note this syntax uses Python so you need to have the SPSS Python Essentials installed in order to run it.
• The first program block will create some random test folders and files in some root directory called "rdir".
• The actual solution in the second program block will search all subfolders for any .sav files and move them directly into the root directory. The result is that all .sav files will be located in a single folder.

What if File Names aren't Unique?

SPSS Python Syntax Example

Delete Everything in Root Directory Except Data Files

SPSS Python Syntax Example

Split String Variable into Components

Question

"I have a long string variable in my data that actually holds the answers to several questions. These are separated by a semicolon (";"). How can I split this variable into the original answers?"

SPSS Python Syntax Example

Description

• Note that this syntax uses Python. You need to have the SPSS Python Essentials installed for using it.
• The first program block will create a test data set containing a single (long) string variable. If you already have your actual data open in SPSS, you may skip it.
• The second program block defines the function that will split up a string variable into components, given some separator. After running this block just once, the function can be used as many times as necessary until the end of your session. This definition is something you'd typically place in a module.
• After the stringsplitter has been defined, only one short line of code is needed to actually use the function. This is demonstrated in the third program block. Note that the name of the input variable comes first, followed by the separator and both are quoted.
• The new variable names are the original variable names, suffixed by "_sn, where n refers to the nth component of the string.

Assumptions

• It is assumed that the to be created variables do not yet exist in the data to which you apply the function. If so, you may first rename them or modify the default prefix ("_s").
• It is assumed that every occurrence of the separator is meaningful. So if the separator is ";" and a string value ";no;;yes;yes;" occurs, it will be split into 6 new variables holding the values (missing),"no",(missing),"yes","yes",(missing). If this is not to your liking, an easy solution may be to apply basic SPSS string functions (most likely RTRIM, LTRIM and REPLACE) to your string before using the splitter.
• Elaborating on the previous point, if a new variable is empty for all cases, it will be an empty string variable with length 1 (ideally it would have length 0 but this is not allowed in SPSS). It is again presumed that the empty values are present in the string for a reason.

Suffix All Variable Names

Question

“I have a data file in which all variables were measured in 2012. I'd like to suffix their names with "_2012". What's the easiest way to do this?”

SPSS Python Syntax Example

Description

• This syntax will add a suffix to one, many or all variables in your data.
• Variable names can be specified right behind variables = .
• Following SPSS Syntax conventions, variable names should be separated by spaces. The TO and ALL keywords may be used and the entire specification should be enclosed in quotes (''). The desired suffix should be enclosed in quotes as well.
• Next, a suffix can be specified right behind suffix = .
• As a test file for this solution, you could use supermarket.sav.