Data Manipulation in Python

core python

Goals of this Lesson

Students will learn:

1. How to group and categorize data in Python
2. How to generative descriptive statistics in Python

Links to Files and Video Recording

The files for all tutorials can be downloaded from the Columbia Psychology Scientific Computing GitHub page using these instructions. This particular file is located here: /content/tutorials/python/3-datamanipulation/index.ipynb.

For a video recording of this tutorial from the Fall 2020 workshop, please visit the Workshop Recording: Session 3 page.

# load packages we will be using for this lesson
import pandas as pd

Open Dataset and Load Package

This dataset examines the relationship between multitasking and working memory. Link here to original paper by Uncapher et al. 2016.

# use pd.read_csv to open data into python
df = pd.read_csv("uncapher_2016_repeated_measures_dataset.csv")

Familiarize Yourself with the Data

Quick review from Data Cleaning: take a look at the basic data structure, number of rows and columns.

df.head()

df.shape

(136, 17)

df.columns

Index(['subjNum', 'groupStatus', 'numDist', 'conf', 'hitCount', 'allOldCount',
       'rtHit', 'faCount', 'allNewCount', 'rtFA', 'distPresent', 'hitRate',
       'faRate', 'dprime', 'mmi', 'adhd', 'bis'],
      dtype='object')

Selecting Relevant Variables

Sometimes datasets have many variables that are unnecessary for a given analysis. To simplify your life, and your code, we can select only the given variables we’d like to use for now.

df = df[["subjNum", "groupStatus", "adhd", "hitRate", "faRate", "dprime"]]
df.head()

Basic Descriptives

Summarizing data

Let’s learn how to make simple tables of summary statistics. First, we will calculate summary info across all data using describe(), a useful function for creating summaries. Note that we’re not creating a new object for this summary (i.e. not using the = symbol), so this will print but not save.

df.describe()

Grouping Data

Next, we will learn how to group data based on certain variables of interest.

We will use the groupby() function in pandas, which will automatically group any subsequent actions called on the data.

df.groupby(["groupStatus"]).mean()

We can group data by more than one factor. Let’s say we’re interested in how levels of adhd interact with groupStatus (multitasking: high or low).

We will first make a factor for adhd (median-split), and add it as a grouping variable using the cut() function in pandas:

df["adhdF"] = pd.cut(df["adhd"],bins=2,labels=["Low","High"])

Then we’ll check how evenly split these groups are by using groupby() the size() functions:

df.groupby(["groupStatus","adhdF"]).size()

groupStatus  adhdF
HMM          Low      24
             High     44
LMM          Low      46
             High     22
dtype: int64

Then we’ll calculate some summary info about these groups:

df.groupby(["groupStatus","adhdF"]).mean()

A note on piping / stringing commands together

In R, we often use the pipe %>% to string a series of steps together. We can do the same in Python with many functions in a row.

This is how we’re able to take the output of df.groupby(["groupStatus","adhdF"]) and then send that output into the mean() function

Extra: Working with a Long Dataset

This is a repeated measures (“long”) dataset, with multiple rows per subject. This makes things a bit tricker, but we are going to show you some tools for how to work with “long” datasets.

How many unique subjects are in the data?

subList = df["subjNum"].unique()
nSubs = len(subList)
nSubs

68

How many trials were there per subject?

nTrialsPerSubj = df.groupby(["subjNum"]).size().reset_index(name="nTrials")
nTrialsPerSubj.head()

Combine summary statistics with the full dataframe

For some analyses, you might want to add a higher level variable (e.g., subject average hitRate) alongside your long data. We can do this by summarizing the data in a new dataframe and then merging it with the full data.

avgHR = df.groupby(["subjNum"])["hitRate"].mean().reset_index(name="avgHR")
avgHR.head()

df = df.merge(avgHR,on="subjNum")
df.head()

You should now have an avgHR column in df, which will repeat within each subject, but vary across subjects.

Next: Plotting in Python