homework01.ipynb

Kernel: Python 3

Homework 1

Load and validate GSS data

Allen Downey

MIT License

%matplotlib inline

import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(style='white')

import utils
from utils import decorate
from thinkstats2 import Pmf, Cdf

Loading and validation

def read_gss(dirname):
    """Reads GSS files from the given directory.
    
    dirname: string
    
    returns: DataFrame
    """
    dct = utils.read_stata_dict(dirname + '/GSS.dct')
    gss = dct.read_fixed_width(dirname + '/GSS.dat.gz',
                             compression='gzip')
    return gss

Read the variables I selected from the GSS dataset. You can look up these variables at https://gssdataexplorer.norc.org/variables/vfilter

gss = read_gss('gss_eda')
print(gss.shape)
gss.head()

Most variables use special codes to indicate missing data. We have to be careful not to use these codes as numerical data; one way to manage that is to replace them with NaN, which Pandas recognizes as a missing value.

def replace_invalid(df):
    df.realinc.replace([0], np.nan, inplace=True)                  
    df.educ.replace([98,99], np.nan, inplace=True)
    # 89 means 89 or older
    df.age.replace([98, 99], np.nan, inplace=True) 
    df.cohort.replace([9999], np.nan, inplace=True)
    df.adults.replace([9], np.nan, inplace=True)

replace_invalid(gss)

Here are summary statistics for the variables I have validated and cleaned.

gss['year'].describe()

gss['sex'].describe()

gss['age'].describe()

gss['cohort'].describe()

gss['race'].describe()

gss['educ'].describe()

gss['realinc'].describe()

gss['wtssall'].describe()

Exercise

1. Look through the column headings to find a few variables that look interesting. Look them up on the GSS data explorer.

2. Use value_counts to see what values appear in the dataset, and compare the results with the counts in the code book.

3. Identify special values that indicate missing data and replace them with NaN.

4. Use describe to compute summary statistics. What do you notice?

Visualize distributions

Let's visualize the distributions of the variables we've selected.

Here's a Hist of the values in educ:

from thinkstats2 import Hist, Pmf, Cdf
import thinkplot

hist_educ = Hist(gss.educ)
thinkplot.hist(hist_educ)
decorate(xlabel='Years of education', 
         ylabel='Count')

Hist as defined in thinkstats2 is different from hist as defined in Matplotlib. The difference is that Hist keeps all unique values and does not put them in bins. Also, hist does not handle NaN.

One of the hazards of using hist is that the shape of the result depends on the bin size.

Exercise:

1. Run the following cell and compare the result to the Hist above.

2. Add the keyword argument bins=11 to plt.hist and see how it changes the results.

3. Experiment with other numbers of bins.

import matplotlib.pyplot as plt

plt.hist(gss.educ.dropna())
decorate(xlabel='Years of education', 
         ylabel='Count')

However, a drawback of Hist and Pmf is that they basically don't work when the number of unique values is large, as in this example:

hist_realinc = Hist(gss.realinc)
thinkplot.hist(hist_realinc)
decorate(xlabel='Real income (1986 USD)', 
         ylabel='Count')

Exercise:

1. Make and plot a Hist of age.

2. Make and plot a Pmf of educ.

3. What fraction of people have 12, 14, and 16 years of education?

# Solution goes here

# Solution goes here

# Solution goes here

# Solution goes here

# Solution goes here

Exercise:

1. Make and plot a Cdf of educ.

2. What fraction of people have more than 12 years of education?

# Solution goes here

# Solution goes here

# Solution goes here

Exercise:

1. Make and plot a Cdf of age.

2. What is the median age? What is the inter-quartile range (IQR)?

# Solution goes here

# Solution goes here

# Solution goes here

Exercise:

Find another numerical variable, plot a histogram, PMF, and CDF, and compute any statistics of interest.

# Solution goes here

# Solution goes here

# Solution goes here

# Solution goes here

Exercise:

1. Compute the CDF of realinc for male and female respondents, and plot both CDFs on the same axes.

2. What is the difference in median income between the two groups?

# Solution goes here

# Solution goes here

# Solution goes here

# Solution goes here

Exercise:

Use a variable to break the dataset into groups and plot multiple CDFs to compare distribution of something within groups.

Note: Try to find something interesting, but be cautious about overinterpreting the results. Between any two groups, there are often many differences, with many possible causes.

# Solution goes here

# Solution goes here

# Solution goes here

# Solution goes here

Save the cleaned data

Now that we have the data in good shape, we'll save it in a binary format (HDF5), which will make it faster to load later.

Also, we have to do some resampling to make the results representative. We'll talk about this in class.

np.random.seed(19)
sample = utils.resample_by_year(gss, 'wtssall')

Save the file.

!rm gss.hdf5
sample.to_hdf('gss.hdf5', 'gss')

Load it and see how fast it is!

%time gss = pd.read_hdf('gss.hdf5', 'gss')
gss.shape