---
jupytext:
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.10.3
kernelspec:
  display_name: Python 3
  language: python
  name: pymc
---

# Introduction to Pandas

Pandas works hand-in-hand with other python libraries (e.g. matplotlib and numpy) to make manipulating data (what the Pandas team calls "Data Munging") easy.  With pandas it is easy to  

1. Easily access data using variable names, but have full linear algebra capabilities of numpy
2. Group data by values (for example, calculate the mean income by state)
3. Plot and summarize values
4. Join (combine) different sources of data
5. Powerful time series and panel data capabilities (beyond scope of course)

Note: If you want to reinforce some of these concepts on your own, I recommend this superb youtube video (https://www.youtube.com/watch?v=5JnMutdy6Fw) and accompanying coursework (https://github.com/brandon-rhodes/pycon-pandas-tutorial) by Brandon Rhodes.

There are numerous ways to get data into Pandas:

* Import excel and comma delimited data
* Import stata, sas, matlab, and other datasets
* Import data from an SQL server
* Import data scraped from the web
* Manually building (by typing in values)

In this tutorial we will focus on the first two methods for reading data, but just know that there is probably a way to get your data into pandas irrespective of what format it is in.

```{code-cell} ipython3
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sbn
```

## Loading and Cleaning Data

We will be loading 2010 "Trip" data from the NOAA Fisheries Service Recreational Fisheries Statistics Survey (called MRIP).  More detail here: https://www.fisheries.noaa.gov/topic/recreational-fishing-data

The MRIPs data needs to be downloaded from my website (you only need to do this once).

```{code-cell} ipython3
##
## Load some data
##
trips_ = pd.read_pickle("../_static/lectures/appendix/mrips_trips.pickle")
trips_.head()
```

Trips is a very big dataset with lots of columns we'll never use.  Let's trim it down:

```{code-cell} ipython3
trips_.columns
```

This reduces the columns and creates a new data frame called `trips` from `trips_`:
```{code-cell} ipython3
trips = trips_[['id_code','year','wave','intercept_date','st','prim1',
                    'prim1_common','prim2','prim2_common','cnty','ffdays12',
                    'ffdays2']]
```

This is summary statistics for numeric data columns:

```{code-cell} ipython3
trips.describe()
```

### Using Pandas DataFrames

#### Referencing columns in data:


```{code-cell} ipython3
# copy data frame into t (note, this is different than assignment [t=trips])
t=trips.copy()

t.ffdays12.head()
```


```{code-cell} ipython3
t['ffdays12'].head()
```

#### Referencing rows in data:

We can use numpy-like slicing:

```{code-cell} ipython3
# rows 50-54
t.loc[51:55]
```

```{code-cell} ipython3
# rows 1 and 2 for the first 5 columns
t.iloc[0:2,0:5]
```

We can select rows of data based on column values.  Let's select all
the rows where `st` is 51 (Virginia)

```{code-cell} ipython3
t[t.st == 51].head()
```

Find all rows where days spent fishing during past 12 months exceeds 10:

```{code-cell} ipython3
t[t.ffdays12>10].head()
```

#### Math on columns: 

```{code-cell} ipython3
# we can create a new variable and use numpy commands:
t['temp'] = t.ffdays12 + .5*np.random.randn(t.shape[0])
t.head()
```


```{code-cell} ipython3
t.temp.mean()
```

```{code-cell} ipython3
# or we could do this all in one step if we didn't want to create a new column in t:
(t.ffdays12 + .5*np.random.randn(t.shape[0])).mean()
```


Note: You can combine pandas with numpy. This is the standard deviation of the column `ffdays12`:  

```{code-cell} ipython3
np.std(t.ffdays12)
```

We can perform matrix/vector operations on pandas data.

Here, we can transpose a slice of the data: 

```{code-cell} ipython3
t[['ffdays12','ffdays2']].head(10).T
```

We can do matrix multiplication:

```{code-cell} ipython3
np.dot(t[['ffdays12','ffdays2']].head(10).T,t[['ffdays12','ffdays2']].head(10))
```