lleectctuurre 9e 9.. - german cancer research center · pandas ist ein modul, das datencontainer...

Lecture 9.Lecture 9.Modules - pandasModules - pandas

Matthias BiegMatthias Bieg

pandas: Intropandas: Intro

Was ist pandasWas ist pandaspandas ist ein Modul, das Datencontainer anbietet, ähnlich den DataFramesin RDie wich�gsten Datencontainer sind:

pandas.Series, undpandas.DataFrame

Zusätzlich zu den Containern gibt es eine Reihe von Analysetools, sowie dieMöglichkeit zum Plo�en

pandas.Seriespandas.SeriesSeries sind ListenDie Element können mit Iden�fiern versehen werden, um einen intui�venZugriff auf die Liste zu gewährleistenDie einzelnen Elemente können von unterschiedlichem Typ sein (integer,float, string, etc...)

idx1 idx2 ... idxm-1 idxm

d1 d2 ... dm-1 dm

pandas.DataFramepandas.DataFrameDataFrames sind 2 dimensionale MatrizenDie Zeilen uns Spalten können mit Iden�fiern versehen werden, um einenintui�ven Zugriff auf einzelne Elemente zu gewährleistenDie einzelnen Elemente können von unterschiedlichem Typ sein (integer,float, string, etc...)

Col1 Col2 ... Coln-1 Coln

idx1 d1,1 d1,2 ... d1,n-1 d1,n

idx2 d2,1 d2,2 ... d2,n-1 d2,n

... ... ... ... ... ...

idxm-1 dm-1,1 dm-1,2 ... dm-1,n-1 dm-1,n

idxm dm,1 dm,2 ... dm,n-1 dm,n

pandas: Object Creationpandas: Object Creation

Creating a SeriesCreating a SeriesIn [12]: import numpy as np

import pandas as pnd

mu = 181sigma = 10.# List of random body heightsmale_heights = [ sigma * i + mu for i in np.random.randn(10) ]# List of Index Namesindex = [ "Individual"+str(i) for i in range(1, 11) ]

# Create pandas.Series objectmale_heights_series = pnd.Series(male_heights, index=index)

print male_heights_series

Individual1 182.305315Individual2 174.399807Individual3 183.832186Individual4 190.007618Individual5 175.869070Individual6 182.522929Individual7 169.786862Individual8 185.835162Individual9 173.510661Individual10 176.082611dtype: float64

Create a DataFrameCreate a DataFrame

By passing a numpy arrayBy passing a numpy arrayIn [20]: import numpy as np

import pandas as pnd

# Create a list of lists wit random elementsmu = 181l = [ [ j * sigma + mu+offset for j in np.random.randn(5) ] for offset in range(8)]# Defin column and row identifierscolumns = [ "Individual"+str(i) for i in range(1, 6) ]index = [ "group"+str(j) for j in range(8) ]# Create DataFramemale_heights_dataframe = pnd.DataFrame(np.array(l), columns=columns, index=index)

print male_heights_dataframe

Individual1 Individual2 Individual3 Individual4 Individual5group0 177.383691 187.517906 182.945059 178.020507 169.232070group1 179.623133 178.138221 188.529097 185.312991 181.738726group2 178.894888 182.964432 181.898760 183.025629 188.742420group3 188.774235 179.007723 183.765283 168.775728 171.615213group4 186.496567 193.176500 190.919088 176.547197 176.048604group5 196.132422 183.594067 196.302713 184.401609 195.707818group6 200.641538 172.983882 189.127611 190.970565 204.081392group7 189.992194 191.152608 184.546473 200.404814 177.099218

By reading a tab-separated �leBy reading a tab-separated �le

Das pandas Modul bietet eine Methode zum Einlesen von tab-separiertenDatein in pandas.DataFrame Objekte: Die Methode pandas.read_csv

In [49]: import pandas as pnd

male_heights_filename = "data/male_heights.csv"

# Read CSV file into pandas.DataFramemale_heights_dataframe = pnd.read_csv(male_heights_filename, sep="\t", index_col=0)

print male_heights_dataframe


pandas: View Datapandas: View Data

top and bottom of DataFrametop and bottom of DataFrameIn [32]: import pandas as pnd

df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0)

# View top of Dataframeprint df.head(3)

# View bottom of DataFrameprint df.tail(2)

Individual1 Individual2 Individual3 Individual4 Individual5group0 177.383691 187.517906 182.945059 178.020507 169.232070group1 179.623133 178.138221 188.529097 185.312991 181.738726group2 178.894888 182.964432 181.898760 183.025629 188.742420 Individual1 Individual2 Individual3 Individual4 Individual5group6 200.641538 172.983882 189.127611 190.970565 204.081392group7 189.992194 191.152608 184.546473 200.404814 177.099218

Index, column, valuesIndex, column, valuesIn [34]: import pandas as pnd


# Inspect row identifiersprint df.index

# Inspect column identifiersprint df.columns

# Inspect values print df.values

Index([u'group0', u'group1', u'group2', u'group3', u'group4', u'group5', u'group6', u'group7'], dtype='object')Index([u'Individual1', u'Individual2', u'Individual3', u'Individual4', u'Individual5'], dtype='object')[[ 177.38369094 187.51790556 182.94505876 178.02050731 169.23207042] [ 179.62313257 178.1382207 188.52909722 185.31299054 181.73872603] [ 178.89488799 182.96443159 181.89876004 183.02562885 188.74242018] [ 188.774235 179.00772279 183.76528318 168.77572825 171.61521283] [ 186.4965667 193.17650011 190.91908795 176.54719711 176.04860419] [ 196.1324217 183.594067 196.30271347 184.40160934 195.70781803] [ 200.64153809 172.98388189 189.12761056 190.97056472 204.08139213] [ 189.99219434 191.15260754 184.54647267 200.40481373 177.09921787]]

Summary statistics of DataFrameSummary statistics of DataFrame

rowsrowsIn [41]: import pandas as pnd


# Get summary statistics of DataFrame columnsprint df.describe()

Individual1 Individual2 Individual3 Individual4 Individual5count 8.000000 8.000000 8.000000 8.000000 8.000000mean 187.242333 183.566917 187.254260 183.432380 183.033183std 8.400356 6.846464 4.887240 9.569601 12.205277min 177.383691 172.983882 181.898760 168.775728 169.23207025% 179.441071 178.790347 183.560227 177.652180 174.94025650% 187.635401 183.279249 186.537785 183.713619 179.41897275% 191.527251 188.426581 189.575480 186.727384 190.483770max 200.641538 193.176500 196.302713 200.404814 204.081392

columnscolumnsIn [43]: import pandas as pnd


# Get summary statistics of DataFrame rowsprint df.T.describe()

group0 group1 group2 group3 group4 group5 \count 5.000000 5.000000 5.000000 5.000000 5.000000 5.000000 mean 179.019847 182.668433 183.105226 178.387636 184.637591 191.227726 std 6.839235 4.242703 3.570246 8.298185 7.985160 6.609678 min 169.232070 178.138221 178.894888 168.775728 176.048604 183.594067 25% 177.383691 179.623133 181.898760 171.615213 176.547197 184.401609 50% 178.020507 181.738726 182.964432 179.007723 186.496567 195.707818 75% 182.945059 185.312991 183.025629 183.765283 190.919088 196.132422 max 187.517906 188.529097 188.742420 188.774235 193.176500 196.302713

group6 group7 count 5.000000 5.000000 mean 191.560997 188.639061 std 12.151087 8.609574 min 172.983882 177.099218 25% 189.127611 184.546473 50% 190.970565 189.992194 75% 200.641538 191.152608 max 204.081392 200.404814

Sort by column valuesSort by column valuesIn [48]: import pandas as pnd


print df.sort(columns="Individual1")


/usr/local/lib/python2.7/dist-packages/ipykernel_launcher.py:5: FutureWarning: sort(columns=....) is deprecated, use sort_values(by=.....) """

SummarySummaryMethode Bedeutung

DataFrame.head(x) Gibt die ersten x Zeilen zurück

DataFrame.tail(x) Gibt die letzten x Zeilen zurück

DataFrame.index Gibt die Zeilen Iden�fier zurück

DataFrame.columns Gibt die Spalten Iden�fier zurück

DataFrame.values Gibt die Werte des DataFrames zurück

DataFrame.describe()Gibt Summarysta�s�ken für die einzelnenSpalten aus

DataFrame.sort(columns=[c1, c2,...])

Gibt sor�erten DataFrame zurück

Data AccessionData Accession

GettingGettingIn [58]: import pandas as pnd


# Select a single columnprint df["Individual2"]

group0 187.517906group1 178.138221group2 182.964432group3 179.007723group4 193.176500group5 183.594067group6 172.983882group7 191.152608Name: Individual2, dtype: float64

In [56]: # Selecting via [], which slices the rowsprint df[0:3]

Individual1 Individual2 Individual3 Individual4 Individual5group0 177.383691 187.517906 182.945059 178.020507 169.232070group1 179.623133 178.138221 188.529097 185.312991 181.738726group2 178.894888 182.964432 181.898760 183.025629 188.742420

Selection by LabelSelection by LabelIn [61]: import pandas as pnd


# Getting a rowprint df.loc["group1"]

Individual1 179.623133Individual2 178.138221Individual3 188.529097Individual4 185.312991Individual5 181.738726Name: group1, dtype: float64

In [63]: # Getting selected columns of all rowsprint df.loc[:, ["Individual2", "Individual4"]]

Individual2 Individual4group0 187.517906 178.020507group1 178.138221 185.312991group2 182.964432 183.025629group3 179.007723 168.775728group4 193.176500 176.547197group5 183.594067 184.401609group6 172.983882 190.970565group7 191.152608 200.404814

In [65]: import pandas as pnd


# Slice rows by row idsprint df.loc["group3": "group6", :]

Individual1 Individual2 Individual3 Individual4 Individual5group3 188.774235 179.007723 183.765283 168.775728 171.615213group4 186.496567 193.176500 190.919088 176.547197 176.048604group5 196.132422 183.594067 196.302713 184.401609 195.707818group6 200.641538 172.983882 189.127611 190.970565 204.081392179.007722788

In [66]: # Accessing a single cellprint df.loc["group3", "Individual2"]

179.007722788

Selection by PositionSelection by PositionIn [67]: import pandas as pnd


# Select row via position of the passed integersprint df.iloc[3]

Individual1 188.774235Individual2 179.007723Individual3 183.765283Individual4 168.775728Individual5 171.615213Name: group3, dtype: float64

In [70]: # Slice rows and columnsprint df.iloc[2:4, 1:3]

Individual2 Individual3group2 182.964432 181.898760group3 179.007723 183.765283

In [72]: # Select by integer positions of rows and columnsprint df.iloc[[0, 3, 4], [1, 2]]

Individual2 Individual3group0 187.517906 182.945059group3 179.007723 183.765283group4 193.176500 190.919088

In [74]: # Slice rows explicitlyprint df.iloc[0:3, :]

Individual1 Individual2 Individual3 Individual4 Individual5group0 177.383691 187.517906 182.945059 178.020507 169.232070group1 179.623133 178.138221 188.529097 185.312991 181.738726group2 178.894888 182.964432 181.898760 183.025629 188.742420

In [75]: # Slice columns explicitlyprint df.iloc[:, 0:3]

Individual1 Individual2 Individual3group0 177.383691 187.517906 182.945059group1 179.623133 178.138221 188.529097group2 178.894888 182.964432 181.898760group3 188.774235 179.007723 183.765283group4 186.496567 193.176500 190.919088group5 196.132422 183.594067 196.302713group6 200.641538 172.983882 189.127611group7 189.992194 191.152608 184.546473

Boolean IndexingBoolean IndexingIn [81]: import pandas as pnd


# Using a single columns values to select dataprint df[df["Individual1"] > 180]

Individual1 Individual2 Individual3 Individual4 Individual5group3 188.774235 179.007723 183.765283 168.775728 171.615213group4 186.496567 193.176500 190.919088 176.547197 176.048604group5 196.132422 183.594067 196.302713 184.401609 195.707818group6 200.641538 172.983882 189.127611 190.970565 204.081392group7 189.992194 191.152608 184.546473 200.404814 177.099218

In [82]: # Get all values fullfilling a criterionprint df[df > 190]

Individual1 Individual2 Individual3 Individual4 Individual5group0 NaN NaN NaN NaN NaNgroup1 NaN NaN NaN NaN NaNgroup2 NaN NaN NaN NaN NaNgroup3 NaN NaN NaN NaN NaNgroup4 NaN 193.176500 190.919088 NaN NaNgroup5 196.132422 NaN 196.302713 NaN 195.707818group6 200.641538 NaN NaN 190.970565 204.081392group7 NaN 191.152608 NaN 200.404814 NaN

In [88]: # Using the isin() method for filteringdf2 = df.copy().loc[:, "Individual1":"Individual4"]

# Put new column with subgroup information to the end of DataFramedf2["Subgroup"] = ["one", "two", "two", "three", "two", "one", "three", "one"]

print df2

Individual1 Individual2 Individual3 Individual4 Subgroupgroup0 177.383691 187.517906 182.945059 178.020507 onegroup1 179.623133 178.138221 188.529097 185.312991 twogroup2 178.894888 182.964432 181.898760 183.025629 twogroup3 188.774235 179.007723 183.765283 168.775728 threegroup4 186.496567 193.176500 190.919088 176.547197 twogroup5 196.132422 183.594067 196.302713 184.401609 onegroup6 200.641538 172.983882 189.127611 190.970565 threegroup7 189.992194 191.152608 184.546473 200.404814 one

In [89]: print df2[df2["Subgroup"].isin(["one", "three"])]

Individual1 Individual2 Individual3 Individual4 Subgroupgroup0 177.383691 187.517906 182.945059 178.020507 onegroup3 188.774235 179.007723 183.765283 168.775728 threegroup5 196.132422 183.594067 196.302713 184.401609 onegroup6 200.641538 172.983882 189.127611 190.970565 threegroup7 189.992194 191.152608 184.546473 200.404814 one

Setting ValuesSetting ValuesIn [102]: import pandas as pnd

df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0).loc[:, "Individual1":"Individual3"]

# Set a new column using a oandas.Seriesheights_individual4 = [183., 182., 170., 172., 165., 185., 188., 190.]individual4_series = pnd.Series(heights_individual4, index=["group"+str(i) for i inrange(8)])

df["Individual4"] = individual4_series

print df

Individual1 Individual2 Individual3 Individual4group0 177.383691 187.517906 182.945059 183group1 179.623133 178.138221 188.529097 182group2 178.894888 182.964432 181.898760 170group3 188.774235 179.007723 183.765283 172group4 186.496567 193.176500 190.919088 165group5 196.132422 183.594067 196.302713 185group6 200.641538 172.983882 189.127611 188group7 189.992194 191.152608 184.546473 190

In [103]: # Setting values by labeldf.loc["group0", "Individual1"] = 5

# Setting values by positiondf.iloc[1, 1] = 6

print df

Individual1 Individual2 Individual3 Individual4group0 5.000000 187.517906 182.945059 183group1 179.623133 6.000000 188.529097 182group2 178.894888 182.964432 181.898760 170group3 188.774235 179.007723 183.765283 172group4 186.496567 193.176500 190.919088 165group5 196.132422 183.594067 196.302713 185group6 200.641538 172.983882 189.127611 188group7 189.992194 191.152608 184.546473 190

In [104]: # Setting values by boolean indexingdf[df > 180] = 10.

print df

Individual1 Individual2 Individual3 Individual4group0 5.000000 10.000000 10 10group1 179.623133 6.000000 10 10group2 178.894888 10.000000 10 170group3 10.000000 179.007723 10 172group4 10.000000 10.000000 10 165group5 10.000000 10.000000 10 10group6 10.000000 172.983882 10 10group7 10.000000 10.000000 10 10

SummarySummaryOperator Bedeutung

DataFrame[string] Gibt die Spalte mit dem Label string zurück

DataFrame[integer] Gibt die Spalte an Posi�on integer zurück

DataFrame[integer1:integer2]Gibt einen Slice der Spalten von Posi�on integer1 bisinteger2 zurück

DataFrame.loc[x, y]

Gibt die Spalten mit Labels in y aus den Zeilen mitLabels aus x zurück. Hierbei sind x und y entwederskalare Strings, Listen von Strings, oder Slices vonStrings

DataFrame.iloc[x, y]

Gibt die Spalten an den Posi�onen in y aus denZeilen an den Posi�onen aus x zurück. Hierbei sind xund y entweder skalare Integers, Listen von Integers,oder Slices von Integers

DataFrame[Bedingung]Gibt komple�en DataFrame zurück, setzt jedochWerte im DataFrame, die die Bedingung nichterfüllen auf NaN

pandas.DataFram.applypandas.DataFram.apply

Method SpecsMethod Specspandas.DataFrame.apply(func, axis=0, args=())

Argument Default Bedeutung

func - Funk�on, die auf Spalten, bzw. Zeilen angewandt wird

axis 0Auf welche Achse soll di Funk�on angewandt werden(0:=Zeilen, 1:=Spalten)

args () Posi�onale Argumente der Funk�on

Apply to rowsApply to rowsIn [15]: import pandas as pnd


def mean(x):return (sum(x)/len(x))

# Apply a function to rowsrow_means = df.apply(mean, axis=0)print row_means

Individual1 187.242333Individual2 183.566917Individual3 187.254260Individual4 183.432380Individual5 183.033183dtype: float64

Apply to columnsApply to columnsIn [18]: # Apply a function to columns

col_means = df.apply(mean, axis=1)print col_means

group0 179.019847group1 182.668433group2 183.105226group3 178.387636group4 184.637591group5 191.227726group6 191.560997group7 188.639061dtype: float64

Kombinieren von DataFramesKombinieren von DataFrames

pandas.concatpandas.concatIn [27]: import pandas as pnd


print df

df_pieces = [df[:1], df[1:3], df[3:]]

print pnd.concat(df_pieces)

Individual1 Individual2 Individual3 Individual4 Individual5group0 177.383691 187.517906 182.945059 178.020507 169.232070group1 179.623133 178.138221 188.529097 185.312991 181.738726group2 178.894888 182.964432 181.898760 183.025629 188.742420group3 188.774235 179.007723 183.765283 168.775728 171.615213group4 186.496567 193.176500 190.919088 176.547197 176.048604group5 196.132422 183.594067 196.302713 184.401609 195.707818group6 200.641538 172.983882 189.127611 190.970565 204.081392group7 189.992194 191.152608 184.546473 200.404814 177.099218 Individual1 Individual2 Individual3 Individual4 Individual5group0 177.383691 187.517906 182.945059 178.020507 169.232070group1 179.623133 178.138221 188.529097 185.312991 181.738726group2 178.894888 182.964432 181.898760 183.025629 188.742420group3 188.774235 179.007723 183.765283 168.775728 171.615213group4 186.496567 193.176500 190.919088 176.547197 176.048604group5 196.132422 183.594067 196.302713 184.401609 195.707818group6 200.641538 172.983882 189.127611 190.970565 204.081392group7 189.992194 191.152608 184.546473 200.404814 177.099218

pandas.mergepandas.mergeIn [42]: import pandas as pnd

left = pnd.DataFrame(np.array([["foo", 1], ["foo", 2]]), columns=["key", "lval"])print left

key lval0 foo 11 foo 2

In [43]: right = pnd.DataFrame(np.array([["foo", 4], ["foo", 5]]), columns=["key", "rval"])print right

key rval0 foo 41 foo 5

In [44]: left_right_merged = pnd.merge(left, right, on="key")print left_right_merged

key lval rval0 foo 1 41 foo 1 52 foo 2 43 foo 2 5

pandas.DataFrame.append()pandas.DataFrame.append()In [45]: import pandas as pnd

df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0)print df


In [46]: s = df.iloc[7]print df.append(s, ignore_index=True)

Individual1 Individual2 Individual3 Individual4 Individual50 177.383691 187.517906 182.945059 178.020507 169.2320701 179.623133 178.138221 188.529097 185.312991 181.7387262 178.894888 182.964432 181.898760 183.025629 188.7424203 188.774235 179.007723 183.765283 168.775728 171.6152134 186.496567 193.176500 190.919088 176.547197 176.0486045 196.132422 183.594067 196.302713 184.401609 195.7078186 200.641538 172.983882 189.127611 190.970565 204.0813927 189.992194 191.152608 184.546473 200.404814 177.0992188 189.992194 191.152608 184.546473 200.404814 177.099218

pandas.DataFrame.plot()pandas.DataFrame.plot()

Method SpecsMethod Specspandas.DataFrame.plot(kind=plot_type)

Die Plot Methode eines pandas DataFrames kann dazu benutzt werden um einebes�mmte Plotart für alle Spalten zu erzeugen. Das Schlüsselwortargument kind gibthierbei an welche Art von Plot erzeugt werden soll.

Beispiel: BoxplotsBeispiel: BoxplotsIn [31]: import pandas as pnd

%matplotlib inline


In [32]: # Plotte Boxplot für alle Gruppen (d.h. Zeilen)df.T.plot(kind="box")

Out[32]: <matplotlib.axes._subplots.AxesSubplot at 0xad9c82cc>

Beispiel: HistogrammeBeispiel: HistogrammeIn [27]: import numpy as np

import matplotlib.pyplot as pltdf = pnd.DataFrame(np.array([np.random.randn(1000), [ i + 10 for i in np.random.randn(1000) ]]))

In [30]: fig = plt.figure(figsize=(10, 3))

# Plotte Histogram für die erste Zeileplt.subplot(1,2,1)df.iloc[0].T.plot(kind="hist")

# Plotte Histogram für die zweite Zeileplt.subplot(1,2,2)df.iloc[1].T.plot(kind="hist")

Out[30]: <matplotlib.axes._subplots.AxesSubplot at 0xadcfad2c>

lleectctuurre 9e 9.. - german cancer research center · pandas ist ein modul, das datencontainer...

Documents