For many use cases writing pandas in pure python and numpy is sufficient. In some computationally heavy applications however, it can be possible to achieve sizeable speed-ups by offloading work to cython.
This tutorial assumes you have refactored as much as possible in python, for example trying to remove for loops and making use of numpy vectorization, it’s always worth optimising in python first.
This tutorial walks through a “typical” process of cythonizing a slow computation. We use an example from the cython documentation but in the context of pandas. Our final cythonized solution is around 100 times faster than the pure python.
We have a DataFrame to which we want to apply a function row-wise.
In [1]: df = pd.DataFrame({'a': np.random.randn(1000), ...: 'b': np.random.randn(1000), ...: 'N': np.random.randint(100, 1000, (1000)), ...: 'x': 'x'}) ...: In [2]: df Out[2]: N a b x 0 585 0.469112 -0.218470 x 1 841 -0.282863 -0.061645 x 2 251 -1.509059 -0.723780 x 3 972 -1.135632 0.551225 x 4 181 1.212112 -0.497767 x 5 458 -0.173215 0.837519 x 6 159 0.119209 1.103245 x .. ... ... ... .. 993 190 0.131892 0.290162 x 994 931 0.342097 0.215341 x 995 374 -1.512743 0.874737 x 996 246 0.933753 1.120790 x 997 157 -0.308013 0.198768 x 998 977 -0.079915 1.757555 x 999 770 -1.010589 -1.115680 x [1000 rows x 4 columns]
Here’s the function in pure python:
In [3]: def f(x): ...: return x * (x - 1) ...: In [4]: def integrate_f(a, b, N): ...: s = 0 ...: dx = (b - a) / N ...: for i in range(N): ...: s += f(a + i * dx) ...: return s * dx ...:
We achieve our result by using apply
(row-wise):
In [7]: %timeit df.apply(lambda x: integrate_f(x['a'], x['b'], x['N']), axis=1) 10 loops, best of 3: 174 ms per loop
But clearly this isn’t fast enough for us. Let’s take a look and see where the time is spent during this operation (limited to the most time consuming four calls) using the prun ipython magic function:
In [5]: %prun -l 4 df.apply(lambda x: integrate_f(x['a'], x['b'], x['N']), axis=1) 671001 function calls (665992 primitive calls) in 0.242 seconds Ordered by: internal time List reduced from 145 to 4 due to restriction <4> ncalls tottime percall cumtime percall filename:lineno(function) 1000 0.130 0.000 0.189 0.000 <ipython-input-4-eac78f5c232c>:1(integrate_f) 552423 0.059 0.000 0.059 0.000 <ipython-input-3-270d8b4c8fce>:1(f) 3000 0.006 0.000 0.036 0.000 base.py:2537(get_value) 3000 0.004 0.000 0.041 0.000 series.py:620(__getitem__)
By far the majority of time is spend inside either integrate_f
or f
, hence we’ll concentrate our efforts cythonizing these two functions.
Note
In python 2 replacing the range
with its generator counterpart (xrange
) would mean the range
line would vanish. In python 3 range
is already a generator.
First we’re going to need to import the cython magic function to ipython (for cython versions < 0.21 you can use %load_ext cythonmagic
):
In [6]: %load_ext Cython
Now, let’s simply copy our functions over to cython as is (the suffix is here to distinguish between function versions):
In [7]: %%cython ...: def f_plain(x): ...: return x * (x - 1) ...: def integrate_f_plain(a, b, N): ...: s = 0 ...: dx = (b - a) / N ...: for i in range(N): ...: s += f_plain(a + i * dx) ...: return s * dx ...:
Note
If you’re having trouble pasting the above into your ipython, you may need to be using bleeding edge ipython for paste to play well with cell magics.
In [4]: %timeit df.apply(lambda x: integrate_f_plain(x['a'], x['b'], x['N']), axis=1) 10 loops, best of 3: 85.5 ms per loop
Already this has shaved a third off, not too bad for a simple copy and paste.
We get another huge improvement simply by providing type information:
In [8]: %%cython ...: cdef double f_typed(double x) except? -2: ...: return x * (x - 1) ...: cpdef double integrate_f_typed(double a, double b, int N): ...: cdef int i ...: cdef double s, dx ...: s = 0 ...: dx = (b - a) / N ...: for i in range(N): ...: s += f_typed(a + i * dx) ...: return s * dx ...:
In [4]: %timeit df.apply(lambda x: integrate_f_typed(x['a'], x['b'], x['N']), axis=1) 10 loops, best of 3: 20.3 ms per loop
Now, we’re talking! It’s now over ten times faster than the original python implementation, and we haven’t really modified the code. Let’s have another look at what’s eating up time:
In [9]: %prun -l 4 df.apply(lambda x: integrate_f_typed(x['a'], x['b'], x['N']), axis=1) 118576 function calls (113567 primitive calls) in 0.056 seconds Ordered by: internal time List reduced from 142 to 4 due to restriction <4> ncalls tottime percall cumtime percall filename:lineno(function) 3000 0.006 0.000 0.038 0.000 base.py:2537(get_value) 3000 0.004 0.000 0.043 0.000 series.py:620(__getitem__) 9027 0.004 0.000 0.007 0.000 {built-in method builtins.getattr} 1 0.003 0.003 0.055 0.055 {pandas._libs.lib.reduce}
It’s calling series... a lot! It’s creating a Series from each row, and get-ting from both the index and the series (three times for each row). Function calls are expensive in python, so maybe we could minimise these by cythonizing the apply part.
Note
We are now passing ndarrays into the cython function, fortunately cython plays very nicely with numpy.
In [10]: %%cython ....: cimport numpy as np ....: import numpy as np ....: cdef double f_typed(double x) except? -2: ....: return x * (x - 1) ....: cpdef double integrate_f_typed(double a, double b, int N): ....: cdef int i ....: cdef double s, dx ....: s = 0 ....: dx = (b - a) / N ....: for i in range(N): ....: s += f_typed(a + i * dx) ....: return s * dx ....: cpdef np.ndarray[double] apply_integrate_f(np.ndarray col_a, np.ndarray col_b, np.ndarray col_N): ....: assert (col_a.dtype == np.float and col_b.dtype == np.float and col_N.dtype == np.int) ....: cdef Py_ssize_t i, n = len(col_N) ....: assert (len(col_a) == len(col_b) == n) ....: cdef np.ndarray[double] res = np.empty(n) ....: for i in range(len(col_a)): ....: res[i] = integrate_f_typed(col_a[i], col_b[i], col_N[i]) ....: return res ....:
The implementation is simple, it creates an array of zeros and loops over the rows, applying our integrate_f_typed
, and putting this in the zeros array.
Warning
You can not pass a Series
directly as a ndarray
typed parameter to a cython function. Instead pass the actual ndarray
using the .values
attribute of the Series. The reason is that the cython definition is specific to an ndarray and not the passed Series.
So, do not do this:
apply_integrate_f(df['a'], df['b'], df['N'])
But rather, use .values
to get the underlying ndarray
apply_integrate_f(df['a'].values, df['b'].values, df['N'].values)
Note
Loops like this would be extremely slow in python, but in Cython looping over numpy arrays is fast.
In [4]: %timeit apply_integrate_f(df['a'].values, df['b'].values, df['N'].values) 1000 loops, best of 3: 1.25 ms per loop
We’ve gotten another big improvement. Let’s check again where the time is spent:
In [11]: %prun -l 4 apply_integrate_f(df['a'].values, df['b'].values, df['N'].values) 203 function calls in 0.001 seconds Ordered by: internal time List reduced from 53 to 4 due to restriction <4> ncalls tottime percall cumtime percall filename:lineno(function) 1 0.001 0.001 0.001 0.001 {built-in method _cython_magic_9df59845e7ff190f442cda6f3d60e56c.apply_integrate_f} 1 0.000 0.000 0.001 0.001 {built-in method builtins.exec} 3 0.000 0.000 0.000 0.000 internals.py:3865(iget) 3 0.000 0.000 0.000 0.000 generic.py:1837(_get_item_cache)
As one might expect, the majority of the time is now spent in apply_integrate_f
, so if we wanted to make anymore efficiencies we must continue to concentrate our efforts here.
There is still hope for improvement. Here’s an example of using some more advanced cython techniques:
In [12]: %%cython ....: cimport cython ....: cimport numpy as np ....: import numpy as np ....: cdef double f_typed(double x) except? -2: ....: return x * (x - 1) ....: cpdef double integrate_f_typed(double a, double b, int N): ....: cdef int i ....: cdef double s, dx ....: s = 0 ....: dx = (b - a) / N ....: for i in range(N): ....: s += f_typed(a + i * dx) ....: return s * dx ....: @cython.boundscheck(False) ....: @cython.wraparound(False) ....: cpdef np.ndarray[double] apply_integrate_f_wrap(np.ndarray[double] col_a, np.ndarray[double] col_b, np.ndarray[int] col_N): ....: cdef int i, n = len(col_N) ....: assert len(col_a) == len(col_b) == n ....: cdef np.ndarray[double] res = np.empty(n) ....: for i in range(n): ....: res[i] = integrate_f_typed(col_a[i], col_b[i], col_N[i]) ....: return res ....:
In [4]: %timeit apply_integrate_f_wrap(df['a'].values, df['b'].values, df['N'].values) 1000 loops, best of 3: 987 us per loop
Even faster, with the caveat that a bug in our cython code (an off-by-one error, for example) might cause a segfault because memory access isn’t checked.
A recent alternative to statically compiling cython code, is to use a dynamic jit-compiler, numba
.
Numba gives you the power to speed up your applications with high performance functions written directly in Python. With a few annotations, array-oriented and math-heavy Python code can be just-in-time compiled to native machine instructions, similar in performance to C, C++ and Fortran, without having to switch languages or Python interpreters.
Numba works by generating optimized machine code using the LLVM compiler infrastructure at import time, runtime, or statically (using the included pycc tool). Numba supports compilation of Python to run on either CPU or GPU hardware, and is designed to integrate with the Python scientific software stack.
Note
You will need to install numba
. This is easy with conda
, by using: conda install numba
, see installing using miniconda.
Note
As of numba
version 0.20, pandas objects cannot be passed directly to numba-compiled functions. Instead, one must pass the numpy
array underlying the pandas
object to the numba-compiled function as demonstrated below.
Using numba
to just-in-time compile your code. We simply take the plain python code from above and annotate with the @jit
decorator.
import numba @numba.jit def f_plain(x): return x * (x - 1) @numba.jit def integrate_f_numba(a, b, N): s = 0 dx = (b - a) / N for i in range(N): s += f_plain(a + i * dx) return s * dx @numba.jit def apply_integrate_f_numba(col_a, col_b, col_N): n = len(col_N) result = np.empty(n, dtype='float64') assert len(col_a) == len(col_b) == n for i in range(n): result[i] = integrate_f_numba(col_a[i], col_b[i], col_N[i]) return result def compute_numba(df): result = apply_integrate_f_numba(df['a'].values, df['b'].values, df['N'].values) return pd.Series(result, index=df.index, name='result')
Note that we directly pass numpy
arrays to the numba function. compute_numba
is just a wrapper that provides a nicer interface by passing/returning pandas objects.
In [4]: %timeit compute_numba(df) 1000 loops, best of 3: 798 us per loop
numba
can also be used to write vectorized functions that do not require the user to explicitly loop over the observations of a vector; a vectorized function will be applied to each row automatically. Consider the following toy example of doubling each observation:
import numba def double_every_value_nonumba(x): return x*2 @numba.vectorize def double_every_value_withnumba(x): return x*2 # Custom function without numba In [5]: %timeit df['col1_doubled'] = df.a.apply(double_every_value_nonumba) 1000 loops, best of 3: 797 us per loop # Standard implementation (faster than a custom function) In [6]: %timeit df['col1_doubled'] = df.a*2 1000 loops, best of 3: 233 us per loop # Custom function with numba In [7]: %timeit df['col1_doubled'] = double_every_value_withnumba(df.a.values) 1000 loops, best of 3: 145 us per loop
Note
numba
will execute on any function, but can only accelerate certain classes of functions.
numba
is best at accelerating functions that apply numerical functions to numpy arrays. When passed a function that only uses operations it knows how to accelerate, it will execute in nopython
mode.
If numba
is passed a function that includes something it doesn’t know how to work with – a category that currently includes sets, lists, dictionaries, or string functions – it will revert to object mode
. In object mode
, numba will execute but your code will not speed up significantly. If you would prefer that numba
throw an error if it cannot compile a function in a way that speeds up your code, pass numba the argument nopython=True
(e.g. @numba.jit(nopython=True)
). For more on troubleshooting numba
modes, see the numba troubleshooting page.
Read more in the numba docs.
eval()
The top-level function pandas.eval()
implements expression evaluation of Series
and DataFrame
objects.
Note
To benefit from using eval()
you need to install numexpr
. See the recommended dependencies section for more details.
The point of using eval()
for expression evaluation rather than plain Python is two-fold: 1) large DataFrame
objects are evaluated more efficiently and 2) large arithmetic and boolean expressions are evaluated all at once by the underlying engine (by default numexpr
is used for evaluation).
Note
You should not use eval()
for simple expressions or for expressions involving small DataFrames. In fact, eval()
is many orders of magnitude slower for smaller expressions/objects than plain ol’ Python. A good rule of thumb is to only use eval()
when you have a DataFrame
with more than 10,000 rows.
eval()
supports all arithmetic expressions supported by the engine in addition to some extensions available only in pandas.
Note
The larger the frame and the larger the expression the more speedup you will see from using eval()
.
These operations are supported by pandas.eval()
:
<<
) and right shift (>>
) operators, e.g., df + 2 * pi / s ** 4 % 42 - the_golden_ratio
2 < df < df2
df < df2 and df3 < df4 or not df_bool
list
and tuple
literals, e.g., [1, 2]
or (1, 2)
df.a
df[0]
pd.eval('df')
(this is not very useful)sin
, cos
, exp
, log
, expm1
, log1p
, sqrt
, sinh
, cosh
, tanh
, arcsin
, arccos
, arctan
, arccosh
, arcsinh
, arctanh
, abs
and arctan2
.This Python syntax is not allowed:
is
/is not
operationsif
expressionslambda
expressionslist
/set
/dict
comprehensionsdict
and set
expressionsyield
expressionseval()
Examplespandas.eval()
works well with expressions containing large arrays.
First let’s create a few decent-sized arrays to play with:
In [13]: nrows, ncols = 20000, 100 In [14]: df1, df2, df3, df4 = [pd.DataFrame(np.random.randn(nrows, ncols)) for _ in range(4)]
Now let’s compare adding them together using plain ol’ Python versus eval()
:
In [15]: %timeit df1 + df2 + df3 + df4 11.7 ms +- 1.01 ms per loop (mean +- std. dev. of 7 runs, 100 loops each)
In [16]: %timeit pd.eval('df1 + df2 + df3 + df4') 7.93 ms +- 162 us per loop (mean +- std. dev. of 7 runs, 100 loops each)
Now let’s do the same thing but with comparisons:
In [17]: %timeit (df1 > 0) & (df2 > 0) & (df3 > 0) & (df4 > 0) 22.7 ms +- 435 us per loop (mean +- std. dev. of 7 runs, 10 loops each)
In [18]: %timeit pd.eval('(df1 > 0) & (df2 > 0) & (df3 > 0) & (df4 > 0)') 8.7 ms +- 270 us per loop (mean +- std. dev. of 7 runs, 100 loops each)
eval()
also works with unaligned pandas objects:
In [19]: s = pd.Series(np.random.randn(50)) In [20]: %timeit df1 + df2 + df3 + df4 + s 22.8 ms +- 1.9 ms per loop (mean +- std. dev. of 7 runs, 100 loops each)
In [21]: %timeit pd.eval('df1 + df2 + df3 + df4 + s') 8.35 ms +- 205 us per loop (mean +- std. dev. of 7 runs, 100 loops each)
Note
Operations such as
1 and 2 # would parse to 1 & 2, but should evaluate to 2 3 or 4 # would parse to 3 | 4, but should evaluate to 3 ~1 # this is okay, but slower when using eval
should be performed in Python. An exception will be raised if you try to perform any boolean/bitwise operations with scalar operands that are not of type bool
or np.bool_
. Again, you should perform these kinds of operations in plain Python.
DataFrame.eval
methodIn addition to the top level pandas.eval()
function you can also evaluate an expression in the “context” of a DataFrame
.
In [22]: df = pd.DataFrame(np.random.randn(5, 2), columns=['a', 'b']) In [23]: df.eval('a + b') Out[23]: 0 -0.246747 1 0.867786 2 -1.626063 3 -1.134978 4 -1.027798 dtype: float64
Any expression that is a valid pandas.eval()
expression is also a valid DataFrame.eval()
expression, with the added benefit that you don’t have to prefix the name of the DataFrame
to the column(s) you’re interested in evaluating.
In addition, you can perform assignment of columns within an expression. This allows for formulaic evaluation. The assignment target can be a new column name or an existing column name, and it must be a valid Python identifier.
New in version 0.18.0.
The inplace
keyword determines whether this assignment will performed on the original DataFrame
or return a copy with the new column.
Warning
For backwards compatability, inplace
defaults to True
if not specified. This will change in a future version of pandas - if your code depends on an inplace assignment you should update to explicitly set inplace=True
In [24]: df = pd.DataFrame(dict(a=range(5), b=range(5, 10))) In [25]: df.eval('c = a + b', inplace=True) In [26]: df.eval('d = a + b + c', inplace=True) In [27]: df.eval('a = 1', inplace=True) In [28]: df Out[28]: a b c d 0 1 5 5 10 1 1 6 7 14 2 1 7 9 18 3 1 8 11 22 4 1 9 13 26
When inplace
is set to False
, a copy of the DataFrame
with the new or modified columns is returned and the original frame is unchanged.
In [29]: df Out[29]: a b c d 0 1 5 5 10 1 1 6 7 14 2 1 7 9 18 3 1 8 11 22 4 1 9 13 26 In [30]: df.eval('e = a - c', inplace=False)
© 2008–2012, AQR Capital Management, LLC, Lambda Foundry, Inc. and PyData Development Team
Licensed under the 3-clause BSD License.
http://pandas.pydata.org/pandas-docs/version/0.22.0/enhancingperf.html