cProfile for performance profiling
Overview
cProfile is a built-in Python tool that measures exactly how long each function in your code takes to run. Imagine taking a long road trip that took 10 hours instead of 5. You look at your GPS history to see exactly where you spent the most time (3 hours at a restaurant, 2 hours in traffic). cProfile is the GPS history for your code. It shows you which functions are the slow 'traffic jams'. ```python import cProfile def slow_task(): # A loop that does a lot of work total = 0 for i in range(1000000): total += i return total # This runs the function and prints a speed report # cProfile.run('slow_task()') ``` The rule is: **never guess what makes your code slow; use cProfile to measure it.** It gives you a report showing the number of times a function was called (`ncalls`) and the total time spent inside it (`tottime`).
It prevents you from wasting hours optimizing the wrong part of your code. You find the exact bottleneck instantly.
Where used: Performance tuning, Data science scripts, Backend APIs
Why learn this
- You will know exactly which 5 lines of code are making your entire application slow.
- You will sound professional when you say, 'I profiled the code and found the bottleneck'.
Code walkthrough
def fast():
return sum(range(10))
def slow():
return sum(range(1000000))
print('cProfile would show slow() taking 99% of the time.')Focus: print('cProfile would show slow() taking 99% of the time.')
Aha moment
def process_data():
total = 0
for i in range(1000):
total += i
return total
print('Total time is calls * time_per_call')Prediction: If a function is extremely simple and fast, can it ever be the main bottleneck in a cProfile report?
Common guess: No, because it is simple.
Yes, if it is called millions of times in a loop. `ncalls` matters just as much as `tottime`.
Common mistakes
- Guessing instead of measuring: You assume a database query is slow and spend days fixing it, when cProfile would have shown you the real delay was a slow `for` loop.
- Ignoring the call count: A function might be very fast, but if `ncalls` shows it was called 1 million times, it becomes the bottleneck.
Recall questions
- Explain what cProfile does in your own words, as if to a classmate.
- Why is it important to look at the 'ncalls' column in a cProfile report?
Understanding checks
You profile your code. Function A takes 0.001 seconds per call but is called 10,000 times. Function B takes 2.0 seconds but is called once. Which one takes more total time?
Function A takes 10 seconds total (0.001 * 10,000). Function B takes 2 seconds. Function A is the real bottleneck.
Never guess what makes your code slow; use cProfile to measure it. The total time spent in a function is what actually slows down your app.
A teammate says, 'My code is slow, so I will rewrite every single function to make it faster.' What is a better approach?
Use cProfile to find the one or two slowest functions, and only rewrite those.
Most of the execution time is usually spent in a very small portion of the code. Profiling prevents you from wasting time optimizing fast code.
Practice tasks
Simulate profiling choice
Given this code, change the returned string to state which function you would optimize first based on the simulated times.
Challenge
Write a slow function
Write a simple function called `heavy_work()` that uses a `for` loop to add the numbers from 0 to 500000 into a total, returning the total. This is the kind of function cProfile easily catches.
Continue learning
Previous: Defining functions and return values
Previous: for and while loops