How to Profile Memory Usage in Python

Memory management is one way Python makes development faster and easier than other programming languages such as C and C++ or CMS such as WordPress. In Python, this is straightforward since the language takes care of memory management for you. The fact that recollection has been lost does not imply that it should be erased. Instead, good developers will keep track of how much memory their program uses and search for opportunities to reduce memory consumption. Using 

What Are Python Memory Profilers, And How Do They Work?

The use of profiling programs always involves considerations such as CPU, memory, and so on. Python programs, on the other hand, are susceptible to memory management problems. This is primarily because Python uses in Data Science and Machine Learning and can handle a large volume of data. Additionally, Python depends on its built-in Memory Management system by default rather than relying on the user to do it.

Because Python code runs within containers managed by a distributed processing framework, each container has a limited amount of RAM available to it. If the code execution exceeds the memory limit, the container will be forced to shut down automatically. Thus, memory problems occur throughout the development process at this point.

On the other hand, this isn't always the case. There are times when coders are entirely baffled as to what is going on. Perhaps an object is tethered to a reference when it isn't meant to be, and the weight of the thing accumulates over time. When it reaches its zenith, memory difficulties begin to emerge.

The quickest and most straightforward approach is to raise the RAM allotment. However, this is not a realistic solution because it may result in a waste of valuable resources. Additionally, unanticipated memory spikes may endanger the overall stability of the program.

As a result, we require the assistance of Python memory profilers. The goal of Python memory profilers is to identify memory leaks and optimize the use of memory in your Python code. In addition, these sorts of Python memory profilers can determine the space efficiency of the code and packagesy.

Memory Profiler

There are a few Python Memory Profilers such as Pympler, Guppy3, Fil, Memory Profiler, and more. But in this article, we will learn about “Python Profiling memory usage with memory profiler”.

Memory Profiler is a python module that depends on psutil. It can be used for monitoring memory consumption in a process or we can use it for a line-by-line analysis of the memory consumption of our codes.

Where psutil is a library for retrieving information on running processes and system utilization (CPU, memory, disks, network, sensors) in Python.

Installation Steps

Step 1 – Install via pip:

pip3 install memory_profiler

Step 2 – Add First Code

import memory_profiler

Step 3 – Define function, we need to use our decorator on the function that we are profiling.

@memory_profiler.profile

def big_array():

    x = [1] * (10**5)

    y = [2] * (10**7)

Step 4 – Then, we use a del keyword to delete the memory used by array y.

    del y

Step 5 – Then,

    return x

Step 6 – Call Function

if __name__== '__main__':

Step 7 – Call Function, Big array

    big_array()

Step 8 – We need to run our code with python3 and have to add a parameter on the command line to load the module so it can run against our script. If the file name is example.py, this would result in:

python3 - u -m memory_profiler example.py

Step 9 – Hit Enter and Output will as follow:

Line #    Mem usage    Increment  Occurences   Line Contents

============================================================

     3   38.0 MiB   38.0 MiB           1       @memory_profiler.profile

     4                                             def big_array():

     5   38.7 MiB    0.7 MiB           1           x = [1] * (10**5)

     6  115.0 MiB   76.3 MiB           1           y = [2] * (10**7)

     7   38.8 MiB  -76.2 MiB           1           del y

     8   38.8 MiB    0.0 MiB           1           return x

Profiling of the Single Function

The open-source memory-profiler package is the quickest and most convenient way to profile a single method or function. It's similar to the line profiler, which I've previously talked about in this blog post.

The @profile decorator may be placed around any function or method, and then the command python -m memory profiler myscript may be used to execute the memory profiling. Once your script has finished running, you will monitor the memory use line by line.

This is incredibly valuable if you want to profile a portion of memory-intensive code. Still, it won't assist you much if you don't know where most of the memory is being consumed. In such a situation, a more high-level approach to profiling is required first before proceeding.