< Back

February 11th, 2023

Techniques for Improving the performance of your Django app

{ Engineering }

Techniques for Improving the performance of your Django app

Daniel Coker
by Daniel Coker

Introduction

According to research, users expect a website to load in two seconds or less. If it doesn’t load within this time frame, there’s a good chance they’ll leave. We put a lot of effort into attracting users to try out our app offerings for managing their wealth; it would be a shame to lose them because our app is slow.

Every user is important to us. The app slowing down for a few more milliseconds could be the reason we lose a user, which can also have an impact on our north star metric — Assets Under Management (AUM). Cowrywise takes pride in the fact that we can provide our users with a swift and enjoyable experience while using our app. Keeping this going is always on our minds. This pushes us to constantly monitor and improve the performance of our app.

To improve performance, you must first understand how the app is currently performing. In this regard, an Application Performance Management/Monitoring (APM) tool is useful. Elastic APM is our performance monitoring tool of choice.

Elastic APM provides us with a lot of insights about how the app is performing, particularly two key metrics related to performance and speed — latency and throughput.

In this article, we’ll look at some of the techniques we use to improve our app’s performance and speed.

Optimize Database Queries

It is critical to make inexpensive database queries in order to increase performance and speed. Our APM tool reveals which requests take the most time. The trace information from these requests may indicate that we perform too many database queries or that some database queries take too long. This is improved in two ways:

  • Creating database indexes
  • Using select_related and prefetch_related

Creating Database Indexes

When a database table grows in size, an index is often always necessary. Indexes speed up database retrieval queries. It optimises the database in such a way that data can be retrieved quickly.

When we discover that a database query takes a long time using our APM tool. The query is most likely searching through a large table using an unindexed field. To fix this, we create an index on this field, and the query speed increases almost magically.

For example, imagine we have a table that stores transactions and each transaction has a reference and status. To update the status of a transaction, we must retrieve it by its reference. Given how large this transactions table can be, retrieving a single transaction from it can take a long time. To improve the database retrieval query speed, we create an index on the transaction reference field, like this:

CREATE INDEX idx_transactions_transaction_reference ON transactions (transaction_reference)

Django also lets us create indexes from models. In the Django model, we can use the db_index option in the field definition to create an index on the transaction reference field, like this:

class Transaction(models.Model):
  ...
  transaction_reference = models.CharField(max_length=20, db_index=True)
  ...

Creating this index will make fetching a transaction by its reference much faster.

Indexes are useful for increasing retrieval speed, but they should only be used when necessary. Indexes are designed to be optimised for database read operations. Write operations may become slower if there are too many indexes. As a result, indexes should not be used everywhere. Learn more about database indexes here.

Using select_related and prefetch_related

select_related and prefetch_related helps us to solve the N+1 query problem. An N+1 query problem is a performance anti-pattern that involves running a query for each result returned by a previous query. The trace information from our APM tool can give us some indications that an N+1 query problem exists. For example, a single request to retrieve a list of items resulting in multiple database queries being triggered to retrieve additional information about each item could be an indication of an N+1 query problem.

Let’s use the previous example’s transactions table to help us understand the problem. Assume we want to retrieve all transactions and then print the plan name associated with each one. We do this:

transactions = Transaction.objects.order_by("-created_on")

for transaction in transactions:
  print(f"{transaction.reference} - {transaction.plan.name})

In this simple code snippet, we have a Transaction Queryset with a for loop that iterates over it and prints the transaction reference and plan name associated with it. This shows that there are two models, Transaction and Plan. The Plan model is related to the Transaction. Let’s see how many queries this code snippet generates.

We create a Transaction Queryset and iterate through the transactions. The Queryset is evaluated and the result is fetched. We now have one query. Then, for each loop iteration, we print the reference and the plan name. Because the reference is on the Transaction model, no additional queries will be made because it was fetched in the first query.

However, the plan name is going to be different because the plan is a foreign key. Django will have to execute another query to get the corresponding plan for the given plan_id. At the end of the loop, we’d have N queries to get all the plans. Where N is the number of transactions. Adding the N queries with the initial transactions query gives us a total of N+1 queries.

You can begin to see the problem with this when you Imagine us having 1m transactions.

To solve this N+1 query problem, we can use the select_related and prefetch_related Queryset methods provided by Django.

select_related returns a Queryset containing the data of the related object when a query is executed. As a result, Django will not need to run a new query for each result. Using this method with our previous code will give us this:

transactions = Transaction.objects.select_related("plan").order_by("-created_on")

for transaction in transactions:
  print(f"{transaction.reference} - {transaction.plan.name})

We will now have only one query that selects the transaction and its related plans.

prefetch_related returns a Queryset that will automatically retrieve related objects for each of the specified lookups in a single batch. Using this method with our previous code will give us this:

transactions = Transaction.objects.prefetch_related("plan").order_by("-created_on")

for transaction in transactions:
  print(f"{transaction.reference} - {transaction.plan.name})

This will execute two queries. One query returns all transactions, while the other query returns all plans that appeared in the transactions. The number of queries is reduced from N+1 to just two.

Both of these methods achieve the same goal of reducing the number of queries required to get related objects, but they use different strategies. select_related works by creating an SQL join and including the fields of the related object in the SELECT statement. It is suitable for foreign key and one-to-one relationships. prefetch_related, on the other hand, performs a separate lookup for each relationship and performs the ‘joining’ in Python. This works well in both many-to-many and many-to-one relationships.

Use Asynchronous Tasks

There are usually some long-running tasks involved in the process of serving a request. For example, sending an email after an action is done on the app by the user. Users typically do not want to wait for these types of tasks to be completed before moving on to performing other actions on the app. One solution is to move the long-running task to the background. As a result, the app can continue to respond quickly to users while the expensive, long-running tasks are completed asynchronously in the background.

Offloading the long-running tasks to the background allows users to benefit from faster response time and allows us to benefit from parallel processing.

Sending emails is a very common use case for us. Several Cowrywise app actions end with us sending an email. For example, after a user top up their plan, transfer their matured savings to their Stash, log into their account, or join a new Savings Circle, we typically send an email. To make these actions a bit faster, we offload sending these emails to the background.

Another use case for asynchronous tasks on the Cowrywise app is when a user requests their plan statement. This action involves retrieving the plan transactions, converting them to a downloadable format, and then sending an email with a download link to the user. We definitely don’t want the user to have to wait for all of these to happen before doing anything else on the app. This will be an unpleasant experience for them. Instead, we receive the request, respond with a success message informing them to check their email for a download link, and then complete the rest of the process in the background.

Long-running tasks can be offloaded to be processed in the background, which is a very efficient technique for improving performance and speed. Celery helps us in orchestrating our asynchronous tasks. There are numerous articles explaining how to use Celery with Django. This is one of them here.

Cache Expensive Operations

Another way we speed up our app is to cache frequently requested data and data that do not change for each user. As the application grows, retrieving data from the database can become expensive and slow; caching allows us to avoid running queries multiple times. Data that is cached is easier and faster to retrieve.

Caching is the process by which fetched data is stored in a cache so that subsequent requests fetch from the cache instead of from the source.

Sentry

Caches are used in a variety of ways in the Cowrywise app. The Savings Circles Leaderboard is one of them. Each Savings Circle has a leaderboard. The leaderboard displays the Circle members and their ranks (depending on how much they have saved). If the Circle has a large number of members, computing the leaderboard could be an expensive operation.

Given the leaderboard is frequently viewed on the app, imagine us computing the leaderboard every time a user wants to view the leaderboard on the app. It has the potential of significantly reducing speed. This gives us the opportunity to make use of caching. We cache the Circle’s leaderboard and update the cache when a member makes a successful transaction. As a result, the leaderboard will no longer need to be computed each time a user wants to view it. It can always be retrieved from the cache.

Looking for frequently requested data or expensive operations in your app and caching for faster retrievals has the potential to significantly increase the performance and speed of your app.

Conclusion

Making sure our users have a quick and seamless experience is always on our minds, which is why we are constantly looking for performance improvement opportunities. Having a high-performant app allows us to better serve our customers while also increasing our chances of meeting our business objectives. There are numerous other performance enhancement techniques that can be implemented, but we hope you find these ones useful.