Django offers a wide variety of built-in lookups for filtering (for example,
icontains). This documentation explains how to write custom lookups and how to alter the working of existing lookups. For the API references of lookups, see the Lookup API reference.
Let’s start with a simple custom lookup. We will write a custom lookup
ne which works opposite to
Author.objects.filter(name__ne='Jack') will translate to the SQL:
"author"."name" <> 'Jack'
This SQL is backend independent, so we don’t need to worry about different databases.
There are two steps to making this work. Firstly we need to implement the lookup, then we need to tell Django about it. The implementation is quite straightforward:
from django.db.models import Lookup class NotEqual(Lookup): lookup_name = 'ne' def as_sql(self, compiler, connection): lhs, lhs_params = self.process_lhs(compiler, connection) rhs, rhs_params = self.process_rhs(compiler, connection) params = lhs_params + rhs_params return '%s <> %s' % (lhs, rhs), params
To register the
NotEqual lookup we will just need to call
register_lookup on the field class we want the lookup to be available. In this case, the lookup makes sense on all
Field subclasses, so we register it with
from django.db.models.fields import Field Field.register_lookup(NotEqual)
Lookup registration can also be done using a decorator pattern:
from django.db.models.fields import Field @Field.register_lookup class NotEqualLookup(Lookup): # ...
We can now use
foo__ne for any field
foo. You will need to ensure that this registration happens before you try to create any querysets using it. You could place the implementation in a
models.py file, or register the lookup in the
ready() method of an
Taking a closer look at the implementation, the first required attribute is
lookup_name. This allows the ORM to understand how to interpret
name__ne and use
NotEqual to generate the SQL. By convention, these names are always lowercase strings containing only letters, but the only hard requirement is that it must not contain the string
We then need to define the
as_sql method. This takes a
SQLCompiler object, called
compiler, and the active database connection.
SQLCompiler objects are not documented, but the only thing we need to know about them is that they have a
compile() method which returns a tuple containing an SQL string, and the parameters to be interpolated into that string. In most cases, you don’t need to use it directly and can pass it on to
Lookup works against two values,
rhs, standing for left-hand side and right-hand side. The left-hand side is usually a field reference, but it can be anything implementing the query expression API. The right-hand is the value given by the user. In the example
Author.objects.filter(name__ne='Jack'), the left-hand side is a reference to the
name field of the
Author model, and
'Jack' is the right-hand side.
process_rhs to convert them into the values we need for SQL using the
compiler object described before. These methods return tuples containing some SQL and the parameters to be interpolated into that SQL, just as we need to return from our
as_sql method. In the above example,
('"author"."name"', ) and
('"%s"', ['Jack']). In this example there were no parameters for the left hand side, but this would depend on the object we have, so we still need to include them in the parameters we return.
Finally we combine the parts into an SQL expression with
<>, and supply all the parameters for the query. We then return a tuple containing the generated SQL string and the parameters.
The custom lookup above is great, but in some cases you may want to be able to chain lookups together. For example, let’s suppose we are building an application where we want to make use of the
abs() operator. We have an
Experiment model which records a start value, end value, and the change (start - end). We would like to find all experiments where the change was equal to a certain amount (
Experiment.objects.filter(change__abs=27)), or where it did not exceed a certain amount (
This example is somewhat contrived, but it nicely demonstrates the range of functionality which is possible in a database backend independent manner, and without duplicating functionality already in Django.
We will start by writing an
AbsoluteValue transformer. This will use the SQL function
ABS() to transform the value before comparison:
from django.db.models import Transform class AbsoluteValue(Transform): lookup_name = 'abs' function = 'ABS'
Next, let’s register it for
from django.db.models import IntegerField IntegerField.register_lookup(AbsoluteValue)
We can now run the queries we had before.
Experiment.objects.filter(change__abs=27) will generate the following SQL:
SELECT ... WHERE ABS("experiments"."change") = 27
Transform instead of
Lookup it means we are able to chain further lookups afterwards. So
Experiment.objects.filter(change__abs__lt=27) will generate the following SQL:
SELECT ... WHERE ABS("experiments"."change") < 27
Note that in case there is no other lookup specified, Django interprets
This also allows the result to be used in
ORDER BY and
DISTINCT ON clauses. For example
SELECT ... ORDER BY ABS("experiments"."change") ASC
And on databases that support distinct on fields (such as PostgreSQL),
SELECT ... DISTINCT ON ABS("experiments"."change")
Ordering and distinct support as described in the last two paragraphs was added.
When looking for which lookups are allowable after the
Transform has been applied, Django uses the
output_field attribute. We didn’t need to specify this here as it didn’t change, but supposing we were applying
AbsoluteValue to some field which represents a more complex type (for example a point relative to an origin, or a complex number) then we may have wanted to specify that the transform returns a
FloatField type for further lookups. This can be done by adding an
output_field attribute to the transform:
from django.db.models import FloatField, Transform class AbsoluteValue(Transform): lookup_name = 'abs' function = 'ABS' @property def output_field(self): return FloatField()
This ensures that further lookups like
abs__lte behave as they would for a
When using the above written
abs lookup, the SQL produced will not use indexes efficiently in some cases. In particular, when we use
change__abs__lt=27, this is equivalent to
change__lt=27. (For the
lte case we could use the SQL
So we would like
Experiment.objects.filter(change__abs__lt=27) to generate the following SQL:
SELECT .. WHERE "experiments"."change" < 27 AND "experiments"."change" > -27
The implementation is:
from django.db.models import Lookup class AbsoluteValueLessThan(Lookup): lookup_name = 'lt' def as_sql(self, compiler, connection): lhs, lhs_params = compiler.compile(self.lhs.lhs) rhs, rhs_params = self.process_rhs(compiler, connection) params = lhs_params + rhs_params + lhs_params + rhs_params return '%s < %s AND %s > -%s' % (lhs, rhs, lhs, rhs), params AbsoluteValue.register_lookup(AbsoluteValueLessThan)
There are a couple of notable things going on. First,
AbsoluteValueLessThan isn’t calling
process_lhs(). Instead it skips the transformation of the
lhs done by
AbsoluteValue and uses the original
lhs. That is, we want to get
ABS("experiments"."change"). Referring directly to
self.lhs.lhs is safe as
AbsoluteValueLessThan can be accessed only from the
AbsoluteValue lookup, that is the
lhs is always an instance of
Notice also that as both sides are used multiple times in the query the params need to contain
rhs_params multiple times.
The final query does the inversion (
-27) directly in the database. The reason for doing this is that if the
self.rhs is something else than a plain integer value (for example an
F() reference) we can’t do the transformations in Python.
In fact, most lookups with
__abs could be implemented as range queries like this, and on most database backends it is likely to be more sensible to do so as you can make use of the indexes. However with PostgreSQL you may want to add an index on
abs(change) which would allow these queries to be very efficient.
AbsoluteValue example we discussed previously is a transformation which applies to the left-hand side of the lookup. There may be some cases where you want the transformation to be applied to both the left-hand side and the right-hand side. For instance, if you want to filter a queryset based on the equality of the left and right-hand side insensitively to some SQL function.
Let’s examine the simple example of case-insensitive transformation here. This transformation isn’t very useful in practice as Django already comes with a bunch of built-in case-insensitive lookups, but it will be a nice demonstration of bilateral transformations in a database-agnostic way.
We define an
UpperCase transformer which uses the SQL function
UPPER() to transform the values before comparison. We define
bilateral = True to indicate that this transformation should apply to both
from django.db.models import Transform class UpperCase(Transform): lookup_name = 'upper' function = 'UPPER' bilateral = True
Next, let’s register it:
from django.db.models import CharField, TextField CharField.register_lookup(UpperCase) TextField.register_lookup(UpperCase)
Now, the queryset
Author.objects.filter(name__upper="doe") will generate a case insensitive query like this:
SELECT ... WHERE UPPER("author"."name") = UPPER('doe')
Sometimes different database vendors require different SQL for the same operation. For this example we will rewrite a custom implementation for MySQL for the NotEqual operator. Instead of
<> we will be using
!= operator. (Note that in reality almost all databases support both, including all the official databases supported by Django).
We can change the behavior on a specific backend by creating a subclass of
NotEqual with an
class MySQLNotEqual(NotEqual): def as_mysql(self, compiler, connection): lhs, lhs_params = self.process_lhs(compiler, connection) rhs, rhs_params = self.process_rhs(compiler, connection) params = lhs_params + rhs_params return '%s != %s' % (lhs, rhs), params Field.register_lookup(MySQLNotEqual)
We can then register it with
Field. It takes the place of the original
NotEqual class as it has the same
When compiling a query, Django first looks for
as_%s % connection.vendor methods, and then falls back to
as_sql. The vendor names for the in-built backends are
In some cases you may wish to dynamically change which
Lookup is returned based on the name passed in, rather than fixing it. As an example, you could have a field which stores coordinates or an arbitrary dimension, and wish to allow a syntax like
.filter(coords__x7=4) to return the objects where the 7th coordinate has value 4. In order to do this, you would override
get_lookup with something like:
class CoordinatesField(Field): def get_lookup(self, lookup_name): if lookup_name.startswith('x'): try: dimension = int(lookup_name[1:]) except ValueError: pass else: return get_coordinate_lookup(dimension) return super().get_lookup(lookup_name)
You would then define
get_coordinate_lookup appropriately to return a
Lookup subclass which handles the relevant value of
There is a similarly named method called
get_lookup() should always return a
Lookup subclass, and
Transform subclass. It is important to remember that
Transform objects can be further filtered on, and
Lookup objects cannot.
When filtering, if there is only one lookup name remaining to be resolved, we will look for a
Lookup. If there are multiple names, it will look for a
Transform. In the situation where there is only one name and a
Lookup is not found, we look for a
Transform and then the
exact lookup on that
Transform. All call sequences always end with a
Lookup. To clarify:
myfield.get_transform('mytransform'), and then
.filter(myfield__mytransform)will first call
myfield.get_lookup('mytransform'), which will fail, so it will fall back to calling
© Django Software Foundation and individual contributors
Licensed under the BSD License.