$ cat /posts/django-pagination-handling-large-data-sets.md

Django Pagination: Handling Large Data Sets

drwxr-xr-x2026-01-235 min0 views

Pagination divides large datasets into manageable pages improving page load times, reducing server load, and enhancing user experience by displaying content in digestible chunks. Without pagination, pages displaying thousands of records suffer from slow queries, excessive memory usage, poor rendering performance, and overwhelming user interfaces. Django's pagination framework provides simple yet powerful tools for splitting QuerySets into pages with customizable page sizes, navigation controls, and integration with class-based views. This comprehensive guide explores Django 6.0 pagination including using the Paginator class for manual pagination, implementing ListView pagination in class-based views, creating custom pagination templates with previous and next links, handling edge cases like empty pages and invalid page numbers, implementing AJAX pagination for seamless navigation, adding page number navigation and page ranges, optimizing pagination queries with select_related and prefetch_related, and best practices for pagination design. Mastering pagination enables building scalable applications handling millions of records efficiently while maintaining fast response times and excellent user experiences.

Using Paginator Class

Django's Paginator class handles pagination logic splitting QuerySets or lists into Page objects. The Paginator accepts any sequence supporting count and slicing including QuerySets, lists, and tuples. Each Page object contains items for that page plus navigation methods checking for previous and next pages. Understanding Paginator properties like count, num_pages, and page_range enables building comprehensive pagination interfaces.

pythonpaginator_basics.py

# Basic pagination with Paginator
from django.core.paginator import Paginator, EmptyPage, PageNotAnInteger
from django.shortcuts import render
from .models import Article

def article_list(request):
    # Get all articles
    article_list = Article.objects.all().order_by('-created_at')
    
    # Create Paginator instance (25 items per page)
    paginator = Paginator(article_list, 25)
    
    # Get page number from query parameter
    page_number = request.GET.get('page', 1)
    
    try:
        # Get page object
        page_obj = paginator.get_page(page_number)
    except PageNotAnInteger:
        # If page is not an integer, deliver first page
        page_obj = paginator.get_page(1)
    except EmptyPage:
        # If page is out of range, deliver last page
        page_obj = paginator.get_page(paginator.num_pages)
    
    return render(request, 'article_list.html', {'page_obj': page_obj})

# Paginator properties and methods
paginator = Paginator(queryset, 25)

# Total number of items
print(paginator.count)  # e.g., 150

# Number of pages
print(paginator.num_pages)  # e.g., 6 (150/25)

# Range of page numbers
print(list(paginator.page_range))  # [1, 2, 3, 4, 5, 6]

# Get specific page
page = paginator.get_page(1)

# Page object properties
print(page.number)  # Current page number
print(page.object_list)  # Items on this page
print(page.has_previous())  # True if previous page exists
print(page.has_next())  # True if next page exists
print(page.has_other_pages())  # True if more than one page

if page.has_previous():
    print(page.previous_page_number())  # Previous page number

if page.has_next():
    print(page.next_page_number())  # Next page number

# Safe get_page method (Django 2.0+)
page = paginator.get_page(page_number)
# Returns first page for invalid numbers, never raises exception

# Custom per-page parameter
def article_list_custom(request):
    articles = Article.objects.all()
    per_page = request.GET.get('per_page', 25)
    
    # Limit per_page to reasonable values
    try:
        per_page = int(per_page)
        per_page = max(10, min(per_page, 100))  # Between 10 and 100
    except ValueError:
        per_page = 25
    
    paginator = Paginator(articles, per_page)
    page_obj = paginator.get_page(request.GET.get('page', 1))
    
    return render(request, 'article_list.html', {
        'page_obj': page_obj,
        'per_page': per_page
    })

ListView Pagination

Django's ListView class-based view provides built-in pagination support through the paginate_by attribute. Setting paginate_by automatically creates a Paginator and provides page_obj context variable simplifying pagination implementation. ListView handles page parameter extraction, invalid page handling, and context generation requiring minimal code for paginated lists.

pythonlistview_pagination.py

# Class-based view pagination
from django.views.generic import ListView
from .models import Article

class ArticleListView(ListView):
    model = Article
    template_name = 'article_list.html'
    context_object_name = 'articles'
    paginate_by = 25  # Items per page
    ordering = ['-created_at']
    
    # Optional: Custom queryset
    def get_queryset(self):
        queryset = super().get_queryset()
        # Add filtering or additional queries
        return queryset.select_related('author')
    
    # Optional: Add extra context
    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        # page_obj is automatically available
        # paginator is also available
        context['total_articles'] = self.get_queryset().count()
        return context

# Dynamic paginate_by
class CustomArticleListView(ListView):
    model = Article
    template_name = 'article_list.html'
    
    def get_paginate_by(self, queryset):
        # Allow users to choose page size
        per_page = self.request.GET.get('per_page', 25)
        try:
            per_page = int(per_page)
            return max(10, min(per_page, 100))
        except ValueError:
            return 25

# Pagination with filtering
class FilteredArticleListView(ListView):
    model = Article
    template_name = 'article_list.html'
    paginate_by = 20
    
    def get_queryset(self):
        queryset = super().get_queryset()
        
        # Filter by category
        category = self.request.GET.get('category')
        if category:
            queryset = queryset.filter(category=category)
        
        # Search
        search = self.request.GET.get('q')
        if search:
            queryset = queryset.filter(title__icontains=search)
        
        return queryset.order_by('-created_at')
    
    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        # Preserve query parameters in pagination links
        query_params = self.request.GET.copy()
        if 'page' in query_params:
            del query_params['page']
        context['query_string'] = query_params.urlencode()
        return context

# urls.py
from django.urls import path
from .views import ArticleListView

urlpatterns = [
    path('articles/', ArticleListView.as_view(), name='article-list'),
]

Pagination Templates

Pagination templates display page navigation controls including previous and next buttons, page numbers, page ranges, and current page indicators. Django provides page_obj context variable containing pagination state enabling template logic for navigation. Effective pagination UI shows current page, total pages, accessible page numbers, and disabled states for unavailable navigation.

htmlpagination_templates.html

<!-- Basic pagination template -->
<!-- article_list.html -->
{% for article in page_obj %}
    <div class="article">
        <h2>{{ article.title }}</h2>
        <p>{{ article.excerpt }}</p>
    </div>
{% endfor %}

<!-- Simple pagination controls -->
<div class="pagination">
    {% if page_obj.has_previous %}
        <a href="?page=1">First</a>
        <a href="?page={{ page_obj.previous_page_number }}">Previous</a>
    {% endif %}
    
    <span class="current-page">
        Page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }}
    </span>
    
    {% if page_obj.has_next %}
        <a href="?page={{ page_obj.next_page_number }}">Next</a>
        <a href="?page={{ page_obj.paginator.num_pages }}">Last</a>
    {% endif %}
</div>

<!-- Advanced pagination with page numbers -->
<div class="pagination">
    {% if page_obj.has_previous %}
        <a href="?page={{ page_obj.previous_page_number }}" class="btn">Previous</a>
    {% else %}
        <span class="btn disabled">Previous</span>
    {% endif %}
    
    {% for num in page_obj.paginator.page_range %}
        {% if page_obj.number == num %}
            <span class="btn current">{{ num }}</span>
        {% elif num > page_obj.number|add:'-3' and num < page_obj.number|add:'3' %}
            <a href="?page={{ num }}" class="btn">{{ num }}</a>
        {% endif %}
    {% endfor %}
    
    {% if page_obj.has_next %}
        <a href="?page={{ page_obj.next_page_number }}" class="btn">Next</a>
    {% else %}
        <span class="btn disabled">Next</span>
    {% endif %}
</div>

<!-- Pagination with query parameters -->
<div class="pagination">
    {% if page_obj.has_previous %}
        <a href="?{{ query_string }}&page={{ page_obj.previous_page_number }}">Previous</a>
    {% endif %}
    
    {% for num in page_obj.paginator.page_range %}
        {% if page_obj.number == num %}
            <strong>{{ num }}</strong>
        {% else %}
            <a href="?{{ query_string }}&page={{ num }}">{{ num }}</a>
        {% endif %}
    {% endfor %}
    
    {% if page_obj.has_next %}
        <a href="?{{ query_string }}&page={{ page_obj.next_page_number }}">Next</a>
    {% endif %}
</div>

<!-- Bootstrap 5 pagination -->
<nav aria-label="Page navigation">
    <ul class="pagination">
        {% if page_obj.has_previous %}
            <li class="page-item">
                <a class="page-link" href="?page=1">First</a>
            </li>
            <li class="page-item">
                <a class="page-link" href="?page={{ page_obj.previous_page_number }}">&laquo;</a>
            </li>
        {% else %}
            <li class="page-item disabled">
                <span class="page-link">&laquo;</span>
            </li>
        {% endif %}
        
        {% for num in page_obj.paginator.page_range %}
            {% if page_obj.number == num %}
                <li class="page-item active">
                    <span class="page-link">{{ num }}</span>
                </li>
            {% elif num > page_obj.number|add:'-3' and num < page_obj.number|add:'3' %}
                <li class="page-item">
                    <a class="page-link" href="?page={{ num }}">{{ num }}</a>
                </li>
            {% endif %}
        {% endfor %}
        
        {% if page_obj.has_next %}
            <li class="page-item">
                <a class="page-link" href="?page={{ page_obj.next_page_number }}">&raquo;</a>
            </li>
            <li class="page-item">
                <a class="page-link" href="?page={{ page_obj.paginator.num_pages }}">Last</a>
            </li>
        {% else %}
            <li class="page-item disabled">
                <span class="page-link">&raquo;</span>
            </li>
        {% endif %}
    </ul>
</nav>

<!-- Showing results info -->
<p class="pagination-info">
    Showing {{ page_obj.start_index }} to {{ page_obj.end_index }} of {{ page_obj.paginator.count }} results
</p>

For large datasets with many pages, limit displayed page numbers using filters like num > page_obj.number|add:'-3' to show only nearby pages preventing overwhelming navigation bars with hundreds of page links.

Custom Pagination Logic

Custom pagination logic handles advanced requirements including limiting displayed page ranges, adding ellipsis for skipped pages, implementing infinite scroll, or creating mobile-optimized pagination. Template tags or context processors encapsulate pagination logic enabling reusable pagination components across templates.

pythoncustom_pagination.py

# Custom pagination range
def get_page_range(page_obj, on_each_side=3, on_ends=2):
    """
    Return list of page numbers with ellipsis for large page ranges
    """
    paginator = page_obj.paginator
    current_page = page_obj.number
    
    # If few pages, show all
    if paginator.num_pages <= 10:
        return list(paginator.page_range)
    
    # Calculate visible ranges
    left_range = range(
        max(1, current_page - on_each_side),
        current_page
    )
    right_range = range(
        current_page,
        min(current_page + on_each_side, paginator.num_pages) + 1
    )
    
    # Build page list with ellipsis
    page_list = []
    
    # Start pages
    for i in range(1, min(on_ends + 1, paginator.num_pages + 1)):
        page_list.append(i)
    
    # Left ellipsis
    if current_page - on_each_side > on_ends + 1:
        page_list.append('...')
    
    # Middle range
    for i in left_range:
        if i > on_ends:
            page_list.append(i)
    
    for i in right_range:
        if i <= paginator.num_pages - on_ends and i not in page_list:
            page_list.append(i)
    
    # Right ellipsis
    if current_page + on_each_side < paginator.num_pages - on_ends:
        page_list.append('...')
    
    # End pages
    for i in range(max(paginator.num_pages - on_ends + 1, on_ends + 1), paginator.num_pages + 1):
        if i not in page_list:
            page_list.append(i)
    
    return page_list

# Custom template tag
# templatetags/pagination_tags.py
from django import template

register = template.Library()

@register.inclusion_tag('pagination/pagination.html', takes_context=True)
def paginate(context, page_obj):
    return {
        'page_obj': page_obj,
        'request': context['request'],
        'page_range': get_page_range(page_obj)
    }

# Usage in template:
# {% load pagination_tags %}
# {% paginate page_obj %}

# AJAX pagination view
from django.http import JsonResponse
from django.template.loader import render_to_string

def ajax_article_list(request):
    page_number = request.GET.get('page', 1)
    articles = Article.objects.all().order_by('-created_at')
    paginator = Paginator(articles, 10)
    page_obj = paginator.get_page(page_number)
    
    if request.headers.get('X-Requested-With') == 'XMLHttpRequest':
        html = render_to_string('article_list_partial.html', {
            'articles': page_obj
        })
        return JsonResponse({
            'html': html,
            'has_next': page_obj.has_next(),
            'next_page': page_obj.next_page_number() if page_obj.has_next() else None
        })
    
    return render(request, 'article_list.html', {'page_obj': page_obj})

# JavaScript for infinite scroll
"""
let loading = false;
let page = 2;

window.addEventListener('scroll', function() {
    if (loading) return;
    
    if (window.innerHeight + window.scrollY >= document.body.offsetHeight - 500) {
        loading = true;
        
        fetch(`/articles/?page=${page}`, {
            headers: {'X-Requested-With': 'XMLHttpRequest'}
        })
        .then(response => response.json())
        .then(data => {
            document.querySelector('#article-list').insertAdjacentHTML('beforeend', data.html);
            if (data.has_next) {
                page++;
                loading = false;
            }
        });
    }
});
"""

Pagination Best Practices

Optimize queries: Use select_related and prefetch_related with paginated QuerySets reducing database queries
Reasonable page sizes: Choose page sizes between 10-50 items balancing load times with user experience
Preserve query parameters: Maintain filters and search terms in pagination links ensuring consistent user experience
Handle edge cases: Use get_page() instead of page() for safer pagination handling invalid page numbers gracefully
Limit displayed pages: Show limited page ranges for large datasets preventing overwhelming navigation bars
Add loading indicators: Provide visual feedback during page loads improving perceived performance

Conclusion

Django's pagination framework provides robust tools for handling large datasets improving performance and user experience through manageable page sizes. The Paginator class handles pagination logic splitting QuerySets into pages with comprehensive navigation support. ListView integration through paginate_by attribute simplifies pagination implementation in class-based views requiring minimal configuration. Pagination templates display navigation controls including previous and next buttons, page numbers, and current page indicators. Custom pagination logic implements advanced features including page range limiting, ellipsis for skipped pages, and infinite scroll patterns. Optimizing paginated queries with select_related and prefetch_related reduces database overhead maintaining fast response times. Following best practices including reasonable page sizes, query parameter preservation, safe page handling, limited page ranges, and loading indicators ensures excellent user experiences. Understanding pagination enables building scalable applications efficiently displaying millions of records while maintaining performance throughout Django 6.0 development.