Django: Development to Deployment (Part 2)

Update: Part 3 is out now: Deploying to AWS.

In Part 1 of our series I covered setting up your local environment using Vagrant for virtualization and Fabric in combination with several other tools to do the bootstrapping. In Part 2, I’ll cover my Django setup and the development process.

Django Settings

At the end of part 1 we created a blank project and had a web server running that allowed us to reach the congratulations page. The next step of our setup is to setup Django, mostly by modifying our settings.py file. This file controls all the settings for Django, and if very important.

The first thing I do is add some helper methods to resolve the absolute path to the current directory. This has an edge case related to how we use symlinks inside Vagrant, that we need to check for, but is otherwise straightforward:


import os
import os.path
import sys
import datetime

##### Path Resolution

# Get the current path
BASE_PATH = os.path.dirname(os.path.abspath(__file__))

# Hack for vagrant
if BASE_PATH == "/project/project":
  BASE_PATH = "/server/env.example.com/project/project"

# Makes a normalized path from the base path
def make_abs_path(*rel_path):
  args = (BASE_PATH,) + rel_path
  return os.path.normpath(os.path.join(*args))

# Modify the python path
sys.path.append(make_abs_path("core/"))

# Make the tmp paths
TMP_PATHS = ["django","query"]
for p in TMP_PATHS:
  path = make_abs_path("../../tmp/",p)
  if not os.path.exists(path):
    os.makedirs(path)

# Get the date string
NOW = datetime.datetime.now()
DATE_STR = NOW.strftime("%Y-%m-%d")

Once we have these settings, we can use the make_abs_path() method to generate our paths. This is used in several places, such as for template locations, static media, logging, etc. Many of the other settings are pretty typical for Django, but one common requirement is to support multiple “environments”. You may want to have separate settings for development, staging, and production. Clearly, you wouldn’t want to use the same databases or caches. To support this, I check for the existence of two files to help determine the current environment. If a file named PRODUCTION is in the project folder, we set our environment to production, and import production settings. Likewise, if we see a STAGING file, we do the same. In the absence of these files, we load our development settings.


# Check which environment to load
if os.path.exists(make_abs_path("PRODUCTION")):
  from settings_prod import *
  ENVIRONMENT = "PRODUCTION"
elif os.path.exists(make_abs_path("STAGING")):
  from settings_stage import *
  ENVIRONMENT = "STAGING"
else:
  from settings_dev import *
  ENVIRONMENT = "DEV"

I prefer this approach due to its simplicity. I create a blank file with the appropriate name and only add it to the correct git branch, and the proper settings are loaded. Here are the steps to create a new staging and release branch, with the files that trigger the environmental settings:


$ git checkout -b stage
$ touch project/STAGING
$ git add project/STAGING
$ git commit -m "Add staging file"
$ git push -u origin stage
$ git checkout -b release
$ git mv project/STAGING project/PRODUCTION
$ git commit -m "Add production file"
$ git push -u origin release
$ git checkout master

Our staging machines use the stage branch from the git repo, and our production machines use the release branch. Now that we have added the proper files to each of the branches, the proper settings will be loaded for each environment. Usually, the only settings that need to be set per environment are for the database and caches. Here is an example of the production settings:


DEBUG = False
TEMPLATE_DEBUG = False

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'exampleproddb',
        'USER': 'produser',
        'PASSWORD': 'ExampleProdPass',
        'HOST': 'db.example.com',
        'PORT': '',
    }
}

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': ['memcache.example.com:11211'],
        'KEY_PREFIX': 'PROD',
    }
}

It is critical to disable DEBUG on production for performance reasons. In addition, we use a standard MySQL database configuration and a Memcached backend. For caches, always set a KEY_PREFIX to prevent staging or development environments from clobbering the cached production data. Once Django has been configured you can create your first app.

Apps and Templates

In Django, an app is merely an organizational unit. An app is nothing more than a collection of views and models. I try to maintain a clean folder structure by placing apps in a subdirectory of the project folder. As you can see in our example repository, I’ve created a folder in project called apps/ and added our main app in there. I’m not going to cover the details of using Django, so suffice it to say that our app renders only the index page using a template:


from django.shortcuts import render_to_response
from django.template import RequestContext

def index(request):
  return render_to_response("home/index.html", context_instance=RequestContext(request))

We use the convenience method render_to_response to handle template rendering. We’ve configured Django to look in the project/templates/ directory for templates, so index.html should be inside project/templates/home/. It is very standard to have a base template (project/templates/base.html), and then have [templates](https://github.com/armon/DjangoProjectExample/blob/master/project/templates/home/index.html) which extend the base template with the page specific content. This is a simple method of maintaining a unified look while respecting the DRY principle.

Models and Migrations

We now have a very simple app that is able to generate our index page and show off our lolcat. However, we can’t do much beyond serving this simple static page. Most non-trivial applications will need to make use of a database to provide a persistent datastore. In Django, this is done by writing Models which are built on top of Django’s ORM framework. Models provide a convenient and simple means of accessing our data, and are automatically transformed into tables and rows in our database.

One annoyance with Django out of the box is the inability to change the schema. Lets say you define a Person model with age and name. You may later decide that you would also like to store their gender. Django provides no good solution to this problem, however there is a third-party plugin called South which solves it. South provides simple schema migrations and allows you to be somewhat more flexible and adaptive. I highly recommend it and use it for every project.

The only change that is made when you adopt South, is that instead of using the typical syncdb command, you must now use the migrate command. This command might need to apply multiple migrations to bring the database up to the current state. As an example, we might define our model as:


from django.db import models

class Person(models.Model):
  name = models.CharField(max_length=64,db_index=True)
  age = models.IntegerField(max_length=512)

Once we have our initial model, we need to instruct South to create our initial schema. This is made slightly more complicated due to vagrant and virtualenv, but is nonetheless simple:


$ vagrant ssh
$ cd /server/env.example.com/
$ source bin/activate
$ cd project/project/
$ python manage.py schemamigration --initial main
Creating migrations directory at '/project/project/apps/main/migrations'...
Creating __init__.py in '/project/project/apps/main/migrations'...
 + Added model main.Person
Created 0001_initial.py. You can now apply this migration with: ./manage.py migrate main
$ python manage.py syncdb ; python manage.py migrate

After we run all these commands, we will have created our first migration and applied it to create the table. At this point, we might decide to extend our Person to add a gender.


class Person(models.Model):
  name = models.CharField(max_length=64,db_index=True)
  age = models.IntegerField(max_length=512)
  gender = models.CharField(max_length=6, null=True, default=None)

After we do this, we can generate a new migration that will add the column to our table. This is all done automatically by South, we just need to instruct it to generate the necessary files and apply them:


$ python manage.py schemamigration --auto main
 + Added field gender on main.Person
Created 0002_auto__add_field_person_gender.py. You can now apply this migration with: ./manage.py migrate main
$ python manage.py migrate
Running migrations for main:
 - Migrating forwards to 0002_auto__add_field_person_gender.
 > main:0002_auto__add_field_person_gender
 - Loading initial data for main.

As you can see, South is very simple to use but also is extremely useful in allowing us to change our schema. In addition to supporting forward migrations, it can also handle reverse migrations to rollback an unwanted change.

In our examples, we needed to manually SSH in to apply our migrations. This can be a bit tedious, so there is a Fabric task to handle it. If all we need to do is run syncdb and migrate, which are usually done after deploying new code, then we can use the syncdb task:


$ fab vagrant syncdb

This will run both tasks without needed to SSH and activate virtualenv.

Development Server

Our current setup uses Nginx to handle the incoming requests and uWSGI as our application server. However, uWSGI is designed to be performant and as such it caches our application code in-memory. This means that when we update our python files our changes are not reflected until uWSGI is restarted or reloaded (to reload uWSGI send the HUP signal to it). For rapid development this can be inconvenient since we need to constantly restart it. To minimize the overhead of development, we can run the Django development server which automatically reloads on code change. This allows us to make changes and immediately see the results.

To do this, we just use a Fabric task to start the development server:


$ fab vagrant dev_server

To use this server, we just use local port 7000 instead of 8080. Where port 8080 will use uWSGI to serve requests, port 7000 is used in development only and uses the dev server to handle requests. We don’t use the development server in production because it is not particularly fast or efficient.

Summary

In Part 2 I tried to cover some of the tips and tricks for development in our environment. We covered how to configure Django settings such that we can control the settings on a per-environment basis and easily generate absolute paths. Django apps are placed in a sub-folder and templates in a separate folder to keep the folder structure clean. South was introduced as a simple tool for enabling schema migrations. Lastly, we covered using the Django development server for more rapid development.

I hope some of this was useful as an example of how to structure a Django project, and some tricks for development. In the next part we will cover using Git to manage the stage and release branches and deploying our project onto Amazon EC2.

Resources:

  1. armondadgar posted this