Serverless Django

Introduction

In the last few years, since around 2016, there’s a concept that’s been gaining popularity. This concept is known as “Serverless Infrastructure”.

Having a serverless infrastructure allows you to have a project running live, but without any permanent infrastructure. This means that when no requests are arriving for the server, there will be no servers actively running. When a request is received, the server will turn on, answer the request, and then shut down.

This serverless approach has one big benefit: server resources will only be in use while they are required, leaving in the past the problem of idle servers.

Providers

Nowadays there are different cloud providers that support this kind of infrastructure. Some of the most popular are:

1. Amazon Web Services (AWS)
2. Google Cloud
3. Microsoft Azure
4. IBM cloud

This tutorial will focus on AWS as it is one of the most widely used cloud providers in the world.

AWS provides serverless capabilities through one of their services called Lambda. Quoting AWS Lambda description:

“AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume - there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service - all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.”

Quite cool, right? Especially these two aspects:

• Takes care of running and scaling your code (no need to worry about the horizontal scale, Lambda has that covered).
• Only pay for what you use (save money by only paying for actual usage, no usage, no charge).

Frameworks

Regarding deploying Django apps with a serverless infrastructure, there are two big players that can help us:

• Serverless (Currently over 21900 stars on Github)
• Zappa (Currently over 6000 stars on Github)

Both frameworks are up to the task, but 'Zappa' is focused on serverless Python, while 'Serverless' allows you to deploy code of several programming languages like Node, Python, Java, PHP, among others.

For this tutorial, we will use Zappa for its simplicity and because it’s python focused.

Getting started

First, we need to make sure we have an AWS account with root access. If you don’t have an account with that level of permissions you will need to create an IAM user with programmatic access that has permissions to the following services:

• Cloud Formation
• IAM
• Lambda
• S3

• API Gateway
• CloudWatch
• Other services you would want to use

There is currently an open task in Zappa for a proper definition of the permissions needed. In the meanwhile, there are two approaches you can take: be flexible with the permissions or do some trial and error until reaching the minimal permissions you need for your app. It’s recommended to remove the IAM permission after Zappa creates the user the first time.

After the user is ready, make sure you have the proper awscli configuration, you can find quite a good guide on the Serverless page.

The Project

We will be deploying a simple project that was developed for this tutorial. The project itself is not really important, so please focus on learning how to add Zappa support to an existing Django project. The project can be found here.

First of all, let’s install the project:

#git clone https://github.com/innuy/innuy_lambda #pip install -r requirements/dev.txt

** You will have to create a bucket to locate your sqlite database, then update the database settings

Then let’s create a configuration file using the following command:

# zappa init

After running that command you'll be asked a few questions. I recommend you leave everything as default if it’s your first time.

A file called “zappa_settings.json” should have been created. It will look something like this:

{
"dev": { "aws_region": "us-east-1", "django_settings": "innuy_lambda.settings.dev", "profile_name": "default", "project_name": "innuy-lambda", "runtime": "python3.6", "s3_bucket": "bucketname" } }

Zappa has many additional configuration settings. If you need something more specific, please take a look at the documentation for more advanced use cases.

Let’s now run the following command to upload our code to Lambda:

# zappa deploy

If the deployment process succeeds you should see an url like this:

https://some_random_string.execute-api.us-east-1.amazonaws.com/dev

If you add “/admin” to the url the result is:

https://some_random_string.execute-api.us-east-1.amazonaws.com/dev/admin

You should be taken to the django admin… but static files are missing!

We can solve this by setting a few variables in the dev settings file (or you can do it by env variables in zappa). In this case just set the bucket where you want the statics files to live (if you don’t have a bucket, you can create one with a wizard in AWS S3):

AWS_STORAGE_BUCKET_NAME = 'innuylambda-static'

Then update the code in lambda using:

# zappa update

Now lets collect the static files:

# zappa manage dev "collectstatic --noinput"

You should now see the admin perfectly. You can try creating a user by running the project locally and running the createsuperuser command from manage.py.

Your app should be running completely serverless!!

To undeploy run the following command:

# zappa undeploy

And everything is off (the buckets are still there though).

Deploying a Django project in Lambda using Zappa is quite easy. 

I recommend you to try it out!

Read More

Simple SMS response for your Django App using Twilo

Twilio allows software developers to programmatically make and receive phone calls and send and receive text messages using its web service APIs.

On this guide we will set up our Django project to send text messages using Twilio.

We will assume that the Django project is already created.

Let's get to work!

1. Create a Twilio account at the Twilio Official page

2. Get a phone number to send messages. To do that we will login with our Twilio account and get one from the console section (you can get the first one for free).

3. Install the twilio-python library on our Django project by typing the following command on the terminal:

$ Pip install twilio

Almost there...

The last thing we have to do is include the code in the view that will be in charge of sending the message:

from twilio.rest import Client
account = "ACXXXXXXXXXXXXXXXXX"
token = "YYYYYYYYYYYYYYYYYY"
client = Client(account, token)

message = client.messages.create(to="+12316851234", from_="+15555555555",
                                 body="Hello there!")

Things to take into account:

• Your account SID and token are written on your account’s Console section. You should save those fields on your environment variables so that you won’t have that sensible information exposed in your code.

• You may have to give permissions to send SMS outside the US. Check if the country where you are trying to send a text message is enabled on the SMS Geographic permissions section.

• The from_ field must be filled up with the number you got for your twilio account.

And that’s it!!

Just call that view from a URL and there you go. Start sending messages!

References:

• https://www.twilio.com
• https://github.com/twilio/twilio-python

Read More

Dockerize your Django Web Application

Docker platform is becoming more and more popular thanks to it’s ability to create, deploy, and run applications easily by using containers.  

In this guide we will talk about how to dockerize our django web application using nginx, gunicorn and postgresql.
If you are not deeply familiarized with docker, check out this Docker Basic Guide.

Prerequisites

• Python 2.7 or 3.x

• Docker (Mac users: you can follow this Mac Installation guide)

• Install docker-compose (we can do that by running pip install docker-compose).

Create an empty directory

On this directory (we will name it "myproject") we will put all the required files to set up our docker configuration properly.

Create a subdirectory named web

This folder will contain our django project, let's name it mydjangoproject.

Place the requirements.txt file inside the Web folder

If you don't have it already, create it. It should be placed at the same level as mydjangoproject and contain all your project dependencies, including at least these three:

Django==1.11.2
gunicorn==19.7.1
psycopg2==2.6

Also inside "web", make a file named DockerFile and add the following lines:

FROM alpine

# Install required packages
RUN apk update
RUN apk upgrade
RUN apk add --update python python-dev py-pip postgresql-client postgresql-dev build-base gettext

# Initialize
RUN mkdir -p /data/web
COPY . /data/web/
WORKDIR /data/web/

#Setup
RUN pip install --upgrade pip
RUN pip install -r requirements.txt

#Prepare
RUN mkdir -p mydjangoproject/static/admin

By adding this lines we are setting up our container by installing the necessary packages such as pip and postgresql, adding our project to the container and installing the required dependencies.

Create a file called run_web.sh and place it inside the "web" folder

This file contains a script that will be executed when the container starts. We will add the following lines to it:

#!/bin/sh

python manage.py migrate                  # Apply database migrations
python manage.py collectstatic --noinput  # Collect static files

# Start Gunicorn
exec gunicorn myproject.wsgi:application \
  --bind 0.0.0.0:8008 \
  "$@"

Here we can see how we apply the migrations, collect the static files and start our gunicorn server on the localhost at the port 8008. Don’t forget to add the host '0.0.0.0' to your allowed hosts in your Django settings.

Go back to myproject directory and create a file named docker-compose.yml

This file is the one that will contain the configuration of all the services and how they interact with each other. Let’s add the following code:

version: '2'
services:
  # Postgres database
  postgres:
    restart: always
    image: postgres:latest
    volumes:
      - ./postgres/docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
    env_file: ./env
    expose:
      - "5432"
  # Django web service
  web:
    build: ./web
    ports:
      - "3000:8008"
    env_file: ./env
    links:
      - postgres
    depends_on:
      - postgres
    volumes:
      - ./web/mydjangoproject/static:/static
    working_dir: /data/web/mydjangoproject/
    command: /data/web/run_web.sh
  nginx:
    restart: always
    build: ./nginx/
    ports:
    - "8001:8001"
    volumes_from:
    - web
    links:
    - web

We can see here three services: “nginx”, “web” and “postgresql”. “nginx” is linked to “web” and “web” to “postgresql” in order to make them reachable between each other. Also it’s specified that “postgres” service will use the latest version of postgres image from dockerhub and both web and nginx are built using their dockerfiles. Finally, we can see that the web service will run the script written in run_web.sh when the container gets started.

All commands can be found on the docker official page.

Nginx configuration:

To configure nginx we are going to create a directory inside myproject called nginx and create a configuration file called default.conf inside it. Then we will write the following lines:

server {

    listen 8001;

    charset utf-8;

    location /static/ {

        root /;

    }

    location / {

        proxy_pass http://web:8008;

        proxy_set_header Host $host;

        proxy_set_header X-Real-IP $remote_addr;

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    }

}

On this configuration we specify the path of our static files (we have to write the same one that is specified on our settings.py file) and also we will set up the reverse proxy to redirect nginx to our gunicorn server. If you can’t see your static folder with your static files, run the command 'python manage.py collectstatic' and it will automatically collect the statics for you on the root of the project.

Also, as nginx is built and not taken from a dockerhub image, we will create a file named Dockerfile inside our nginx directory and write the following code.

      FROM nginx:alpine



RUN apk update

RUN apk upgrade

RUN apk add --update curl

ADD default.conf /etc/nginx/conf.d/default.conf

Finally, set up the database configuration

This configuration will be done on the settings file from our Django project. The database section should look like this:

 DATABASES = {  

      'default': {

        'ENGINE': 'django.db.backends.postgresql_psycopg2',

        'NAME': 'postgres',

        'USER': 'postgres',

        'HOST': 'postgres',

        'PORT': 5432,

      }

} 

We have linked the service "web" and "postgres" on the docker-compose.yml file already. That means that the "web" service will be able to establish connection with "postgres". To do that, take into account that database settings have to use the host name "postgres" to reach the container.

The last step is to create a file named 'env' inside 'myproject' folder with the database configuration. This file is referenced in our “env_file” variable on the docker-compose file.

We should write the following lines on it:

POSTGRES_DB=socialNetwork

POSTGRES_USER=root

DB_SERVICE=postgres

DB_PORT=5432

POSTGRES_PASSWORD=password
    

Environment variables can change depending on the database configuration. All the options can be found on the docker’s environment variables section for postgres.

It’s done!

Everyting should be set up now. We only need to go into 'myproject' folder (folder where docker-compose.yml is located) and run 'docker-compose build' to build the containers and 'docker-compose up' to start the containers.

Then if you go to localhost:8001 you should see the django project working.

Read More