Structuring a Python application

Mathieu Dugue

2020-09-26

In this short article I will explain all the different ways of structuring a Python application, from a quick script to a more complex web application.

Single python file containing all the code

That’s okay for code without dependencies or using a virtual environment. This is generally used for a quick script that doesn’t need complexity in its file structure.

my-single-file-project/
├── .gitignore
├── helloworld.py
├── test.py
├── pyproject.toml
├── LICENSE
└── README.md

Example of the helloworld.py :

#!/usr/bin/env python
# helloworld.py

from datetime import datetime

def do_hello():
    print(f"Hello, today is {datetime.now().date()}")

if __name__ == '__main__':
    do_hello()

Installable Single Package

When your app get more complex you generally wants to group your different files and helpers into a module.

You also need to improve your tests so they cover all the files → test coverage. You can also add the use of a linter.

my-single-package-app/
├── helloworld/
│   ├── __init__.py
│   ├── helloworld.py
│   └── utils.py
├── tests/
│   ├── __init__.py
│   ├── test_helloworld.py
│   └── test_utils.py
├── runtest.sh
├── .gitignore
├── pyproject.toml
├── LICENSE
└── README.md

Now we grouped the files from our app in two folders, the helloworld app folder and the tests folder.

We match the test_****.py files name with the files in the app module folder.

The __init__.py files are mandatory for python to know it is package and can be empty.

#!/usr/bin/env python
# helloworld.py

from datetime import datetime

from helloworld.utils import show_message

def do_hello():
    show_message(f"Hello, today is {datetime.now().date()}")

if __name__ == '__main__':
    do_hello() # pragma: no cover

runtests.sh file is a script to run all the test with the coverage dependency that you can install with poetry add coverage -D :

#!/bin/bash

find . -name "*.pyc" -exec rm {} \;
coverage run -p --source=tests,helloworld -m unittest # this is where you add the
                                                      # folders you want to test
if [ "$?" = "0" ]; then
    coverage combine
    echo -e "\n\n================================================"
    echo "Test Coverage"
    coverage report
    echo -e "\nrun \"coverage html\" for full report"
    echo -e "\n"

    # pyflakes or its like should go here
fi

Application with internal packages

In bigger apps you will probably have multiple modules, particularly if you do a GUI.

Possible additions to your application are :

bin directory, wrapper for your entry point.
data directory, always separate your data from your code

app/
├── helloworld/
│   ├── __init__.py
│   ├── hello/
│   │   ├── __init__.py
│   │   ├── hello.py
│   │   └── utils.py
│   ├── world/
│   │   ├── __init__.py
│   │   └── world.py
├── tests/
│   ├── __init__.py
│   ├── hello/
│   │   ├── __init__.py
│   │   ├── test_hello.py
│   │   └── test_utils.py
│   ├── world/
│   │   ├── __init__.py
│   │   └── test_world.py
├── bin/
│   └── helloworld*
├── data/
│   └── translate.csv
├── docs/
│   ├── conf.py
│   ├── index.rst
│   ├── hello.rst
│   ├── world.rst
│   └── Makefile
├── runtest.sh
├── .gitignore
├── pyproject.toml
├── LICENSE
└── README.md

bin/

This new folder holds any executable files the users will use. The script drop the .py ending. It should have very little code logic, it’s just a wrapper for the main module’s entry point.

You can configure setup.py to package this if you build a wheel, the script will be put on the path.

data/

If there are files that are not code that is usually where they should end up. It is also useful for test data.

docs/

The documentation is very important if you want to share your app with users.

A very useful library for that is sphinx, it uses pydoc comments to build the doc automatically.

poetry add sphinx -D
poetry add sphinx-rtd-theme -D
cd docs
sphinx-quickstarts
make html

The .rst files are generated by sphinx-quickstart.

Django web application

After you’ve installed Django you can use this included command to generate the architecture of the project as Django expects it :

django-admin startproject django_world
django-admin startapp hello

You will end up with this basic folder tree where just a few things has been manually added :

django_world/
├── docs/
├── static/
│   └── style.css
├── templates/
│   └── base.html
├── resetdb.sh
├── runserver.sh
├── hello/
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── migrations/
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── django_world/
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── manage.py

The hello/ folder is where all the logic, model, views and so on live.

The static folder contains all the static contents (css, js,…).

The templates folder has all the html templates.

runserver.sh is a script that has only one line :

python manage.py runserver

resetdb.sh is a script for wiping all the database and start from scratch, it is just a helper tool :

#!/bin/bash

find . -name "*.pyc" -exec rm {} \;
rm db.sqlite3

python manage.py migrate

# python manage.py create_test_data
# comment that last line to use premade test data

Flask

Flask is proud to show a working web app with 10 lines of code, but in practice you’ll never do that. Here is a complete folder organization :

flaskr/
│
├── flaskr/
│   ├── ___init__.py
│   ├── db.py
│   ├── schema.sql
│   ├── auth.py
│   ├── blog.py
│   ├── templates/
│   │   ├── base.html
│   │   ├── auth/
│   │   │   ├── login.html
│   │   │   └── register.html
│   │   │
│   │   └── blog/
│   │       ├── create.html
│   │       ├── index.html
│   │       └── update.html
│   │
│   └── static/
│       └── style.css
│
├── tests/
│   ├── conftest.py
│   ├── data.sql
│   ├── test_factory.py
│   ├── test_db.py
│   ├── test_auth.py
│   └── test_blog.py
│
├── .gitignore
└── MANIFEST.in

We can see that flask really uses python package way of working.

Source : https://realpython.com/python-application-layouts/