Overview¶
Streamwebs is an science education platform for educators and students in Oregon’s public schools. The application allows students to enter data about streams and wetlands gathered on field trips, then graph or export the data for further analysis.
This application is a re-write of the original Drupal-based Streamwebs application. Until the new application goes live, the original can be found at Streamwebs.org.
Core Components¶
Streamwebs is written in Python usign the Django web framework. It follows Python’s PEP8 coding style and standard Django MVC structure. Streamwebs also makes use of geographic information and the Google Maps API. The underlying data store is Postgres with geographic data extensions (PostGIS).
Streamwebs also uses the D3 javascript library for generating graphs, and the Materialize CSS framework for the front-end UI elements.
Django Structure¶
Django’s MVC model is slightly odd, what would normally be called a Controller is a View, and what would be called a View is a Template. Models are what you would expect, a description of the data model, for use with Django’s ORM. URLs are defined in a separate file.
Models¶
All models are defined in streamwebs_frontend/streamwebs/models.py
Views¶
Views are typically divided logically into several files according to the model
they refer to. Streamwebs doesn’t do this, all general views are defined in
streamwebs_frontend/streamwebs/views/general.py
, and a separate set of
views specifically dealing with CSV export are in
streamwebs_frontend/streamwebs/views/export.py
Views recieve the current http request object object and returns an HTTP response object. See Django Request and response objects for their structure.
Views may also be ‘decorated’ to restrict access to logged in users. See Django View Decorators for information on what can be done with decorators.
For example, you will see this construction frequently in the views file:
@login_required
def create_site(request):
created = False
...
The @login_required is a decorator provided by the Django Authentication Framework and prevents non-authenticated users from accessing this view.
Templates and TemplateTags¶
Django by default uses a template language similar to Jinja2, other backends
can be defined, but Streamwebs sticks to the default. Templates have the
.html extension and are found in streamwebs_frontend/streamwebs/templates/
Templates have an inheritance structure, our base template sets up the HTML page, and other templates inherit this structure. See Django Templates for more information.
Streamwebs also makes use of TemplateTags, which can be used to transform text
in templates. These are defined in
streamwebs_frontend/streamwebs/templatetags/filters.py
.
See Custom Django Template Tags for more information.
In templates, filters are included using {% load filters %}
at the topof
the file. A filter is applied with the |
character appended to a rendered
template variable:
{{ 'zone'|get_zone_labels }}
URLs¶
Streamwebs defines its URLs in streamwebs_frontend/streamwebs/urls.py
.
This file defines all URLs and connects them to their view method, which should then render the appropriate template.
URLs are namespaced, the app_name = 'streamwebs'
line ensures that URLs are
relative to the streamwebs
app. In practicat terms, the important thing to
know is that links in templates are specified with the streamwebs namespace:
{% url 'streamwebs:register' %}
This contruction will render the URL named register
in the streamwebs
application.
Sites¶
The Primary object in Streamwebs is the Site, which consists of a name, description, location (coordinates), and optionally an image. Site locations are point objects and are displayed in a map view as markers.
All data is associated with a site, and the site view is the starting point for entering, viewing, graphing or downloading collected data.
See Sites for more details.
Datasheets¶
Datasheets are associated with a site, and also with a school. They store collected data about a stream - air temperature, water temperature, species detected, etc.
There are six datasheets currently defined:
- Canopy Cover: records percentage of the sky obscured by forest canopy
- Macroinvertibrates: records the numbers of specific species of invertebrates
- Photo Point: records images taken at a specific location
- Riparian Transect: records plant types in a cross-section of a streambed
- Soil Survey: records the types of soil found
- Water Quality: records various water measurments such as temperature and oxygen level
Each datasheet has its own model, form, view and database table, and may have associated models, for example mulitple samples for a specific measurement.
Datasheet views are designed to resemble as closely as possible the paper versions that are taken into the field to record data. Calculated values are automatically calculated on data entry. PDF files for each sheet are provided for printing.
See Datasheets for more details.
Users¶
Students and teachers share a general ‘user’ role. Users are authorized to enter data and create sites, the only reserved permissions are resource file uploads and viewing site statistics.
Users are associated with a School. Birth date is a required field for account creation, and students must be 13 years of age or older to sign up.
Users can self-register, by default they will be placed in the general user role, a admin can promote any user to an admin role. The initial admin account is the Django ‘superadmin’, created on deployment, and this superadmin should designate one or more user accounts as admins.
See Users for more details.
Resources¶
Several types of files are available for download from the site. A generic ‘resource’ model is used to store datasheets, educator kits, publications and tutorial videos, and these resources are displayed on type-specific pages. An index of all resources is located at /resources.
See Resources for more details.
Statistics¶
Basic site statistics are available to admin users. Number of users, site and datasheets are the primary statistics.
See Statistics for more details.
Schools¶
A model containing a list of known public schools in Oregon, used for tracking which school contributed data to a site. Also associated with users.
Note
Data is associeted explicitly with a school, we do not rely on the account of the user who entered data for determining which school created the data.
See Schools for more details.
Test Suite¶
All views, forms, permissions and models are tested with unit tests. These tests use the Django test framework.
In addition to tests, files should be analyzed by the Flake8 python linter, which enforces Python standard PEP8.
Development of new features should begin with writing a test for that feature.
See Tests for more details.
Internationalization¶
Streamwebs uses the Django translation framework to translate strings into supported languages. Supported languages can be selected using a pull-down selector in the application.
Supported languages are set using the LANGUAGES
setting. The default
application application language is set in LANGUAGE_CODE
and defaults to
en-us
.
Translation in the templates is done using the trans
template tag, and in
python code by the _() method. Translations of these strings are stored in
message files, which contain string identifiers (typically just the original
string in the default application language) and that strings translation into
the target language.
To create a new messages file for language <lang>
use the command
django-admin makemessages -l <lang>
This will extract translatable strings from the code (strings in a trans
tag or _()
method) and write them to a message file:
locale/<lang>/LC_MESSAGES/django.po
See the Django translation framework documentation for much more information about the translation framework.
See Translation for more details about translation implementation in Streamwebs.
Data Import¶
On initial deployment, the application will be seeded with data exported from
the old Drupal application. A number of scripts in the data_scripts
directory are responsible for importing data. These scripts will be run by the
deployment script, and should only be run once per application instance. Due to
the complexity of the Drupal exported data, modifying this code is not
recommended.
User accounts will be imported from the previous application, and when the production instance is ready, every active member will be sent an email explaining how to reset their password for the new system.
See Data Import for more details.
Dev Environment¶
The Streamwebs project uses Docker and docker-compose for running test and a local instances of the application for development purposes.
The configuration in docker-compose.yml
will build a postgres database
container with the necessary PostGIS extensions, and a ‘web’ container running
the application.
Standard docker commands can be used to run the test suite or other management commands in the web container.
See the contents of dockerfiles/
for the Docker container definition and
startup/cleanup scripts.
Setup and configuration¶
First, make sure you have a working Docker install, the Docker daemon is running, and your user has permission to run Docker.
Second, make sure you have the python package docker-compose installed. The easiest way to do this is to create a local python virtualenv and install docker-compose into that.
virtualenv venv
source venv/bin/activate
pip install docker-compose
Finally, configure the application. Streamwebs ships with some default settings in several ‘dist’ files. These files need to be copied to their proper names before the application will run.
These files may be edited, but should be adequate as-is for the Docker dev environment.
The main application settings:
cp streamwebs_frontend/streamwebs_frontend/settings.py.dist \
/streamwebs_frontend/streamwebs_frontend/settings.py
The docker environment:
cp dockerfiles/Dockerfile.env.dist dockerfiles/Dockerfile.env
Running the Dev Docker Environment¶
When docker-compose
is installed and setttings are in place, you can launch
the dev environment with:
docker-compose up web
This will build the PostGis container and a new CentOS container with the
streamwebs application running on port 8000. You should see the application at
http://localhost:8000
.
Running management commands¶
To run management command in the docker environment, or to import data, you can run a bash shell in the container:
docker-compose run web bash
Migrating the database
The database schema is managed by migration files in
streamwebs_frontend/streamwebs/migrations
.
If this is the first time you’ve built the docker PostGis container, you will need to migrate the fresh database with the initial migration:
docker-compose run web bash
cd streamwebs_frontend/streamwebs_frontend
./manage.py migrate
Any changes to the schema should be expressed in a new migration. These can be written manually, but it is much easier and safer to let Django generate them based on changes to the models code.
After making changes to models that effect the schema, make new migrations:
docker-compose run web bash
cd streamwebs_frontend
./manage.py makemigrations
And migrate to update the schema:
docker-compose run web bash
cd streamwebs_frontend
./manage.py migrate
Importing Data
Data import scripts are idempotent, they load their data from CSV files in the
csvs
directory. The get_all.sh
shell script will run all
of the python data import scripts in the correct order. The database should be
migrated prior to importing data.
docker-compose run web bash
cd data_scripts
./get_all
You can also import individual data types using the python scripts in this directory.
Creating a superuser
To create a superuser (site admin account with all permissions):
docker-compose run web bash
cd streamwebs_frontend
./manage.py createsuperuser
The script will ask for a name, password and email address.
Run a database shell
Django provides a convenient console to the database, saving you the time of manually connecting:
docker-compose run web bash
cd streamwebs_frontend
./manage.py dbshell
This will place you at a the psql commandline for the streamwebs Postgres
database running in the PostGis container. Type help
for a listing of psql
commands, or \d
to see Streamwebs’ tables.