============ Architecture ============ .. contents:: :depth: 2 High level overview =================== EJ adopts a monolithic architecture with the main server that controls business logic and some additional services that complement the application. The recommended way to deploy EJ is to run each service in a separate container in order to provide a good level of isolation. The aforementioned services are organized in a simple 3-layer architecture described bellow: .. image:: img/ej_arch.svg Nginx Web traffic should not be handled directly by the application service. Gunicorn is not efficient to serve static files and an additional reverse proxy can also be used to enforce more strict security policies and more efficient caching. EJ adopts Nginx. It serves static files and redirect dynamic routes to the application service described bellow. The Nginx container must share some volumes with the Django application in order to find static files. Django Application EJ is written mostly in Python and uses the Django web framework. The main application service is responsible for all dynamic routes, which are generated by Django using the templating language Jinja2. The Django application should be run with a WSGI compatible service such as Gunicorn. This is the default approach adopted in the application container. The default task uses a number of workers equal to the number of CPU cores, which is usually the recommended configuration. SQL Database EJ does not use any database-specific functionality or raw SQL commands. This means that it can run in any database supported by Django such as Postgres SQL, MariaDB, Sqlite3, etc. We recommend Postgres (v10.0), which is used in the default installation. The database connection is controlled by the DJANGO_DB_URL environment variable in the main application container. Rocket.Chat integration ----------------------- Instances that deploy the optional Rocket.Chat integration must consider the additional services. Rocket.Chat Rocket.Chat is a Node.js application based on the Meteor framework that serves static files and the dynamic routes. EJ is compatible with the official Rocket.Chat container in `Docker Hub`_, which is the recommended deployment method. Mongo DB Rocket.Chat persistence layer uses Mongo. The Rocket.Chat container is compatible with the official Docker Hub image and requires minimal configuration. .. _Docker Hub: https://hub.docker.com/_/rocketchat Frontend ........ EJ Frontend is implemented using the Jinja2 templating language and uses progressive enhancement to include styles via CSS and custom behaviors with JavaScript. The following presents a brief overview of the technologies used in each of those layers: CSS CSS is implemented with Sass using a ITCSS-inspired architecture (Inverse Triangle CSS). The CSS module is implemented using the `Mendeleev.css`_ framework for atomic CSS and can be easily customized using themes. The CSS assets are statically compiled and served by Nginx. Sass compilation requires libsass, which is bundled into the Python dependencies of the application. JavaScript/TypeScript EJ does not adopt any traditional JavaScript framework, but instead relies on progressive enhancement to add optional functionalities. EJ uses Unpoly_ in conjunction to jQuery_ to provide the core functionality. EJ-specific components are created using TypeScript and enhance tags annotated with the "is-component" attribute with extra behaviors and functionalities. TypeScript compilation requires the Node Package Manager (NPM) and Parcel_. .. _Mendeleev.css: https://www.npmjs.com/package/mendeleev.css .. _Unpoly: https://unpoly.com .. _jQuery: https://jquery.com .. _Parcel: https://parceljs.org Django application ================== Django splits a web system into modules called "apps" which implement reusable database models, routes and functionalities. This section describes all "apps" implemented in EJ. EJ project ---------- The `ej` module is not properly an app, but a regular Python package used to coordinate apps by defining settings, common functionality and loading static assets like Javascript, CSS, images, themes etc. The following is an overview of the main sub-packages and modules: ``ej.all`` Useful namespace to be used in a interactive section as ``from ej.all import *``. It imports models and managers of all ej apps and examples in the global namespace. ``ej.components`` Similarly to ``ej.roles``, this module defines renderers for reusable UI elements. The difference between the two modules is that components can have a more complicated structure and may not be directly associated with some known Python data type. ``ej.contrib`` Location to include ad-hoc migrations for specific deployments. Most users and developers should never touch this. ``ej.fixes`` Monkey patch third part modules that have known issues with EJ or any of its dependencies. ``ej.forms`` Base form classes that are used in other EJ apps. Forms are derived from django.forms. ``ej.jinja2`` EJ uses Jinja2 as the default templating language. This module configures the Jinja2 environment and set global functions and filters. ``ej.roles`` Functions that define Hyperpython roles. Roles are mappings of ``(type, name) -> HTML`` that define how a certain object should be rendered in a given context or role. This module defines many reusable UI elements as Python functions. ``ej.routes`` Define some global view functions such as the home page that do not have functionality tied to any app. ``ej.services`` Helper functions to initialize connections to external services such as Postgres SQL database and redis (if enabled). ``ej.settings`` Django settings module. Defines configuration using Django Boogie's configuration framework in which configuration is defined in reusable classes instead of a flat Python module. ``ej/templates/jinja2`` Contains templates available globally. The global ``base.jinja2`` template defines the base page HTML structure (navigation bars, meta information, etc) that is shared among most pages in the site. ``ej.testing`` Helper tools used in testing across apps. ``ej.tests`` Global tests. Most tests are implemented in app-specific test folders. ``ej.urls`` URL mapping for the project. Most URLs are included from an app's own ``routes.py``. ``ej.utils`` Utility functions module. ``ej.wsgi`` Django wrapper for the WSGI interface. Applications ------------ The listing bellow describes all apps implemented inside EJ source tree. ``ej_conversations`` .. image:: orm/ej_conversations.svg :target: ../_images/ej_conversations.svg This is the main application and defines models for conversations, comments, and votes. The ej_applications app implements the UI for creating, configuring and interacting with conversations. ``ej_users`` .. image:: orm/ej_users.svg :target: ../_images/ej_users.svg This app defines the main User model for EJ and all routes related to authentication and account management (e.g., reset passwords, cancel account, etc). EJ can be used with Django's regular users, although this is not encouraged. ``ej_profiles`` .. image:: orm/ej_profiles.svg :target: ../_images/ej_profiles.svg Implements profile management UI and defines a model that store profile information. This app can be easily modified to include extra profile fields or to remove unwanted fields for some particular installation. ``ej_clusters`` .. image:: orm/ej_clusters.svg :target: ../_images/ej_clusters.svg Implements the mathematical routines to classify users into opinion groups. The ej_clusters.math module implements our modified K-means algorithm that takes into account "opinion stereotypes" and also provides interfaces to manage those stereotypes and the resulting clusters. ``ej_dataviz`` .. image:: orm/ej_dataviz.svg :target: ../_images/ej_dataviz.svg Implements routines to visualize data about conversations. It generates structured reports and export data to spreadsheet-compatible formats. This module also implements visualization techniques such as Word Cloud and Scatter Maps of user opinions. ``ej_gamification`` .. image:: orm/ej_gamification.svg :target: ../_images/ej_gamification.svg The gamification app implements the points and badges system in EJ. Most interactions in the platform are rewarded with points. Users that achieve pre-defined levels of participation receive badges that recognize different types of interactions such as voting on comments, creating popular conversations, etc. ``ej_boards`` .. image:: orm/ej_boards.svg :target: ../_images/ej_boards.svg The boards app allow regular users to have their own "board" or "timeline" of conversations. The default conversation feed in "/conversations/" can only be managed by users with special permissions. ``ej_experiments`` This optional app is responsible for creating and saving testing data in the database. It is useful for development, but it is not enabled in deployment installations. Third party Apps ---------------- ``boogie.apps.fragments`` The Boogie fragments app implements configurable text or HTML fragments. This allows a greater level of configurability by allowing administrative users to customize parts of the platform without using any code. ``rules`` Django-rules_ implements a mechanism to define business logic rules by registering simple predicate functions. This package nicely integrates with Django's own permission mechanisms. The business rules relevant to each EJ application are implemented into the respective "rules.py module of each Django app and can be overridden by third party apps or modules. ``Django taggit`` Django-taggit_ is a Django application that implements tags to arbitrary models. It is used to support tagging of EJ conversations. ``rest_framework`` The Django-Rest-Framework_ (DRF) is a powerful toolkit to develop REST Web APIs. EJ uses DRF through the rest_api module of Django-Boogie. ``allauth, allauth.account, allauth.socialaccount`` The `allauth`_ project implement authentication and authorization workflows and integration with third party OAuth providers such as Google, Twitter and Facebook. .. _Django-rules: https://github.com/dfunckt/django-rules .. _allauth: https://www.intenct.nl/projects/django-allauth/ .. _Django-taggit: https://github.com/jazzband/django-taggit .. _Django-Rest-Framework: https://www.django-rest-framework.org App structure ============= EJ uses Django Boogie adopts an architecture that may be slightly different from a typical Django app. One important goal of the architecture is to make lightweight view functions and models. This is accomplished by either moving functionality to the Boogie framework itself or to splitting functionality into different modules. A typical EJ App has the following structure: ``.admin`` Django admin classes and functions. ``.api`` Defines fields and API routes for the models defined by the app. Normally, functionalities implemented in this module simply supplement the main API declarations that are created using the ``@rest_api`` decorator directly on models. ``.apps`` Django AppConfig mechanism. EJ apps usually should override the ``ready()`` method of the app config and import the api, roles and rules modules. ``.enums`` This modules defines any enum type that is eventually used by models. Enums are usually imported into the model base namespace, so they should have no dependency on models. ``.forms`` Django forms defined for the app. Usually forms should inherit from ej.forms instead of using django.forms directly. ``.math`` All math functions should be defined in this module. Complicated mathematical transformations should be implemented as ScikitLearn transformations or pipelines. ``.managers`` In Django, model classes defines row logic and managers and querysets implements table logic. All methods that query or create models or filter querysets should be implemented in the "managers" module ``.models`` Just like in regular Django apps, this module defines the models for the app. Models should avoid implementing business logic inside them and ideally should restrict to database actions such as querying, validation, etc. ``.mommy_recipes`` EJ uses Model Mommy to create random fixtures for tests. This module should define a class that derives from ``ej.testing.EjRecipes`` and implements fixtures for each model defined in the app. This is only used in tests. ``.routes`` Regular Django apps have a views.py and a urls.py files. Django Boogies encourages to join both files into a single routes.py that defines view functions and maps them to routes using decorators. ``.rules`` Business rules are implemented as regular functions inside this module. This helps avoiding the "fat-models" anti-pattern that is common in Django projects. The rules module can define both permissions (which are user-centric predicate functions) and regular "values", which can return non-boolean values (e.g., number of comments user still has in conversation). ``.roles`` Hyperpython roles are simple functions that render objects in given contexts. For instance, we can register a "card" role to the Conversation class that renders the input conversation as a card in a listing view. Role functions must be associated with a type and a name describing is role and must return a Hyperpython html structure. In cases that Jinja2 is more convenient than Hyperpython, the ``ej.roles.with_template`` decorator can be used to associate the role with a Jinja template. ``.tests`` App's unit tests. ``.validators`` Implement the validation functions used in model fields or form fields inside the app. Templates --------- Templates reside inside the ``/jinja2/`` folder. We use Django best practices and save app-specific templates inside ``jinja2//.jinja2``. Templates names usually mirror the names of view functions in the ``routes.py`` file. For instance, a edit view for some conversation would be declared as:: urlpatterns = Router(template='ej_conversations/{name}.jinja2') @urlpatterns.route("//edit/") def edit(request, conversation): ... # Implementation This view function is automatically associated with the ``ej_conversations/edit.jinja2`` template, unless specified otherwise. Most templates inherit from a base template at ``src/ej/templates/jinja2/base.jinja2``. This template imports navigation elements such as menus and toolbars.