Developer notes

These are notes for those developing on baselayer, and also for developing apps running on baselayer.

Testing

Each project using baselayer will choose its own testing infrastructure. At the moment, we recommend [pytest-playwright])(https://playwright.dev/python/docs/test-runners).

Baselayer is tested against the baselayer template app.

To execute the test suite, clonet that repo with submodules and do:

  1. pip install -r baselayer/requirements.txt -r baselayer/requirements.test.txt

  2. playwright install firefox

  3. Run the web server: make run

    • Run the test suite: pytest

    • Run a single test: pytest template_app/tests/frontend/<test_file>.py::test_<specific_test>

To make the above steps a bit easier in CI, we have a tool ./tools/test_frontend.py <test>. It runs the server, waits for it to become available, and then runs the specified test.

Tests are run in headless mode by default. To run visibly, use pytest --headed.

We run tests in Firefox by default. Edit pytest.ini if you want to try another.

Debugging

  • Run make log to watch log output

  • Run make stop to stop any running web services.

  • Run make attach to attach to output of webserver, e.g. for use with pdb.set_trace()

  • Run make check-js-updates to see which Javascript packages are eligible for an upgrade.

Database

All interactions with the database are performed by way of SQLAlchemy using the Pyscopg2 backend. Some standard—but not necessarily obvious—usage patterns we have include:

  • Logic for connecting to the DB, refreshing tables, etc. is found in baselayer/model_utils.py:

from baselayer.app.env import load_env
from baselayer.app.models import DBSession, init_db
env, cfg = load_env()
init_db(**cfg['database'])

  • The session object controls various DB state operations:

DBSession().add(obj)  # add a new object into the DB
DBSession().commit()  # commit modifications to objects
DBSession().rollback()  # recover after a DB error

  • Generic logic applicable to any model is included in the base model class baselayer.app.models.Base (to_dict, __str__, etc.), but can be overridden within a specific model

  • Models can be selected directly (User.query.all()), or more specific queries can be constructed via the session object (DBSession().query(User.id).all())

  • Convenience functionality:

    • Join relationships: some multi-step relationships are defined through joins using the secondary parameter to eliminate queries from the intermediate table; e.g., User.acls instad of [r.acls for r in User.roles]

    • Association proxies: shortcut to some attribute of a related object; e.g., User.permissions instead of [a.id for a in User.acls]

    • Joined loads: this allows for a single query to also include child/related objects; often used in handlers when we know that information about related objects will also be needed.

    • to_json(): often from a handler we return an ORM object, which gets converted to JSON via json_util.to_json(obj.to_dict()). This also includes the attributes of any children that were loaded via joinedload or by accessing them directly. For example:

      • User.query.first().to_dict() will not contain information about the user’s permissions

      • u = User.query.first(); u.acls; u.to_dict() does include a list of the user’s ACLs

New SQL Alchemy 2.0 style select statements

To start a database session without write-permission verification:

with DBSession() as session:
  ...

The context manager will make sure the connection is closed when exiting context.

To use a verified session, one that verifies write permissions by an authenticated user, use:

with VerifiedSession(user_or_token) as session:
  ...
  session.commit()

This does the same checks that are performed when calling self.verify_and_commit() inside of any handler.

TODO: Update write access enforcement documentation.

We have fairly sophisticated access control, but this requires further documentation. For now, refer to the SkyPortal Access Controldocumentation. Look at CustomUserAccessControl and all derivative classes of UserAccessControl. Also see examples of how it is used in SkyPortal models.

Each handler class can also call self.Session() as a stand-in for VerifiedSession(self.current_user):

with self.Session() as session:
  ...
  session.commit()

To quickly get rows from a table using the new “select” methods, use one of these (replace User with any class):

user = User.get(id_or_list, user_or_token, mode='read', raise_if_none=False, options=[])
all_users = User.get_all(user_or_token, mode='read', raise_if_none=False, options=[], columns=None)
stmt = User.select(user_or_token, mode='read', options=[], columnns=None)

The get and get_all functions open a session internally and retrieve the objects specified, if they are accessible to the user. In the case of get, if any of the IDs given (as a scalar or list) are not accessible to do not exist in the DB, the function returns None, or raises an AccessError (if raise_if_none=True is specified). The get_all just retrieves all rows that are accessible from that table. Note that these two methods will produce an object not associated with the external session, if any. Thus, if the call is made while an external context is used, the object has to be added to that session before it can, e.g., load additional relationships, or be saved, or do any other operation that involves the database. As an example:

with self.Session() as session:
  user = User.get(user_id, self.current_user, mode='read')
  session.add(user)  # must have this to load additional relationships
  tokens = user.tokens  # will fail if user is not in session

On the other hand, the select function will return a select statement object that only selects rows that are accessible. This statement can be further filtered with where() and executed using the session:

with VerifiedSession(user_or_token) as session:
  stmt = User.select(user_or_token).where(User.id == user_id)
  user = session.execute(stmt).scalars().first()  # returns a tuple with one object
  # can also call session.scalars(stmt).first() to get the object directly
  user.name = new_name
  session.commit()

If not using commit(), the call to VerifiedSession(user_or_token) can be replaced with DBSession() with no arguments.

Choosing a session

Which session to use depends on two things: whether the code runs inside a web request, and whether its writes have to be checked against a user’s permissions.

Inside a handler, use self.Session(), which stands in for VerifiedSession(self.current_user) and checks write permissions on commit, or DBSession() where that check is unwanted. Both are scoped on session_context_id, a context variable that BaseHandler.prepare() sets to a fresh value on every request, so each request works against its own session.

Outside a request that context variable keeps its default of None, so every DBSession() call in the process resolves to one shared session; in a microservice, a script, or work handed off to a thread, that sharing means one caller’s rollback() discards another caller’s pending work. Use new_session() there, which builds an independent session on each call:

from baselayer.app.models import new_session

with new_session() as session:
    session.add(record)
    session.commit()

new_session() applies no access-control check, so it suits code acting on its own behalf; where a write has to be checked against a user’s permissions, use VerifiedSession(user_or_token).

The async factories are the counterparts of DBSession and VerifiedSession, opt-in per handler, with the sync engine and session remaining authoritative for the rest of the codebase: async_session_factory() verifies on commit, and async_plain_session_factory() skips that check. app/access.py uses the plain one:

from baselayer.app import models

async with models.async_plain_session_factory() as session:
    ...

Reach the async factories through the module rather than importing their names directly: init_db() rebinds those globals, so from baselayer.app.models import async_plain_session_factory captures the None they hold before that call.

Standards

We use ESLint to ensure that our JavaScript & JSX code is consistent and conforms with recommended standards.

  • Install ESLint using make lint-install. This will also install a git pre-commit hook so that any commit is linted before it is checked in.

  • Run make lint to perform a style check

Upgrading Javascript dependencies

The ./tools/check_js_updates.sh script uses npm-check to search updates for packages defined in package.json. It then provides an interactive interface for selecting new versions and performing the upgrade.