Developer notes

These are notes for those developing on baselayer, and also for developing apps running on baselayer.

Testing

Each project using baselayer will choose its own testing infrastructure. At the moment, we recommend [pytest-playwright])(https://playwright.dev/python/docs/test-runners).

Baselayer is tested against the baselayer template app.

To execute the test suite, clonet that repo with submodules and do:

  1. pip install -r baselayer/requirements.txt -r baselayer/requirements.test.txt

  2. playwright install firefox

  3. Run the web server: make run

    • Run the test suite: pytest

    • Run a single test: pytest template_app/tests/frontend/<test_file>.py::test_<specific_test>

To make the above steps a bit easier in CI, we have a tool ./tools/test_frontend.py <test>. It runs the server, waits for it to become available, and then runs the specified test.

Tests are run in headless mode by default. To run visibly, use pytest --headed.

We run tests in Firefox by default. Edit pytest.ini if you want to try another.

Debugging

  • Run make log to watch log output

  • Run make stop to stop any running web services.

  • Run make attach to attach to output of webserver, e.g. for use with pdb.set_trace()

  • Run make check-js-updates to see which Javascript packages are eligible for an upgrade.

Database

All interactions with the database are performed by way of SQLAlchemy using the Pyscopg2 backend. Some standard—but not necessarily obvious—usage patterns we have include:

  • Logic for connecting to the DB, refreshing tables, etc. is found in baselayer/model_utils.py:

from baselayer.app.env import load_env
from baselayer.app.models import DBSession, init_db
env, cfg = load_env()
init_db(**cfg['database'])

  • The session object controls various DB state operations:

DBSession().add(obj)  # add a new object into the DB
DBSession().commit()  # commit modifications to objects
DBSession().rollback()  # recover after a DB error

  • Generic logic applicable to any model is included in the base model class baselayer.app.models.Base (to_dict, __str__, etc.), but can be overridden within a specific model

  • Models can be selected directly (User.query.all()), or more specific queries can be constructed via the session object (DBSession().query(User.id).all())

  • Convenience functionality:

    • Join relationships: some multi-step relationships are defined through joins using the secondary parameter to eliminate queries from the intermediate table; e.g., User.acls instad of [r.acls for r in User.roles]

    • Association proxies: shortcut to some attribute of a related object; e.g., User.permissions instead of [a.id for a in User.acls]

    • Joined loads: this allows for a single query to also include child/related objects; often used in handlers when we know that information about related objects will also be needed.

    • to_json(): often from a handler we return an ORM object, which gets converted to JSON via json_util.to_json(obj.to_dict()). This also includes the attributes of any children that were loaded via joinedload or by accessing them directly. For example:

      • User.query.first().to_dict() will not contain information about the user’s permissions

      • u = User.query.first(); u.acls; u.to_dict() does include a list of the user’s ACLs

New SQL Alchemy 2.0 style select statements

To start a database session without write-permission verification:

with DBSession() as session:
  ...

The context manager will make sure the connection is closed when exiting context.

To use a verified session, one that verifies write permissions by an authenticated user, use:

with VerifiedSession(user_or_token) as session:
  ...
  session.commit()

This does the same checks that are performed when calling self.verify_and_commit() inside of any handler.

TODO: Update write access enforcement documentation.

We have fairly sophisticated access control, but this requires further documentation. For now, refer to the SkyPortal Access Controldocumentation. Look at CustomUserAccessControl and all derivative classes of UserAccessControl. Also see examples of how it is used in SkyPortal models.

Each handler class can also call self.Session() as a stand-in for VerifiedSession(self.current_user):

with self.Session() as session:
  ...
  session.commit()

To quickly get rows from a table using the new “select” methods, use one of these (replace User with any class):

user = User.get(id_or_list, user_or_token, mode='read', raise_if_none=False, options=[])
all_users = User.get_all(user_or_token, mode='read', raise_if_none=False, options=[], columns=None)
stmt = User.select(user_or_token, mode='read', options=[], columnns=None)

The get and get_all functions open a session internally and retrieve the objects specified, if they are accessible to the user. In the case of get, if any of the IDs given (as a scalar or list) are not accessible to do not exist in the DB, the function returns None, or raises an AccessError (if raise_if_none=True is specified). The get_all just retrieves all rows that are accessible from that table. Note that these two methods will produce an object not associated with the external session, if any. Thus, if the call is made while an external context is used, the object has to be added to that session before it can, e.g., load additional relationships, or be saved, or do any other operation that involves the database. As an example:

with self.Session() as session:
  user = User.get(user_id, self.current_user, mode='read')
  session.add(user)  # must have this to load additional relationships
  tokens = user.tokens  # will fail if user is not in session

On the other hand, the select function will return a select statement object that only selects rows that are accessible. This statement can be further filtered with where() and executed using the session:

with VerifiedSession(user_or_token) as session:
  stmt = User.select(user_or_token).where(User.id == user_id)
  user = session.execute(stmt).scalars().first()  # returns a tuple with one object
  # can also call session.scalars(stmt).first() to get the object directly
  user.name = new_name
  session.commit()

If not using commit(), the call to VerifiedSession(user_or_token) can be replaced with DBSession() with no arguments.

Standards

We use ESLint to ensure that our JavaScript & JSX code is consistent and conforms with recommended standards.

  • Install ESLint using make lint-install. This will also install a git pre-commit hook so that any commit is linted before it is checked in.

  • Run make lint to perform a style check

Upgrading Javascript dependencies

The ./tools/check_js_updates.sh script uses npm-check to search updates for packages defined in package.json. It then provides an interactive interface for selecting new versions and performing the upgrade.