FMN Replacement (April 2022)

FMN is a web application allowing users to create filters on messages sent to fedmsg and forward these as notifications on to email or IRC

Issues to solve

Message Delivery Lag

In times of peak congestion on the message queue (e.g. during a mass rebuild) it can take days for messages to be delivered to users.

fedmsg

The current version of FMN was implemented uses the deprecated fedmsg bus, not the currently supported Fedora Messaging (rabbitmq) message queue.

Additionally, the FMN code is deeply integrated with the fedmsg ecosystem (e.g. the configuration setup for FMN is kept in the fedmsg.d/ directory)

UI

After adhoc discussions with the development community one of the major issues is the FMN web UI:

  • The interface for creating new filters is complicated and confusing.

  • Users have to define filters seperately for different destentations (i.e. email and IRC) This is due to how the database is designed.

Design options

  • jcline’s option: write a RabbitMQ extension. While it would probably be way more efficient we don’t have the knowledge to write or maintain this code. Also, it would require adding headers for the fields we want to filter on, such as usernames and packages, and we’ll probably loose in flexibility.

  • write a fedora-messaging consumer that would triage incoming message and produce add notifications to one queue per destination (email, irc, matrix). They would be on a FMN-specific vhost on the RabbitMQ server. Then write AMQP consumers for these queues to actually send the notifications. This would allow the IRC and Matrix notifiers to maintain a permanent connection to the IRC/Matrix server(s).

This work requires all applications to use Fedora Messaging Message Schemas, because the triage component will rely on schemas to extract information from the messages.

Later on, when we refer to artifacts, we mean artifacts produced by the Fedora infrastructure: packages, containers, modules, flatpaks.

Stakeholders

Requirements

Tracking

  • As a Fedora Contributor, I want to ask to be notified of what happens on artifacts I own

  • As a Fedora Contributor, I want to ask to be notified of what happens on artifacts owned by a group I am a member of

  • As a Fedora User, I want to ask to be notified of what happens on any artifact by entering its type and its name

  • As a Fedora User, I want to be notified of events referring to my username

  • As a Fedora User, I want to be notified of events referring to a group I am a member of (requires a new feature in Fedora Messaging)

  • As a Fedora User, I want to select which groups I want to be notified about among the groups I’m a member of (some groups own so many artifacts it would be too many notifications, eg the python maintainers group)

  • As a Fedora User, I want to be able to follow someone (for example, my mentee)

Filtering

  • As a Fedora User, I want to be able to set the minimum severity level on each rule (Fedora Messages have the notion of severity) (optional)

  • As a Fedora User, I want to be able to block or allow notifications from a particular application (koji, bodhi, dist-git, etc)

  • As a Fedora User, I want to be able to disable notifications for my own actions. They should be disabled by default.

  • As a Fedora User, I want to be able to block or allow notifications from a particular topic. The list of known topics can be obtained from the installed schemas. This overrides the app allow/block list, for example I don’t want koji messages except for the “build complete” ones.

Destinations

  • As a Fedora User, I want to my notifications to be sent via email if I set so.

  • As a Fedora User, I want to my notifications to be sent via IRC if I set so.

  • As a Fedora User, I want to my notifications to be sent via Matrix if I set so (optional).

  • As a Fedora User, I want to be able to choose one or more destination(s) on each rule (my objects, object foo, object bar, etc)

  • As a Fedora user, I want my address for each destination to be retreived from my FAS account. If I have multiple values set for a destination, I want to be able to choose one.

  • As a Fedora User, I want to be able to do filtering on my notification email based on application name and/or topic. A good way to offer this possibility would be to set these values as email headers

Deployment

  • As a Fedora Sysadmin, I want to be able to run multiple instance of any component of FMN in Openshift if I need to, for performance reasons.

  • As a Fedora User, I want only one notification sent when a message matches my filters. Do note that during schema transition periods, the same event may produce multiple messages on different topics (fedora.event.v1 and fedora.event.v2). I think it would be preferrable to subscribe the triage component to a fixed list of known topics instead of subscribing it to everything (#).

  • As a Fedora Sysadmin, I want the Destination Handler components to be as simple and fast as possible because it will not be easy to have multiple instances of them (not sure if IRC allows multiple connections of the same account)

Misc

  • As an Application Developper, I want to set the summary and the body (where applicable) of what will be sent out to users in my message schema.

  • As a Fedora User, I want to set my notification rules via a web UI

Suggested design

../../_images/workflow.png

Data model and configuration

  • There are multiple Tracking Rules defined in FMN, that accept zero or more parameters.

  • There are multiple supported Destinations in FMN

  • There are multiple Filters defined in FMN, that accept zero or more parameters.

  • Users can activate zero or more Tracking Rules and set the corresponding parameters if applicable.

  • Users can set one or more Destinations for a Tracking Rule

  • Users can set one or more Filters for a Tracking Rule and a Destination, and set the corresponding parameters if applicable.

Workflow

  • A Runner is created for each Tracking Rule

  • Incoming messages are passed to each Runner (parallelism is possible here)

  • The Runner applies the Tracking Rule over the messages, which produces a list of usernames to notify

  • For each username:

    • check which destination(s) they want for this Tracking Rule

    • For each destination:

      • check if the incoming message’s properties pass all the Filters for this Tracking Rule and this Destination

      • If so, lookup the desired address in the DB

      • check that the value is still in the user’s FAS profile, or fallback to the FAS profile’s first value

      • Produce a Notification for this incoming message. The Notification will contain the destination name, the address, a subject (optional), and a body.

  • Notifications produced by all the Tracking Rules are de-duplicated before being added to the Destination queues. One incoming message should produce at most one Notification for a user and a Destination.

  • Destination Handlers are separate processes that will get notifications from their queue and send the notification to the corresponding service.

Possible tracking rules

Artifact owner

For each artifact in the message, get the artifact owners. For each owner, check in the DB whether they have selected to be notified for their artifacts. If so, add a notification for each selected destination if they pass filtering.

Artifact owned by one of my selected groups

For each artifact in the message, get the artifact owner groups. For each groups, get the list of members and check in the DB whether they have selected to be notified for artifacts in this group. If so, add a notification for each selected destination if they pass filtering.

Artifact follower

For each artifact in the message, get in the DB the list of users who follow those artifacts. For each of them, add a notification for each selected destination if they pass filtering.

Events referring to me

For each username in the message, check in the DB whether they have selected to be notified for events referring to them. If so, add a notification for each selected destination if they pass filtering.

Followers

For each username in the message, get in the DB the list of users who follow them, and add a notification for each selected destination if they pass filtering.

Possible Filters

The Filters should take the destination into account, to be able to filter a noisy app on IRC but not on email for example.

Application

If the application is in the user’s blocklist for the destination, and not in the user’s allowlist: cancel the notification. The list of supported applications to present in the UI can be extracted from the installed message schemas.

Severity

If the severity is below the minimal severity threshold that the user has set for this destination, cancel the notification.

My actions

If the message results of this user’s action, cancel the notification.

Topic

If the topic matches (fnmatch) any of the user’s topic blocklist entries, and does not match any of the user’s topic allowlist entries: cancel the notification

Existing FMN

Current FMN code and usage analysis

The current FMN application implements a fedmsg based consumer, and a Flask based webui for users to create, modify, and manage their filters.

Current Users

Currently there are 6,777 users with accounts on FMN.

Contexts

A context in FMN is the top-level for a user to define their filters. A context is where a message is sent after being matched. FMN in production has 2 contexts that users can use:

  • email

  • irc

Each of these contexts can be enabled, which disables all the filters defined under that context.

Of the 6,777 users in FMN, the following is a breakdown of these users’ enabled contexts:

  • Both email and irc enabled: 1048 users

  • Only email enabled: 4619 users

  • Only irc enabled: 169 users

  • No contexts enabled: 941 users

Filters and Rules

For each context, a user defines zero to many filters. Each filter is made up of 1 to many different rules which are defined in code.

Filters defaults

When a user logs into FMN for the first time, they are set up with a copy of the default filters.

The current set of default filters and rules are:

  • Events on packages that I own’:

    • [‘fmn.rules:user_package_filter’, {‘fasnick’: username}],

    • [‘fmn.rules:user_filter’, {‘fasnick’: username}],

    • [‘fmn.rules:koji_instance’, {‘instance’: ‘ppc,s390,arm’}],

    • [‘fmn.rules:taskotron_result_new’, {}],

    • [‘fmn.rules:git_pkgdb2branch_start’, {}],

    • [‘fmn.rules:git_pkgdb2branch_complete’, {}],

    • [‘fmn.rules:fedoratagger_catchall’, {}],

    • [‘fmn.rules:faf_report_threshold1’, {}],

    • [‘fmn.rules:faf_problem_threshold1’, {}],

    • [‘fmn.rules:bodhi_masher_start’, {}],

    • [‘fmn.rules:mdapi_repo_update’, {}],

    • [‘fmn.rules:ci_step_complete’, {}],

    • [‘fmn.rules:copr_build_start’, {}],

    • [‘fmn.rules:copr_chroot_start’, {}],

    • [‘fmn.rules:copr_worker_create’, {}],

    • [‘fmn.rules:koji_scratch_build_started’, {}],

    • [‘fmn.rules:koji_build_started’, {}],

    • [‘fmn.rules:koji_rpm_sign’, {}],

    • [‘fmn.rules:koji_tag’, {}],

    • [‘fmn.rules:koji_untag’, {}],

    • [‘fmn.rules:koji_repo_done’, {}],

    • [‘fmn.rules:koji_repo_init’, {}]

    • [‘fmn.rules:koji_package_list_change’, {}]

    • [‘fmn.rules:summershum_catchall’, {}]

    • [‘fmn.rules:anitya_catchall’, {}]]

  • ‘Events referring to my username’: [[‘fmn.rules:user_filter’, {‘fasnick’: username}],]

  • ‘Critical taskotron tasks on my packages’

    • [‘fmn.rules:user_package_filter’, {‘fasnick’: username}],

    • [‘fmn.rules:taskotron_release_critical_task’, {}],

    • [‘fmn.rules:taskotron_task_particular_or_changed_outcome’, {‘outcome’: ‘FAILED’}]

Because the configuration of the default setup has changed over the years, users that logged in earlier have older default filter setups, making it harder to figure out who is using defaults.

Current UI

Login page
apps.fedoraproject.org/notifications/
../../_images/login.png
About page
apps.fedoraproject.org/notifications/about
../../_images/about.png
Profile page
apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/
../../_images/profile.png
Filter list per context
https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/irc/
https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/email/
../../_images/filterlist.png
Create / Edit Filter page
https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/irc/<filter_id>/
https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/email/<filter_id>/
../../_images/createfilter.png

Skillsets

  • fedora messaging

  • Web application UI

  • Python (database)

  • Rabbitmq (used in the backend of FMN and could be useful to know)

  • Openshift

  • IRC - connectivity to new systems in use (matrix)

Open Questions

  • What are the core services the new FMN will need to support?

  • Are there fedora-messaging schemas for each of the core services in place already?

  • Does the schemas need revision, ie do they still work?

  • Who would be good to get a review on UI? Mo Duffy potentially? Could this go in as a blog post of sorts?

Possible Timeline

In 1 month

  • Email notifying community about work started on new FMN service has gone out

  • The development team has been onboarded

  • The backlog has epic level tickets created

  • Testing criteria is loosely defined for performance, ie what is the min/max acceptable time for receiving a notification

In 3 months

  • An update is sent to the lists on work underway

  • Infra leads have been invited to a review of the work to date

  • A firmer timeline for project delivery has been defined

  • There is testing criteria/benchmarking agreed to

In 6 months

  • The service is deployed in staging or its in production but it is not enabled for everyone yet

  • Testers are invited to onboard into the new service and help test performance

  • An outage period for cutover to the new service has been identified and planned if needed

In 9 months

  • Hotfixes are applied