Skip to content

Weather Watchdog

This example application compares weather forecast information from Open-Meteo with meteorological airport reports (METAR) from AVWX in real time. It will retrieve the forecast every day for the next 24h and for a number of airports using their geographical coordinates. Then every time some new METAR data is available, it will compare it with the forecast.

This showcases how Renelick can orchestrate tasks, events and data all together in a single application with several services interfacing with third-party APIs. The produced output is real data and might happen to be interesting to look at, however it's primarily a by-product for illustrative purposes.

Specifications

Input sources

  • Weather stations: METAR from airports around the world
  • Forecast services: Open-Meteo

Tasks

  • Collecting data as Node objects
  • Processing data: averages, comparisons, correlations etc.
  • Generating public output

Output data

  • Charts, maps
  • Notifications by email and pub/sub events

Flow

A timer service sends pub/sub events at regular intervals with the name of an airport (ICAO code). This triggers some tasks to go and fetch the corresponding METAR data, and if it's different than the last sample then it gets added to the database as a metar node. Only temperature, humidity and wind measurements are being stored. The app keeps a list of airports to monitor; a separate event is sent for each of them every 10 minutes.

Meanwhile, another timer service sends hourly events to request new weather forecast data for the current day. This in turn triggers a task to go and retrieve it from Open-Meteo and store it as a forecast node.

Every time a new METAR entry is added, a pub/sub event is generated which triggers a task to look at the data, compare it against the forecast and decide what to do with it - not implemented yet. Typically, it will store a sub-tree of data nodes in the database with the result of the comparison, using the METAR entry as the parent node.

Instructions

Renelick weather user

A user account called weather is needed to run this application. To create one, run this command with an arbitrary email address:

. scripts/devenv
rki register weather@renelick.io weather
rki login --method=key weather
uid=$(rk get-user --user=weather | jq -r .id)
docker-compose exec api renelick-admin $uid set verified

Note: Setting the full name of a user can currently only be done via the web dashboard. For example, in this case it could be set to "Weather Watchdog".

Then to confirm it worked as expected:

$ rk whoami --user=weather
Authenticating with persistent API key
Connecting to http://172.17.0.1:8000
User profile:
  id             66d9a3245ab47a1fb6aa0433
  username       weather
  email          weather@renelick.io
  full_name      Weather Watchdog
  is_superuser   False
  is_verified    True

External APIs

The METAR data is retrieved from AVWX which requires an API token with a free account. Please follow the Getting Started steps to create a token.

It needs to be stored in the weather user's credentials, which can be done interactively with this command:

rki add-credential weather avwx token

The app will then be able to find it and load it from the stored credentials to access the AVWX API.

No key is needed for non-commercial Open-Meteo clients such as this app, so nothing special needs to be done for it here.

Docker Compose

Like all sample applications provided in this repository, it can be started using docker-compose. Typically, you may want to leave it running in the background to let it gather data over some period of time. This can be done easily by starting the services in detached mode and then monitor the logs:

docker-compose -f apps/weather/docker-compose.yaml up -d
docker-compose -f apps/weather/docker-compose.yaml ps
docker-compose -f apps/weather/docker-compose.yaml logs -f orchestrator

The output should look like this:

          Name                        Command               State    Ports
----------------------------------------------------------------------------
weather_orchestrator_1     /usr/bin/env python3 /home ...   Up      8000/tcp
weather_scheduler_1        /usr/bin/env python3 /home ...   Up      8000/tcp
weather_timer-forecast_1   /usr/bin/env python3 /home ...   Up      8000/tcp
weather_timer-metar_1      /usr/bin/env python3 /home ...   Up      8000/tcp
Attaching to weather_orchestrator_1
orchestrator_1    | Starting orchestrator
orchestrator_1    | User: weather
orchestrator_1    | API:  http://172.17.0.1:8000
orchestrator_1    | Time                  METAR Airport             Temp     Humid   Wind
orchestrator_1    | forecast-LFPG-2024-09-11
orchestrator_1    | forecast-KJFK-2024-09-11
orchestrator_1    | forecast-RJTT-2024-09-11
orchestrator_1    | forecast-SBGR-2024-09-11
orchestrator_1    | 2024-09-11T08:30:00Z  LFPG  Paris (FR)          13.0ºC   82.0%    9.3 km/h
orchestrator_1    | 2024-09-11T07:51:00Z  KJFK  New York (US)       16.0ºC   67.1%    5.6 km/h
orchestrator_1    | forecast-FACT-2024-09-11
orchestrator_1    | 2024-09-11T08:00:00Z  RJTT  Tokyo (JP)          31.0ºC   74.8%   27.8 km/h
orchestrator_1    | 2024-09-11T08:00:00Z  SBGR  São Paulo (BR)      15.0ºC   77.0%
orchestrator_1    | 2024-09-11T08:00:00Z  FACT  Cape Town (ZA)      18.0ºC   45.3%    5.6 km/h
orchestrator_1    | forecast-YPPH-2024-09-11
orchestrator_1    | 2024-09-11T08:30:00Z  YPPH  Perth (AU)          24.0ºC   41.1%   20.4 km/h

Summary

The application currently only creates two kinds of nodes: weather.metar and weather.forecast. The next things to implement to make it more interesting would be to compare the METAR results with the forecasts and add other kinds of nodes with the result.

For now, let's take a look at how we can access the existing data on the command line. For example, to get the latest METAR report for Cape Town:

rk find-nodes \
  kind=weather.metar \
  data.airport="Cape Town (ZA)" \
  created__sort=down \
  --limit=1 \
  --indent=2

{
  "id": "66e1562df1292abef84fbead",
  "name": "metar-FACT-2024-09-11T08:00:00Z",
  "parent": null,
  "artifacts": {},
  "kind": "weather.metar",
  "data": {
    "icao": "FACT",
    "airport": "Cape Town (ZA)",
    "coordinates": {
      "latitude": -33.9648017883,
      "longitude": 18.6016998291
    },
    "time": "2024-09-11T08:00:00Z",
    "humidity": [
      0.45322763633169727,
      null
    ],
    "temperature": [
      18.0,
      "C"
    ],
    "wind_speed": [
      3.0,
      "kt"
    ]
  },
  "task": {
    "attributes": {
      "name": "metar",
      "icao": "FACT"
    },
    "scheduler": "weather-inline",
    "id": "d67708ec-a22f-4c1c-9fef-82b0852e5cfa",
    "timeout": "2024-09-11T08:35:52.414000"
  },
  "path": [
    "metar-FACT-2024-09-11T08:00:00Z"
  ],
  "created": "2024-09-11T08:34:53.792000",
  "owner": "weather"
}

Any field can be used, for example to look for entries with a temperature above 15°C in the southern hemisphere, and using jq to filter the output:

rk find-nodes \
    kind=weather.metar \
    data.coordinates.latitude__lt=0 \
    data.temperature__gt=15 \
    --limit=3 \
    | jq '.data.airport, .data.time, .data.temperature'

"Perth (AU)"
"2024-09-11T08:30:00Z"
[
  24,
  "C"
]
"Cape Town (ZA)"
"2024-09-11T08:00:00Z"
[
  18,
  "C"
]

As a first case study, this is already making use of many Renelick features:

  • trigger events
  • orchestration logic
  • task scheduler
  • data nodes
  • user authentication
  • command-line tool
  • containerised deployments

One particular aspect that's not covered here is the tree structure of the data, as the current nodes don't have any hierarchy. This will start to make more sense with post-processing of the collected data.

That's all, Folks! To be continued...