Skip to content

Renelick logo

Data-Driven Task Orchestration

Renelick is a full-stack solution to orchestrate automated tasks and manage the generated data. Its historical use-case was upstream Linux kernel testing and this can still be done. However, it has now been refocused to become a generic framework for automating pretty much anything: microgrids, home appliances, IT systems monitoring, DevOps...

Renelick cycle

It brings together the classic concepts of events, tasks and data:

  • New data generate events
  • Events trigger tasks
  • Tasks produce new data

It's also possible to use these three features independently to just store and retrieve data, schedule tasks or send messages between services. Applications decide how to make use of this for their own purpose and how to integrate with other systems.

Status & Roadmap

The overall design has now been finalised in many respects although parts of the code are still very much a work in progress. The focus is now on reaching the first v1.0 stable release candidate milestone. For this to work, a sample Weather Watchdog application as well as VIXI, a full-fledged kernel testing project are being developed in parallel.

The implementation is entirely in Python and API bindings for client libraries may be written in other languages based on the OpenAPI description. A standalone deployment is provided using Docker Compose for development and evaluation purposes as described in the Quick-Start Guide. An early Cloud production deployment is also available for Kubernetes with a full solution for GKE. See the k8s directory for more details.

Runtime environments are where tasks get scheduled and run. This can be done anywhere as long as a driver is available. As of today, the inline driver can be used to run tasks direclty in the current Python process and tekton for running Cloud-native pipelines in Tekton. Other runtimes on the roadmap are GCP on-demand VMs, LAVA and local Docker and QEMU.

A basic storage solution is available using SSH for file uploads and Nginx for downloads. For a production deployment, any third-party storage provider may be used with a dedicated driver. The only requirement is that files be reachable via direct URLs since this is how artifacts are managed in the database.

Concepts

As per the introduction, the Renelick architecture is based on three concepts:

  • Nodes are objects stored in the database via the API, to contain arbitrary data and form a tree.
  • Tasks are scheduled by Orchestrators via message queues, then executed on the receiving end by Schedulers in Runtime environments. They typically produce Node objects with some data and upload some artifacts to storage.
  • Events are sent via the pub/sub interface to let Orchestrators know when Nodes are created and generally speaking to provide conditions for scheduling tasks.

There isn't any enforced workflow as all the parts of the system can interact in many different ways to match very different use cases. Here's a typical scenario:

  1. A trigger event is sent to the pub/sub interface, either by a timer or some external service.
  2. Orchestrators receive the event, determine whether it matches any of their conditions and schedule tasks accordingly to the appropriate message queues.
  3. Schedulers receive the task messages from their dedicated queues and run their tasks in a designated Runtime environment.
  4. Tasks will run to complete some arbitrary work and typically upload artifacts to storage, create node objects and update their status with the API when they're done.
  5. Events are published automatically when node objects are created, which the Orchestrators may want to receive as per step 2.

In parallel, other client-side services may be receiving pub/sub events to perform their own work independently of the orchestrators. A basic application will have a single orchestrator, but several could be run to implement different features. It's also possible in principle to have services interacting with different API instances in a truly distributed, federated way although this hasn't been fully characterised yet.

Stack Overview

Here's a basic overview of the current stack, from top to bottom:

Client

  • Command-line tools
  • Web frontend
  • Client-side services
  • Orchestration logic
  • File storage abstraction

API

  • Endpoints and object models with OpenAPI
  • Data handling via Node objects
  • Events via publisher / subscriber interface
  • Task scheduling via message queues
  • User accounts & authentication

Infrastructure

  • MongoDB for the database
  • Redis for pub/sub, message queues and caching
  • Container deployments: Docker Compose, Kubernetes
  • Third-party storage solutions

Development

Please take a look at the Quick-Start Guide to set up a local development instance. The code is all publicly hosted on GitLab with GitLab CI enabled to automatically run static checks and end-to-end smoke tests on merge requests.

License

Unless explicitly mentioned otherwise, the Renelick software is licensed under GNU LGPL v2.1 or later.

The Renelick frontend is licensed under MIT.

The Renelick documentation is licensed under CC BY-SA 4.0