Onyx::Background

A fast background job processing for Crystal.

About

This is a Redis-based background jobs processing for Crystal alternative to sidekiq and mosquito. It's a component of Onyx Framework, but can be used separately.

Features

This publicly available open-source version has the following functionality:

• Enqueuing jobs for immediate processing
• Enqueuing jobs for delayed processing (i.e. in: 5.minutes or at: Time.now + 1.hour)
• Different job queues ("default" by default)
• Concurrent jobs processing (with a separate Redis client for each fiber)
• Verbose Redis logging of all the activity (i.e. every attempt made)
• Moving stale jobs (i.e. with dead workers) to the failed list

Performance

Thorough benchmaring is to be done yet, however, currently a single Worker is able to process more than 7500 jobs per second on my 0.9GHz machine with Redis instance running on itself.

Installation

Onyx::Background works with Redis version ~> 5.0.

Add this to your application's shard.yml:

dependencies:
  onyx-background:
    github: onyxframework/background
    version: ~> 0.1.0

This shard follows Semantic Versioning v2.0.0, so check releases and change the version accordingly.

Usage

API docs

Please refer to API documentation available online: https://api.onyxframework.org/background

Note: it is updated on every master branch commit

Example

require "onyx-background"

struct Jobs::Nap
  include Onyx::Background::Job

  def initialize(@sleep : Int32 = 1)
  end

  def perform
    sleep(@sleep)
  end
end

manager = Onyx::Background::Manager.new
manager.enqueue(Jobs::Nap.new)

puts "Enqueued"

logger = Logger.new(STDOUT, Logger::DEBUG)
worker = Onyx::Background::Worker.new(logger: logger)
worker.run

$ crystal app.cr
Enqueued
I -- worker: Working...
D -- worker: Waiting for a new job...
D -- worker: [fa5b6d65-46fe-4c88-829f-d69023c4c6de] Attempting
D -- worker: [fa5b6d65-46fe-4c88-829f-d69023c4c6de] Performing Jobs::Nap {"sleep":1}...
D -- worker: Waiting for a new job...
D -- worker: [fa5b6d65-46fe-4c88-829f-d69023c4c6de] Completed

It's also highly recommended to run a Watcher process to watch for stale jobs. If you're queuing delayed jobs, the Watcher is required to move the jobs to the ready queue on time:

# watcher.cr
require "onyx-background"

watcher = Onyx::Background::Watcher.new
watcher.run

Development

Redis is flushed during the spec, so you must specify a safe-to-flush Redis database in the REDIS_URL. To run the specs, use the following command:

$ env REDIS_URL=redis://localhost:6379/1 crystal spec

Contributing

1. Fork it (https://github.com/onyxframework/background/fork)
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

Contributors

• @vladfaust Vlad Faust - creator, maintainer

Licensing

This software is licensed under BSD 3-Clause License with "Commons Clause" License Condition v1.0. See LICENSE.