Under engineering scheduled jobs is just fine

At the FT we’ve long fostered a culture and toolset that enables us to deploy new services quickly. Whether we’re using Heroku or Lambda or EC2 to spin up a JavaScript or Python or Go service it’s usually possible to get things up and running by lunchtime. This is really empowering and can enable us to test and learn quickly with minimal expenditure.

In my team we look after many small systems which execute scheduled jobs to pull, prepare and push data around. The default means to deliver systems like these is to create a set of AWS Lambda functions - which in many cases is what we’ve previously done.

But just because we can deploy a Lambda function in a morning doesn’t mean it’s simple to do so. Lambda functions often require secondary resources like an S3 bucket or database access or routes adding to API Gateway and all of these require setting up and managing of granular permissions and keys. To ready the system for production also requires appropriate monitoring so we’ll need to setup CloudWatch and configure the appropriate forwarders to get those logs and metrics into the FT’s Splunk and Prometheus observability stack.

It’s a lot! The extensive stack created is no longer “just” a Lambda function and setting up, maintaining and monitoring the dozens of resources when you just want to run a script once a day starts to feel like a lot of effort.

Recently my team have released even more jobs to pull and push new data sources around, including a new backup solution for all of the FT’s runbooks, but so far we haven’t created a single piece of new AWS infrastructure to run them. Instead, we’ve been under engineering these jobs by utilising one of our most used and depended upon tools; CircleCI.

We already use CircleCI to automatically execute our tests, integrations and deployments. It’s an environment that we’re familiar with and trust. We know that CircleCI can be used to run almost anything, workflows can be triggered flexibly, log output and retries are a click away, debugging can be as simple as enabling SSH, the included insights dashboard provides useful metrics, and we have existing integrations between CircleCI and our other tools. So whilst a CircleCI pipeline might not be quite as cheap to run as a Lambda function, it can be used to achieve similar results with a lot less upfront time and effort.

Configuring a scheduled job on CircleCI takes minutes. For example, to execute our nightly runbooks backup script we added a new run-backup workflow to our existing project guarded with an execute-backup parameter:

parameters:
  execute-backup:
    type: boolean
    default: false

workflows:
  run-tests:
    unless: << pipeline.parameters.execute-backup >>
    jobs: …

  run-backup:
    when: << pipeline.parameters.execute-backup >>
    jobs: …

To trigger the workflow each day we setup a CircleCI trigger in the UI with the run-backup parameter set to true. Easy.

So far so good, but there was still one challenge to solve - monitoring. In business hours Monday to Friday we can view the job status in CircleCI or GitHub and have alerts sent into a Slack channel, but the operations and support team still need to be made aware when something goes wrong so they can give us a nudge or step in when we’re not around. That means we must integrate our CircleCI project with the FT’s observability platform.

To solve this we simply send a JSON formatted log message to Splunk via cURL at the end of our workflow:

commands:
  …
  send-success-log:
    name: Send SUCCESS log to Splunk
    command: |
      curl -H "Authorization: Splunk $SPLUNK_HEC_TOKEN" $SPLUNK_HEC_ENDPOINT \
      -d '{ "sourcetype": "json", "index":"runbooks-backup", "event": "RUNBOOKS_BACKUP_SUCCESS" }'

  send-fail-log:
    name: Send FAILED log to Splunk
    command: |
      curl -H "Authorization: Splunk $SPLUNK_HEC_TOKEN" $SPLUNK_HEC_ENDPOINT \
      -d '{ "sourcetype": "json", "index":"runbooks-backup", "event": "RUNBOOKS_BACKUP_FAILURE" }'

workflows:
  …
  run-backup:
    when: << pipeline.parameters.execute-backup >>
    jobs:
      …
      - send-success-log:
          when: on_success
      - send-fail-log:
          when: on_fail

With log messages now reaching Splunk we can configure an alert which will be propagated across the FT’s observability dashboards so any issues will be spotted by the support team. It’s straightforward and once again takes only a few clicks to configure.

So far we’ve been very pleased using CircleCI to execute our scheduled jobs. They take minimal effort to setup, they’re straightforward to monitor and diagnose any issues, and there’s very little to maintain.