Avenue Code - Contract

I provided consultancy for one of Avenue Code's partner companies. My role involved replacing resource-intensive Airflow jobs with a compact Rust binary. This new solution pushed transformed Kafka results to an AWS S3 bucket, which were subsequently processed by Python Lambdas.

Avenue Code

Business Needs

  • Engineers who previously built an inventory matching system using Rust had since moved on and were no longer available for further knowledge transfer.
  • Business group wanted to make use of the inventory matching system, extending it to match key availability from physical lock boxes.
  • Business metrics were being captured using expensive Airflow instances. Jobs on these instances were run infrequently and management wanted to reduce the high associated costs.

Solutions Provided

  • Evaluate and decipher the past Rust inventory matching system architecture and design.
  • Document how the inventory matching system works and how it can be further extended for key lock box matching.
  • Present findings and answer questions from current employees (NodeJs/Javascript/Typescript skilled).
  • Develop a microservice written in Rust that would ingest Kafka events, transforming them as necessary and making the stream available to their Data Lake, effectively removing the reliance on expensive Airflow batch workflows.
  • Design, implement, test, and deploy services using continuous delivery.
  • Linux environment using:
    • Rust
    • Actix-web
    • Kafka
    • MongoDB
    • Redshift
    • AWS Lamdba
    • NewRelic
    • Python
    • Docker
    • Terraform