Archyblog


Ramblings of yet another developer!
  • Migrating to Partitioned Native Google BigQuery Table From External GCS Files

    Posted on May 22, 2020

    I've recently been working on an Apache Kafka/Confluent data pipeline to analyse event streams. I decided to use Google Cloud BigQuery for the data analysis as it seemed to be easy to get set up with and extremely powerful. But to get up and running I'd need to backfill all my existing data. I also decided to add it to a time-partitioned table to increase performance and reduce costs.

  • Kafka/KSQL Streams Lost When Producing With Golang

    Posted on Mar 14, 2019

    Odd one this, and one that took me a little while to debug. I recently set up a Confluent/Kafka data pipeline with transformations being handled by KSQL and data being produced by an application written in Go. As part of the test process I persisted data using a MongoDB Sink connector. The command line producers had no problems and producing a large file would persist the expected data to MongoDB. However, I ran into issues when producing from Golang, I would notice that somewhere between 7% and 12% of the messages were being persisted to MongoDB, the others were lost somewhere in the processing.

  • Mockery: Mock Return Value Provider Function With Multiple Values

    Posted on Oct 03, 2018

    If you've ever written a test in golang, chances are you've used the testify package to make writing tests a breeze. An immensely useful extension of that is the mockery package, which gives you the ability to autogenerate testify mocks for interfaces with ease.

  • Intermittent Connection/High Packet Loss with Intel Wireless Driver iwlwifi - Ubuntu/Linux NetworkManager

    Posted on Feb 01, 2018

    I recently moved into some co-working space and I immediately had issues with the internet connection. My connection would be intermittent and would frequently drop; sometimes I had hours of uninterrupted connectivy, at other times it wouldn't last more than a few seconds without dropping completely. I had no issues with any other wifi network at home, in offices or in public.

  • Laravel Bug: Eloquent firstOrCreate() Broken With $casts Model Attribute

    Posted on Nov 09, 2017

    There's a bug with Eloquent's firstOrCreate() function where it doesn't work with the $casts attribute on the model; it will always either insert a new instance into the database or give a MySQL duplicate entry error.