Posted on May 22, 2020
I've recently been working on an Apache Kafka/Confluent data pipeline to analyse event streams. I decided to use Google Cloud BigQuery for the data analysis as it seemed to be easy to get set up with and extremely powerful. But to get up and running I'd need to backfill all my existing data. I also decided to add it to a time-partitioned table to increase performance and reduce costs.
Posted on Mar 14, 2019
Odd one this, and one that took me a little while to debug. I recently set up a Confluent/Kafka data pipeline with transformations being handled by KSQL and data being produced by an application written in Go. As part of the test process I persisted data using a MongoDB Sink connector. The command line producers had no problems and producing a large file would persist the expected data to MongoDB. However, I ran into issues when producing from Golang, I would notice that somewhere between 7% and 12% of the messages were being persisted to MongoDB, the others were lost somewhere in the processing.
Posted on Oct 03, 2018
If you've ever written a test in golang, chances are you've used the testify package to make writing tests a breeze. An immensely useful extension of that is the mockery package, which gives you the ability to autogenerate testify mocks for interfaces with ease.
Posted on Feb 01, 2018
I recently moved into some co-working space and I immediately had issues with the internet connection. My connection would be intermittent and would frequently drop; sometimes I had hours of uninterrupted connectivy, at other times it wouldn't last more than a few seconds without dropping completely. I had no issues with any other wifi network at home, in offices or in public.
Posted on Nov 09, 2017
There's a bug with Eloquent's firstOrCreate() function where it doesn't work with the $casts attribute on the model; it will always either insert a new instance into the database or give a MySQL duplicate entry error.