Posted on Mar 14, 2019
Odd one this, and one that took me a little while to debug. I recently set up a Confluent/Kafka data pipeline with transformations being handled by KSQL and data being produced by an application written in Go. As part of the test process I persisted data using a MongoDB Sink connector. The command line producers had no problems and producing a large file would persist the expected data to MongoDB. However, I ran into issues when producing from Golang, I would notice that somewhere between 7% and 12% of the messages were being persisted to MongoDB, the others were lost somewhere in the processing.
Posted on Oct 03, 2018
If you've ever written a test in golang, chances are you've used the testify package to make writing tests a breeze. An immensely useful extension of that is the mockery package, which gives you the ability to autogenerate testify mocks for interfaces with ease.