Water crashing together. 3 states of water, from peaceful to disorderly. Intersecting to create beauty in chaos.
There is beauty in the chaos. Photo Credit: Unspalsh @powwpic

Data comes in all shapes and sizes

It is amusing that when we talk about data the best analogy is typically rooted in water. This makes sense in order to fathom the idea of data — which comes in all shapes and sizes— people tend to embrace the abstract. One could say a single data record could…


What inpires you? Photo via Unsplash

In Pursuit of Writing ✍️

I’ve been writing off and on now for the better half of a decade. Mostly technical writing here on medium now a days, but over the course of the past few years, I’ve had a few articles published, some even showed up in printed…


Photo Credit: https://unsplash.com/@swimstaralex

Anyone working in the field of analytics and machine learning will eventually need to generate strong composite grouping keys, and idempotent identifiers, for the data they are working with. …


An Orchestra Playing a song. Violins and people fading into the background
Photo by Larisa Birta on Unsplash

This tutorial is aimed at engineers who want to understand how to get up and running on Spark on Kubernetes. …


Shows a mirror with many different angles and views of the same scene
Many angles provide many views of the same scene. Cache works with partitions similarly. Credit

Apache Spark provides a few very simple mechanisms for caching in-process computations that can help to alleviate cumbersome and inherently complex workloads. This article aims at providing an approachable mental-model to break down and re-think how to frame your Apache Spark computations. …


Creepy cabin in the woods. Setting an alternative stage for Goldilocks to make a point about decision intelligence.
Was this where Goldilocks found the porridge? Photo Credit: https://unsplash.com/@nate_dumlao

Spoiler: Goldilocks survives. Let us first start by saying things could have turned out much worse. Really worse. Don’t walk into a strangers house and eat their food and nap in their house. This is just a no brainer!

In the real world when we make decisions there is a…


An Open Book with a story about tropical pirates, a treasure chest and pirate ships coming off the page.
Storytelling with Data isn’t as whimsical. It does, however, provide the correct mental model. (Source Pixabay)

What is your story?

Across the world, we are producing and consuming data at exponential rates. This is just a fact. With the advent of 5g networks we should be capable of handling up to 10/Gbs up from 300 Mbs from 4g LTE. This means we have the bandwidth to send and consume more…


Source: Pixabay

One subtle and novel paradigms in computer systems architecture is the concept of message passing. This simple, albeit useful construct allowed for magnitude gains in parallel processing by allowing processes (applications/kernel tasks/etc) within a single computer OS to take part in a conversation. …


source: Pixabay

Evolutionary Communication

Communication is the fundamental foundation upon which modern society was built. Without words, pictures or symbols we wouldn’t have been mentally capable of developing the process of being able to formulate complex thoughts or ideas or even to express the most basic of our primal needs. However by some random…


Source: Pixabay

Everyone has worked under the wrong assumptions at one point or another in their careers and in no place have I found this more apparent than when it comes to legacy data and a lot of what ends up in most companies’ data lakes.

The concept of the data lake…

Scott Haines

Software Architect at Twilio in California. I work on real-time, ridiculously large data (*Big Data) applications with Scala and Spark. Voice Insights. ☎️

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store