Scott Haines – Medium

Published in Towards Data Science

·Pinned

From Data Lakes to Data Reservoirs

Create Clean, Beautiful, Protected Data Resources with Apache Spark and Delta Lake — Data comes in all shapes and sizes It is amusing that when we talk about data the best analogy is typically rooted in water. This makes sense in order to fathom the idea of data — which comes in all shapes and sizes— people tend to embrace the abstract. One could say a single data record could…

Apache Spark

12 min read

3 days ago

Data+AI 2020: Spark Inception:

Harnessing the Spark REPL to power Streaming Notebook Environments. For Fun. For Profit? — This blog post is essentially covering the gist of the presentation, and providing all useful links to the source code, and how-to docs. A friend came to me today at 3:45pm and looked at the slides I was going to present in a matter of some odd minutes (4:45pm), and…

Apachespark

14 min read

Published in Level Up Coding

·Feb 25

Towards Analytics with Redis

Using Hashes and Binary Data Structures — You are probably wondering why anyone would do any kind of analytical processing directly in Redis. Why not just stick to traditional OLAP-style SQL? I’m glad you asked. Redis has been around for a while now and is a proven, high-performance, low-maintenance database and excellent multipurpose cache. Redis is special…

Redis

12 min read

May 26, 2021

Problems All Writers Face

Learning to break mental blocks and unstuck yourself — What inpires you? Photo via Unsplash In Pursuit of Writing ✍️ I’ve been writing off and on now for the better half of a decade. Mostly technical writing here on medium now a days, but over the course of the past few years, I’ve had a few articles published, some even showed up in printed…

Publishing

5 min read

Published in Towards Data Science

·Mar 11, 2021

Analytical Hashing Techniques

Spark SQL Functions to Simplify your Life — Anyone working in the field of analytics and machine learning will eventually need to generate strong composite grouping keys, and idempotent identifiers, for the data they are working with. …

Apache Spark

4 min read

Published in Level Up Coding

·Sep 29, 2020

Spark on Kubernetes

A Hands-On Introduction: Getting Up and Running. — This tutorial is aimed at engineers who want to understand how to get up and running on Spark on Kubernetes. …

Apache Spark

18 min read

Published in Towards Data Science

·Feb 8, 2020

Cache Patterns with Apache Spark

Balancing the choice between disk, ram and the source of truth — Apache Spark provides a few very simple mechanisms for caching in-process computations that can help to alleviate cumbersome and inherently complex workloads. This article aims at providing an approachable mental-model to break down and re-think how to frame your Apache Spark computations. …

Apache Spark

9 min read

Published in The Startup

·Jan 11, 2020

Decision Intelligence and why Goldilocks made bad choices

How observation, probability and common sense leads to better decision making — Spoiler: Goldilocks survives. Let us first start by saying things could have turned out much worse. Really worse. Don’t walk into a strangers house and eat their food and nap in their house. This is just a no brainer! In the real world when we make decisions there is a…

Decision Intelligence

5 min read

Published in Towards Data Science

·Oct 9, 2019

Storytelling with Data

The subtle art of good analytics and explainable insights — What is your story? Across the world, we are producing and consuming data at exponential rates. This is just a fact. With the advent of 5g networks we should be capable of handling up to 10/Gbs up from 300 Mbs from 4g LTE. This means we have the bandwidth to send and consume more…

Big Data

6 min read

Published in 97 Things

·Jun 8, 2019

Pipe Dreams

How messaging, scalability and failure formed a new kind of architecture. — One subtle and novel paradigms in computer systems architecture is the concept of message passing. This simple, albeit useful construct allowed for magnitude gains in parallel processing by allowing processes (applications/kernel tasks/etc) within a single computer OS to take part in a conversation. …

Data Engineering

2 min read