Introduction
This story began two months ago with a seemingly innocuous question. Over the subsequent several weeks, I realized that the computational work I was doing in Alteryx to answer the question took me to the edge of data comprehension and sanity. That isn’t necessarily a bad thing because it proves the power of Alteryx. Luckily, what saved me was being able to visualize the data being created to answer the question. In a sense, Tableau brought me back to reality and restored my sanity.
This interplay between intensive computation and visualization was both an exciting and unexpected development for such an innocuous question, so I decided to write this story.
Background
This story is about actual, big data that exists everywhere and is collected across planet Earth. The multiple data sources are complex, voluminous, and all around us. These data are collected and processed in different ways.
In other words, this case is another example of a gigantic data blending operation that I have grown accustomed to doing using data that has not been designed to work together. It is these types of projects that require the combined usage of Alteryx and Tableau if you want to work efficiently and with high speed.
Our brains seamlessly synthesize this data daily as we move from place to place. The massively parallel processor we have on board (i.e., our brains) can use the various pieces of information to help us navigate the world. Unfortunately, I can’t share detailed information about the project due to confidentiality. Still, I hope to tell the tale nonetheless from the point of view of the worker and the problem-solving team.
My Teammate
I have a partner on this case. He has shown up on this blog before. He is a guy of exceptional ingenuity, of seemingly unlimited potential to untangle complex problems and create working solutions. Being able to brainstorm with someone having his talents is a joy in itself. I’ll call him the Ranger because the scope of his capabilities is enormous (i.e., he covers a lot of ground). In real life, he is known as Allan Walker and he works for Mapbox.
For a couple of months, before I even knew of this project, the Ranger was planning and plotting. He was crunching numbers like a mad scientist, running geographical operations that would hurt your brain if I dared tell you about them. When the Ranger dropped the data sources on me, I felt the digital load like I was performing a heavy deadlift. It was gigabytes of information coming to me in full complexity. I knew I was going to be putting the hurt on Alteryx and Tableau in this case.
The Initial Iteration
At first, the Ranger and I thought we had the project completed within a week. We presented the solution, and the people who requested the work said something like this:
Yes, that is nice. That is indeed very nice. However, what we really want is different than that. We want you to think about the problem like this and present the findings in this way…
Well, Ranger and I returned to the drawing board and began the work needed to conceptualize a working solution. This was when the cycle of weekly updates began, and the second iteration of the project started. Each week we got together and presented results. Each week, we could see new potential, and we received new input on what “we would like to see.” The project demands were growing, as were the expectations of what could be done.
The Second Iteration
As the first five weeks passed, we realized we had experienced significant scope creep on an already challenging problem. During the weekend between weeks 5 and 6, I decided to go deep on the project. This was nearly a big mistake.
Instead of taking the easy way out and letting the Ranger do his magic by burning through hundreds of millions of intensive computations, I wrote an Alteryx workflow to replicate some of the output he would have created.
It was during this time that I had to focus so intently that I realized that I had seemingly walked to the edge of the Grand Canyon and peered over the edge. I wondered whether I should jump or whether I should stay on top. My sanity was in the balance.
The work I was doing was stacking up to be so hard that I realized it was becoming the most demanding professional task that I have ever attempted in my life. By changing directions and having the courage to build a new approach, I unwittingly brought myself to the edge of sanity. For many days, if not weeks in a row, my brain was put into overdrive, constantly computing. I was becoming fatigued in a big way.
I was having trouble deciding if what I was doing was right or wrong. There were too many combinations and permutations to decide if my work was accurate. A little later in this story, you will hear how Tableau saved my sanity and kept me from jumping. You will also learn about a method that can help you save your sanity when you are working on tough-to-solve cases that have data that features a lot of variation and natural complexity.
How to Navigate the Road Called Complexity
I remember the first time I experienced a eureka moment when solving a complex problem. It was as though it happened this morning, although it was in the late 1980s. In my mind, a television commercial exists of this short story, and I can replay that scene whenever I want. That is the powerful impact that these types of moments can have on us.
The imaginary TV camera that filmed my commercial hung from the ceiling just outside my shower curtain. As I leaned my face into the powerful spray of water coming from the showerhead, the solution instantly appeared in my mind. I don’t know how that happened (other than my brain finally solved it while doing background processing), but whatever magic was in that water allowed me to travel an entirely new path toward success. This path was in the opposite direction compared to the path down the road of complexity.
For two or three months before my eureka moment, I was building a contouring algorithm that could be used to post-process results from the numerical models I was building. As the algorithm developed, more and more conditions were needed in my code for things like how to label the contour lines so that they were nicely spaced and produced publication-quality graphics.
The bookkeeping necessary to do these types of things quickly began drowning me in a sea of code. I didn’t have a librarian to help me organize the instructions, so one problem popped up, and once it was solved, another problem was created. It was like watching a dog trying to chase its tail. The complete solution was just out of reach until that morning in the shower.
When the solution hit, I knew I was done. I went to work, scrapped a couple of months of coding, and rewrote everything. I was done in a day or two with all the accomplishments I intended to make.
After that experience happened, I often wondered why the situation unfolded the way that it did. Eventually, after three more decades of serious number crunching and programming, I have learned a few lessons about solving complex problems.
The Lessons Learned
The most important lesson I have learned is a simple one in concept, but it may not be so easy to follow in practice. The lesson is that we have to learn to relax while solving intensive problems. A part of that relaxation is taking the time to think through the problem.
By envisioning the problem and hitting it from different directions, you will create alternative solution techniques that can save you a lot of grief. Although it is tempting to sprint towards the finish line, on the road to complexity, you will be fooled into believing you have made all the right turns.
To explain how this feels, I offer an analogy. Back in my undergraduate years, we learned to look at 3D geological images through stereoscopes. To see the earth in 3D, two aerial photographs that were taken from slightly different positions are laid next to each other. You look at these images through magnifying lenses. If you can let your visual field relax enough, the flat 2D images suddenly jump into 3D, and you feel like a bird flying over the landscape.
It’s that ability to let your vision relax that is similar to what is necessary when attacking a complex problem. In the current case, I had such an intense focus on the problem that the solution only came into view when I let my brain relax.
When the complexity of the problem is overwhelming, the solution is seemingly just out of reach. The real magic happens when you can relax and think of simple solutions that can become part of the composite solution to the complex problem. If you can stack a few of these simple ideas together, the complexity evaporates, and the answers appear.
We often feel that the brute force method is the way to attack complex problems. We now have the luxury of faster computers, terrific software, and ample storage devices, so we naturally sprint for the finish line. We can become complacent and depend upon these things. Sometimes brute force is vital, but sometimes you get bogged down in the details, and you can never reach the solution. This was one of those cases. Luckily for me, I’d been down that road a few times before, so I knew what to do.
The second most important lesson I have learned is one that I cannot entirely control. Once I focus on finding a solution to a problem, I cannot turn my brain off. In other words, my brain keeps computing until a solution is found.
The challenge presented by this situation is this. You may become distant. Sometimes you will be in a “zone,” and people talking to you can’t reach you. The solution to this is working on our ability to stay present, even when our brain doesn’t want to leave the problem-solving space. You have to learn to allow your brain to do its work while at the same time moving back into the other aspects of your life to find balance. This isn’t always easy to do.
Tableau to the Rescue
In the current case, the relationship between two disparate data sources had never been investigated. I didn’t know how one set of measurements computed from one data source would correspond to a measurement made in the other. These measurements were not related in any way. This meant that a device made one measurement, and the other measurement was a natural consequence of how we have built upon the planet. The only thing they had in common was complexity.
When I created my simple solution, I had to compute a directional component of the data. This meant that I had to choose a coordinate system for my calculations. At the time I did this, I didn’t know the orientation of the other data set that was collected by the device because I was waiting for an answer on that topic. For this reason, I chose a coordinate system that I learned in geology. It turns out I was wrong with that choice.
When I created a scatter plot (Figure 1), I immediately knew that there was a correspondence between the data sets, but it appeared that they were not made on the same coordinate system. I expected the data to be oriented along the 45-degree line, which was 90 degrees off based on the plot I created in Tableau. Additionally, I was baffled about the banding. I wondered how the data could create that type of banding. This was a case of data visualization giving insight into the underlying nature of the data. Without the visualization, I couldn’t have predicted this type of relationship.
Figure 1 – The coordinate systems were not in sync. The data should have gone from lower left to upper right.
In one sense, I was excited about these results. On the other side of my fatigued brain, however, I was worried about the difficulty of formulating the calculations for the new coordinate system. In studying Figure 1, I deduced that the coordinate systems were not only starting at different points, but they were also going in different directions! Yikes, is what I thought at the time!
Well, my high school trigonometry teacher would have been proud of me. I did a little research on the behavior of the arctangent function, and before I knew it, Figure 2 was born. Once I plotted the data on maps and wrote a hellacious calculated field, I understood the banding.
Figure 2 – The coordinate systems were in sync, and the banding was understood.
Without the maps and spatial relationships visualized in Tableau, I would not have been able to write that calculated field. Once again, Tableau saved my sanity. Unfortunately, I can’t show the calculation but I’ll say that it took me about two hours to document what it does.
When data from two different systems come together to form a picture of how things are related, it is an exciting moment. Situations like this take me back to my early days of Tableau usage when I had moments where the visualizations blew me out of my chair (for more, read at least the Part 1 article in this link). That is one incredible aspect of Tableau – it allows you to understand relationships between things that you couldn’t otherwise envision or maybe even comprehend.
Wrapping Up
I am currently wrapping up the project. This means that I am still investigating the accuracy of the work. I’m learning how to comprehend and explain what was accomplished. I have to simplify the story to convey what the information is telling us.
I’m trying to document and visualize the most important findings so that people who need the information will be able to understand it. Learning to tell the story, devoid of all the complex calculations, takes practice. Without the combination of Alteryx and Tableau, however, this repeatable analytical framework would never have been born. To do this through manual programming would have taken a very long time.
To my friend Ranger, thank you again for being a great friend and teammate. Thanks for reading.
Pingback: Using Alteryx to Perform Analysis of NASCAR Races | Data Blends