Pablo A. Guzzi, Chief Data & Analytics Officer, Ualá
For several intense years now, I have been dedicated to transforming organizations into data-driven companies.
Those who work in this field know how challenging it can be. Being data-driven can be summarized in a tweet as:
“Making decisions based on data.”
Sounds good, right? But how do we ensure that it is not just the responsibility of one person, but that the entire organization is focused on asking unbiased questions, gathering the necessary data to answer them, analyzing that data, and ultimately using the insights from the analysis to make informed decisions?
If changing individual habits is already difficult, we must understand the magnitude of changing the habits of an entire organization. How do we achieve that?
If you ask me, I believe that one of the most important things one can do to drive change is to have influence, to make the most influential people in an organization aware of the advantages of being data-driven
If you ask me, I believe that one of the most important things one can do to drive change is to have influence, to make the most influential people in an organization aware of the advantages of being data-driven.
Some time ago, and always with this goal in mind, I found myself in a meeting with the heads of my company. It was a great opportunity to convey the benefits and advantages of being data-driven. To do this, I started to conduct thorough research, investing a significant amount of time consuming everything related to the topic.
I found a lot of inspiration and ideas, but nothing was exactly what I wanted, nothing seemed to have the impact I needed to raise awareness about the true effects of being data-driven.
While immersed in this search, a former classmate from my MBA program sent me a video and said, "Do you remember this?" It was a video we had watched in a course that discussed cognitive biases. It was a video of an experiment conducted by Daniel Simons at Harvard University in 2010, and even after more than 10 years, it was still highly relevant.
The video involved asking viewers to count the number of passes made by a team wearing white jerseys in a basketball game. The only premise and objective of the video was to determine the number of passes made by the white team. Let's watch it-
(Monkey Illusion original)If you made it to the end of the video paying attention, many of you may have counted the correct number of passes, while others may have counted a few more or a few less. But what is striking is that half of you (probabilistically speaking) did not see the gorilla.
Even if you saw the gorilla, I'm sure you missed other details that occurred in the video. Like the player in black who leaves the scene, or the gradual change in the background curtain color. Someone might even question the number of passes made by the black team instead of the white team.
You may be wondering what this has to do with being data-driven. The reason we don't see the gorilla while counting the passes is due to a psychological effect: a cognitive bias. This bias can be mitigated thanks to the power of data and analytics.
That's where my humble "eureka" moment came in. This could be solved with an algorithm, eliminating the biases we have. Not only that, we could do many more things than what the original author intended.
It is a complex problem, in fact, it's not just one algorithm, but several working together. But I started to develop an AI that would allow me to answer the initial question from the video, taking into account the complexity of detecting people, the ball, ensuring the algorithm doesn’t confuse them when they intersect or overlap, etc.
I became so immersed that I even did many more things with Computer Vision to answer all kinds of questions. But first, let's see the result:
● ***Monkey Illusion AIEdit description**
It's incredible what technology can provide us. And also how much development time it consumes. Yet, there are many more things I have left to do, just to mention a few:
● Detect Object● Tracking Object
● Heatmap position
● Analytics Pose
● Counter of Object
● Estimation of Age
● Counter of actions
● Count Steps
● Estimation of Weight
With 30 seconds of video, I realized that I had invested many hours of development. I became completely immersed in my desire to extract every possible benefit, and that is a very common mistake. It is quite frequent that when working on a data project, it is difficult to "let it go". One never finishes perfecting it, and that is one of the major problems we face.
While I was immersed, I forgot about the objective, counting the passes of the players in white jerseys (and perhaps finding the gorilla), and I started doing hundreds of other things.
However, getting back to the point, the algorithm fulfilled what was mentioned. It allows us to automatically count passes (and even other things) and answer the question that would
serve as a basis for making a decision (how many passes did the players in white jerseys make). By the way, having a clear question is already a big step that we have resolved in this case.
Ultimately, the video served to help the heads of the organization understand the difference between making an intuitive decision versus making a data-driven decision. But the remaining question is, how do we bring this to the entire organization?
Influence alone is not enough. We must also provide guidance and tools that enable the entire organization to operate in a data-driven manner.
In a simple way, we can describe the data journey like ingesting highly available data of various types, curate the data to ensure reliability, and create service for consumption and processing, from development of datamart, design of dashboards and development of Machine Learning models.
As part of this challenge, we will also encounter the importance of data federation and creating a data-driven culture. Additionally, having a good data catalog and automated data lineage is essential to fully exploit the obtained information.
Perhaps you, as data professionals, already know this and can discuss this topic. Although is important to generate knowledge throughout the organization about the data flow, especially because business people and stakeholders may be focused on other objectives and may not be aware of this process.
One of the most relevant stumbling blocks may be related to data exploitation and how to effectively outsource this task. According to Forrester, between 60 and 70 percent of data goes unused, so it is crucial to federate a data catalog that involves the business in metadata and genre association.
Commitment can be a key factor in data availability, accompanied by democratizing access and discovery, and having automated data lineage to enable autonomous data processing. Careful monitoring of this aspect is essential.
In summary, in order to clearly convey the main idea, we need to present a coherent data diagram.
