The Looking Glass: The Paradoxes of Data

Practical, magical, uncertain, and yet a tool for the most decisive

Dear Readers,

Many of you know that I have been building a company over the last 3 years. It’s long been a dream of mine to run this kind of 0→1 marathon.

Some folks express surprise when I tell them that my company is an analytics and data company.

But… aren’t you a designer? They ask. Didn’t you work on huge consumer products? Why would you jump to enterprise? And to data, of all things?

The simple answer is that my co-founder Chandra had a bold idea, and I believed we could make it happen.

The lucky answer is that along the way, I discovered I love this space.

Today’s theme is on Data, and its many curious facets.

Warmly,
~Julie

In this issue:

1. Data and design
2. The biggest misconception of data
3. What the most decisive leaders know
4. From the archives: Manifesto for the data-informed

For paid subscribers:

1. 5 rules of thumb for designing metrics
2. 5 things to say when your teammates use data to push back against you

Data and design

On the one hand, data and design feel like opposites:

Analysis versus aesthetics. Facts versus feelings. The cold hardness of numbers versus the warm fluidity of experience.

Product decisions are driven by metrics rather than people, complains one side.

Product decisions are made with ego rather than impact, charges the other.

But these differences are mere surface illusions.

In the deep heart of data and design burns the same flame: the search for the truth of a phenomenon.

What are people doing? Who are they? Where are they coming from? Why might they do what they do? How are they likely to behave in the future if we did X or Y or Z?

I have never met a deep practitioner of data or design who did not obsess over these questions.

Data is the contextual stage, the lessons learned, the structured hypotheses.
Design is the sword of action, the next experiment, the possible futures.

We diagnose with data. We treat with design.

The pebbles of past insights are what pave the path toward future invention.

The biggest misconception of data

The biggest misconception of data is that it provides certainty.

That somehow, a set of numbers is scientific, smart, and foolproof.

We think data is angular and rigorous, like a problem set or proof.

But data is not math; data is statistics — the messy world of aggregations, probabilities and percentages. Squishy as marshmallows. Leaky as cliches.

The reason data can never deliver certainty is because it can never escape two subjective human decisions:

1. What do we measure?
2. Over what time frame do we measure?

To answer the first, we must define what we believe to be meaningful.

To answer the second, we must define a time boundary for these beliefs.

Imagine Bob and Alice debating whether Company X’s investments in AI will pay off. Pretend they have magical abilities to measure absolutely anything about Company X. Will they eventually reach the same conclusion?

Well, can they first agree what “pay off” means? 2x revenue growth? 20% of customers use AI features daily? Inclusion of Company X in “Top AI companies” in Gartner reports?

Secondly, how long is the evaluation period? If Company X loses revenue in Years 1, 2 and 3, would we say No, the investment didn’t pay off? What if Company X suddenly adds 5x revenue in Year 4, now do we say Yes it did pay off?

This is assuming we have perfect data, already a tall assumption. (If we do not, we must add in yet another human question — how many samples would be enough to convince us?)

What should we care about? How much patience do we have? — there is no objective way to answer those questions.

Depending on your personal answer, the same facts will tell a completely different story.

What the most decisive leaders know

It’s easy to wish for data when serious questions comes up.

Why has our growth slowed over the past quarter? We need to figure out and fix it!

But if you think of data only as an investigative tool, you’ll be lagging behind and flying blind.

The most decisive leaders think of data not as a diagnostic prop but as an instrument of operational hygiene, similar to regularly scheduled check-ups to the doctor or dentist.

You don’t look at data only when something is wrong; you look at it regularly — every day or at least every week.

This kind of operational hygiene helps you in a few ways:

1. You pick up on issues early — for example, noticing slowing growth before it becomes a Stage 4 problem.
2. You create more accountability for yourself and others — did that feature launch actually move the needle? Is our current strategy actually delivering what we hoped?
3. You develop a better intuitive feel for your business — what’s normal, what’s not? Which further experiments are likely to resonate with customers, which aren’t?
4. You start to notice patterns of questions asked over and over again, which pushes you to better discover the levers of your business.

Ultimately, the grand prize of data hygiene is a nuanced, up-to-date model of the actionable levers of your business.

If something goes wrong, you don’t need to dispatch investigators to crack the case; you already know what happened, and you can immediately act.

Manifesto for the Data-Informed

Why is the promise of big data so much better than the reality?

Data has been extolled as the vanquisher of uncertainty, the harbinger of a robotic future, a necessity in every role from product management to engineering, sales to design.

Yet many of us who have tried to use data to inform decisions in organizations have experienced a different reality. One where we are constantly confused by how metrics are defined, bicker over how to interpret various analyses, and struggle to apply the insights into action.

It’s because building a data-informed culture is hard. Logging user actions, creating dashboards, running A/B tests, and shipping ML models — these are useful. But they are not the foundation of being data-informed.

We believe that being data-informed comes down to internalizing a set of values. These are simple, few, yet exceedingly powerful:

1. Conviction around a purpose rather than searching for meaning in numbers
2. Setting verifiable goals rather than vague aspirations
3. Company-wide familiarity with metrics rather than outsourcing to “data people”
4. Active testing of beliefs rather than seeking support for intuition
5. Accepting probabilities rather than thinking in absolutes

1. Conviction around a purpose rather than searching for meaning in numbers

The first step to being data-informed is understanding what data can’t do: give you a purpose.

Data does not substitute for a mission or a strategy. It cannot uncover a set of values. Metrics are merely proxies for what matters. “Increasing Metric X” is not a purpose; a true purpose must relate in some way to creating value for other humans. If a data-informed team feels at any point like they are optimizing metrics in a way that compromises their mission, they scrap that work.

Before you can collect data to help you track what matters, you need to define what actually matters. Information itself is not an evaluation criteria.

2. Setting verifiable goals rather than vague aspirations

How will you know if you are fulfilling your mission? You imagine outcomes that build towards the future you envision. Then you set goals to help you verify whether you are achieving those outcomes.

Vague goals that are difficult to verify aren’t useful and create confusion within a team. We want to make our users happier. We want to make the world safer. We want people to be more productive. These don’t give you an objective baseline to evaluate different strategies or tactics. And you won’t know whether you’ve achieved them.

Data-informed teams push for quantitative goals to the greatest extent possible because they’re the best way for a team to focus on creating impact. They make progress transparent, force accountability, and rally your team around a shared outcome.

Goal-setting is more art than science. All metrics are proxies and every set of verifiable goals will have shortcomings in what they fail to capture. As you learn those shortcomings, you will iteratively refine your methods of measurement and your targets. Setting goals is a skill — you have to practice it to get better.

Data-informed teams adopt verifiable goals to drive impact while acknowledging these imperfections. They recognize that goals serve the team in their march toward the mission, never the other way around.

3. Company-wide familiarity with metrics rather than outsourcing to “data people”

Everyone has to know the numbers. You cannot outsource a data-informed culture.

You can have the best team of analysts — but if you don’t know the numbers, then your decision-making will suffer.

Data-informed teams regularly meet to review key metrics. Why? Decisions are never made in these meetings, so why have them at all?

These meetings signal something important about the culture — that simply knowing and talking about the key metrics matters. Doing so builds a shared foundation among a team of what is going well or not well, and how they might best prioritize efforts.

Good data-informed teams develop leaders who can weave data into clear narratives. These teams cultivate a skilled understanding of which data patterns are important and which ones are not. They point out conflicts in data and which interpretation is more likely. Most importantly, they are upfront about what the data can and cannot say.

4. Active testing of beliefs rather than seeking support for intuition

We all have hypotheses about what our customers care about, what products will win, and what decisions are best. The more experienced we become, the more we trust — and even pride ourselves on — our intuition.

A well-honed intuition about which path to take is supremely valuable because we don’t have infinite time and resources to try every path.

The danger comes when your pride in your intuition leads you to close yourself off to evidence that you might be wrong. Instead of testing your intuition with data, you seek out data that confirms your intuition.

Openness to being proven wrong is insufficient. Data-informed teams actively seek out information that might disconfirm their assumptions. They value a high velocity of experimentation and setting hold-outs. Like scientists, they are constantly looking to test their hypotheses and validate their beliefs. They don’t view intuition versus data as a forced choice — they use data to refine their intuition over time.

Members of data-informed teams regularly ask each other: “What evidence would convince you that your intuition is wrong?” If the honest answer is “No piece of data would convince me,” then you have strayed off of the data-informed path.

5. Accepting probabilities rather than thinking in absolutes

Data will never give you certainty. Interpreting data means taking on many assumptions that are reasonable but not bulletproof. Even trusting a single piece of analysis means having faith that the events were logged correctly, that the metrics have been calculated without error, that everyone understands exactly what was logged and calculated, and that the interpretation is sound. You want to be 100% sure? Good luck.

People who are not truth-seekers constantly take advantage of this. The easiest way to discredit data is to demand perfection from it.

When data and intuition collide, some people always pick their intuition. They would rather be wrong betting on their intuition than wrong betting on data (this is rampant in the world of sports). That is a defeatist mindset. Your commitment to your mission requires that you make good decisions fast. Sometimes you will trust the data and you will be wrong. But if using data increases the likelihood of making the right call by 5–10%, those benefits will quickly compound.

This is the promise of data — to help us gain an edge in our quest to build wonderful things for other humans.

There are 2 more posts for paid subscribers.

1. 5 rules of thumb for designing metrics
2. 5 things to say when your teammates use data to push back against you

If you like my content and want to support writers like me, consider being a paid subscriber! Thank you thank you to those who have — you keep me caffeinated and motivated!

Read more