On my team at work, I seem to talk endlessly about the importance of “data correctness”¹.

We data practitioners typically focus on building things. We write code, model data, build dashboards, formulate regressions, and develop machine learning models. We are happy when code compiles and unhappy when it doesn’t.

When everything works, we move on. There’s always the next data set, the next dashboard, the next model.

During the building phase, we live in an ecosystem of our own tooling: our database editor, our dashboard, our Jupyter notebook. This is what I call “the left hand of data”. 

The left hand of data is not what our user sees. They see gross billings in Stripe, total delivered emails in Hubspot, or total contacts in Salesforce. They do not see, nor care for, the backend world we live in. The user lives in the frontend.

It is important for data practitioners to realize that, in some ways, the numbers we work with are “not real”. To us, data lives in a data store, such as MySQL or Redshift or parquet files in S3. We operate with the bare essentials: columns, rows and values. 

In the backend world of databases and data stores, we do not see everything that frontend users see. The data is not formatted, not adequately labeled, not always relevant, and not contextualized within the broader business domain. In the backend, we lose the essential understanding for knowing whether a given calculation makes sense or not.

If a marketing analyst sees within Hubspot “812 emails opened last week”, the answer is 812. If we calculate COUNTD(CASE WHEN event_type = ‘open’ THEN event_id END) in our database and get 796, we are wrong. What is real to the user is what they see on their screen, not what we see in the database.

We could be wrong for many reasons - perhaps Hubspot applies custom email open logic, perhaps our ingestion jobs failed to bring in all the data, or perhaps we are simply using the wrong table - but one thing for certain is that a failure to reconcile means we are wrong.

Peering into the frontend

Often, data practitioners do not have access to frontend tooling. They do not have access to the website or to Salesforce or to Google Analytics, and if they do, they do not have permission to view the data.

This means that data practitioners are foreclosed from knowing what the correct answer is.

How can we build data sets, dashboards, analyses and models if we do not even know whether the data is correct? We could be wildly off in record counts or totals and never know because we lack a frontend anchor against which to reconcile.

I often tell my team that “the left and right hands of data must meet”.

While we are naturally proficient in our own tooling, we also need to get into the frontend. We need to have the same permissions as the user and see the same things they are seeing. That is our source of truth. 

When we meet with the user, they’ll often tell us what the numbers mean, what they’re not sure of, what the data lifecycle is, and what the business context is. It is here, in the frontend, that the once skeletal data comes to life.

Reconciling to the frontend is time-consuming. You need to log in, get the right permissions, explore the user interface, build reports, meet with the user, export data, compare differences to the backend, and document discrepancies. I remind my team that nearly a quarter to half of their development time will likely be spent on such “data QA”. Any less tells me we are flying blind.

The alternative of not performing data QA, however, is more costly. It occurs after you spend weeks building a data set or report or model, only to present to your stakeholder and have them object: “Your data looks wrong.” Data is only correct insofar as the left and right hands of data meet, and it is better to verify such correctness before a presentation than during one.

We begin with a melody by Gary Provost:

This sentence has five words. Here are five more words. Five-word sentences are fine. But several together become monotonous. Listen to what is happening. The writing is getting boring. The sound of it drones. It’s like a stuck record. The ear demands some variety.

Now listen. I vary the sentence length, and I create music. Music. The writing sings. It has a pleasant rhythm, a lilt, a harmony. I use short sentences. And I use sentences of medium length. And sometimes, when I am certain the reader is rested, I will engage him with a sentence of considerable length, a sentence that burns with energy and builds with all the impetus of a crescendo, the roll of the drums, the crash of the cymbals–sounds that say listen to this, it is important.

So I write with a combination of short, medium, and long sentences. Create a sound that pleases the reader’s ear. Don’t just write words. Write music.

Back when I was in university, I had an economics professor who was a captivating lecturer. When he spoke, each word seemed to bend and undulate, carefully positioned as the crest or trough of a larger discourse. His pace, too, varied, with some words punctate and punchy, others leisurely and expansive. Then there were the pauses - long, exceptional pauses, meditations even - and then there would be no pauses at all.

What exactly made his lectures so compelling? There were many things of course: mastery of the subject, openness to student questions, clarity of communication. All of these contributed to his appeal, though instead I want to focus only on the manner in which he presented.

One thing about attending his lectures was that it felt like going to the theater. It was performative. There was movement, energy, gesticulation, audience interaction. We were not just to be informed, we were also to be entertained. And so I wondered: what makes a good performance? 

Almost always, a good performance succeeds in conveying emotion. Whereas the conveyance of information is sterile and mechanical, the conveyance of emotion is expressive and human. Whereas the machine aims to subdue variation, like a factory producing widgets, the human aims to amplify it.

And so we return to the introductory quote by Gary Provost. People do not like wholly short sentences, nor do they like wholly long sentences. They do not like overly complicated sentences, nor overly simple ones. They do not like only detail and specificity, nor do they like only abstraction and generality.

What they seek is variation, variability, the exploration of range between the short and the long, the fast and the slow, the high and the low, the paused and the rapid, the mobile and the static. Injecting variation along these axes made my professor’s lectures in economics - no, his presentations of economics - suddenly interesting.

What else, beyond tone and pace, could be varied, I thought?

Indeed, my professor varied the degree in which he interacted with his students, in how he moved around the room, in how he gesticulated, and in how he steered his facial expressions through curiosity and surprise and frustration and excitement.

He crossed his arms for a while, and then didn’t. He raised his voice, then whispered. He paced maddeningly back and forth, then stood perfectly still. He dove headlong into technical detail, then zoomed out to the big picture. He did everything and nothing, the essence of variation.

And so herein lies the trick I learned from my professor, which stuck with me far longer than any economic theory he taught. People appreciate the expressiveness of form, and the expressiveness of form derives from variation. Inject a little into your work and people will be captivated by it.

“Soft skills”, on the other hand, refer to the social interface through which employees interact with other people. Soft skills include clarity of communication, responsiveness, courtesy, and amiability. Hard skills and soft skills collectively form the basis for how we as individuals interact with the outside world.

There is also the question of how we manage ourselves. Are we diligent, organized, conscientious, detail-oriented and resourceful? Or do we drop the ball, hit roadblocks, and procrastinate? For lack of a better word, I often call these “professional skills”.

Individual contributors - the people responsible for delivering work product - must be proficient in hard, soft and professional skills in order to succeed on the job. Without these core skills, they will face friction when dealing with technology systems, other people, and their own work habits.

As an organization grows in size and employee count, it also needs people to coordinate and align all the individual contributors doing the work. These are managers. 

Managers exist to solve the problem of coordination failure, a problem whose cost, thanks to Metcalfe’s law, grows quadratically as the organization scales. They do this by securing alignment, both within their team, and between the team and the broader organization.

The axes of organizational alignment

As Patrick Lencioni explains in The Advantage, organizations which struggle with alignment are “unhealthy organizations”. Just as a person can become unhealthy from an internal lack of alignment and balance, so too can an organization. Symptoms of misalignment include team siloing, withholding information, undermining others, half-hearted agreement, and working around the system instead of working with it.

Lencioni writes: “Most organizations exploit only a fraction of the knowledge, experience, and intellectual capital that is available to them. Healthy ones tap into almost all of it.” This is because parts of a misaligned organization deliberately do not share information, pull in opposite directions, and ultimately produce drag on firmwide productivity.

Although Lencioni does not mention these explicitly, I believe there are four primary axes against which teams can become misaligned. They are:

1. Ability
2. Personality
3. Ideology
4. Incentives

If certain team members are unable to perform at the level of others, they will - despite their best intentions - contribute less, lower standards and throttle team productivity. When outperformers realize that average performance is below them, they intentionally pull back from their potential. Most compensation systems are ordinal rather than ratio-based, meaning that the degree of outperformance (ratio) is not rewarded as much as being “a top performer” (ordinal). Managers must ensure that all team members, when adjusted for level, contribute equally to avoid the appearance of certain individuals not pulling their weight.

Team members can also be misaligned in terms of personality. If some people are vocal while others tend to demur, this can create unspoken conflict within the team. The same goes for those who market themselves versus those who are self-effacing, those who are more analytical versus those who are more emotional, and so on. A manager must be careful to ensure that no single personality dominates, and that different personality styles respect one another.

Team members can become divided due to ideology. I define ideology as axiomatic differences in belief systems which are often, at a fundamental level, irreconcilable. I have seen software engineers dig trenches on engineering matters such as mono- versus multi-repo, static versus dynamically typed languages, and manual versus self-documenting code. When faced with ideological differences, managers should generally appeal to the majority, then ask dissenters to disagree and commit.

Finally there are incentives. If team members are not incentivized - for example, they are not rewarded proportionately for quality work - then they will become demotivated and lower their quality of work. The most obvious example here is compensation, but there are other ways to incentivize staff, including work-life balance, remote flexibility, choice of projects, and choice of manager. Managers should ensure that incentives align between team members and the broader organization.

Across all axes - ability, personality, ideology, and incentives - managers must aim to reduce internal variance. If variance grows, it will create a lopsided and unbalanced team, eventually bifurcating into separate “sides”: our way versus their way. To reduce variance, managers must listen to their team, understanding each individual’s abilities, personalities, ideologies and incentives, ultimately creating a work environment that integrates all.

And if that was not enough, there is more. The firm as a whole has its own average ability, personality, ideology and incentives. An individual team cannot fall out of sync with the greater whole. Therefore, a manager must create alignment not just within the team, but also between the team and the wider organization.

Managerial skills for future managers

Managers should be born out of individual contributors. That is to say, they have excelled in the soft skills of communicating upward and laterally throughout the organization, the hard skills of successfully operating technology and systems, and finally the professional skills of consistently delivering valuable work product. 

These are table stakes. Without fluency in these foundational skill sets, managers inevitably drown in the complexities of people, projects and systems. Many a corporate blunder has been committed by promoting individuals who are not proficient in all three.

But as leaders of a team, managers must acquire additional skills, and these constitute “managerial skills”. Managers must inspire a team, align a team both internally and externally, deliver consistently on a portfolio of projects, and build relationships with various stakeholders throughout the firm.

If I were to distill it into a checklist, it would look as follows:

(1) Managers pitch a compelling vision

Managers are able to “find their own work”, observing the product and organization as a whole, then determining where there are opportunities for growth.

Just as a consultant or investment banker constantly pitches work to potential clients, managers must too pitch potential projects to stakeholders. The manager solicits ideas on what would be valuable to the organization, then generates excitement over what can be built.

Managers always need a handle on the vision for each project. They formalize this vision into a clear roadmap with monthly, quarterly or annual milestones to demonstrate progress. When people get stuck on the day-to-day details, managers keep the team and stakeholders aligned on overall direction and the “why” of it all.

(2) Managers reliably execute on the visions they pitch

While salesmanship is certainly part of the manager’s toolkit, the pitch must ultimately convert into results, otherwise the manager will develop a reputation for making empty promises.

Here, in the execution stage of a project, are where a manager’s individual contributor skills shine. Managers need to be able to closely audit the work produced by their team, and in some cases, serve as a backstop for the actual work to be delivered. 

In addition, managers will need project management skills. As they are no longer responsible for just one or two projects, but instead a large portfolio of projects, they will need to stay organized and apprised of key information and dates relating to each project.

(3) Managers align disparate groups of stakeholders

Within any organization, there are various audiences to consider, each with their own level of interest and involvement in the projects you work on. These will primarily be:

• One’s direct manager (and their manager)
• One’s direct reports
• Clients and internal stakeholders
• Collaborators and partners
• The broader organization

Oftentimes, people will focus on making just their manager happy, or just their stakeholders happy. Such a narrow focus on any one stakeholder often comes at the expense of other stakeholders. The real skill is to make all stakeholders happy at the same time.

How can this be achieved? First and foremost, it is achieved by doing good work. Delivering good work will earn the respect of key stakeholders, and over time, this will materialize as a good reputation.

Second, it is achieved by not biting off more than you can chew. If you make promises to clients or your manager which cannot be reasonably delivered by your team, then once again you will have traded off satisfying one set of stakeholders at the expense of another. 

Third, it is by being overly communicative with all stakeholders on plans, timelines, status updates and changes. I often tell my team that the litmus test of success is that “nobody should ever be surprised”. If you make a commitment to a client that someone on your team is surprised to hear about, then that miscommunication will materialize into misalignment as to what was promised versus what can be delivered.

“Nobody should ever be surprised” extends beyond simple work plans. If good work is delivered and the client is happy, this feedback should be delivered upward, inward and outward. If your manager or the broader organization would be surprised to hear about all the good work being done by your team, that is likely an indicator that you were not communicating enough.

(4) Managers raise the bar for the team

When a team is small, stakeholders are used to working with the manager directly and become accustomed to the quality of work delivered. As a team grows, the manager is responsible for delivering the same quality of work, even if they themselves are not the one doing it.

Therefore, a manager must invest heavily in the training of team members. They must shadow and observe how work is done. They must audit and quality control final deliverables. They must coach and give feedback when things could be improved. They must formalize guidelines and host group trainings to ensure that everyone doing work has a clear sense of the standards expected to qualify as “good work”.

Most importantly, managers must have a clear, opinionated sense of better and worse ways of doing things. They constantly wonder how things could be better. Collectively, these become operating principles for the team, the standards against which team members are evaluated when work is performed.

Wrapping up

Being an effective manager requires a broad array of skills. Managers must not only be able to deliver good work - as indicated by proficiency in hard, soft and professional skills - but they must also be able to manage and coach a team of people responsible for doing the same.

These “managerial skills” come down to (1) understanding and pitching the “why” of work being performed, (2) consistently delivering good work, (3) communicating and aligning with all types of stakeholders, and (4) raising the standard of work within one’s team.

When done well, team management serves as a reliable “factory” by which compelling visions are consistently delivered by motivated and aligned teams.

Several years ago I took a break from the corporate world to work on a startup idea. In the beginning, I was intoxicated by the idea, molding and permutating and tinkering with it every which way. I ideated on its design, its architecture, its business model, its market opportunity. I incubated the idea from dawn until dusk, like a mother hen enveloping her infants under the warmth of her feathers. I coded up the application, built its landing page, and marketed it on social media. Each day whirled by, a tornado of excitement and activity.

Over the following months I continued to make rapid progress. What began as an initial prototype transformed into a robust, living application with now a handful of actual users. I hired a few part-time engineers to continue developing the application while I transitioned to business development. Although we were a small team of 4, we had daily standups, product meetings, and code reviews, just like real companies.

By the 6-month mark, I was fully immersed in sales and marketing. I created content, wrote copy, prospected new leads, drafted emails, and made cold calls. Unlike the world of software development, business development was foreign to me. I learned as I went, but I was not nearly as good at it. I did not see results. The uncertainty of the future increasingly weighed on the present. The days slowed down.

I trudged forward for several more months until I realized our condition was terminal. I explored other options - co-founders, advisors, external funding - but my lack of progress in business development chastened me. The energy and zeal with which I initially applied myself had nearly totally vanished. A once relentless, pulsating drive withered into a paralyzing lethargy. By the 10-month mark, I laid off all the developers and shut down the application. 

After the saga was complete, I was finally able to reflect on the experience. Of course I reflected on what worked and what didn’t: the idea, the product, the market, myself. But I also found it curious how my energy levels, and by extension the purpose with which I lived my life, oscillated so immensely during that period. 

What exactly had changed to cause such profound differences in my energy, my mood, my sense of purpose?

Materially speaking, my day-to-day existence was exactly the same at the end of those 10 months as at the beginning. I awoke in the same bedroom, worked from the same laptop, commuted to the same workspace. And yet me at the beginning and me at the end were two markedly different people. 

It must have been the case that it was all in my head. The story I told myself, which fueled my days both in the frenzied beginning and at despondent end, was all in my head.

The power of narrative

Every story has a beginning, a middle and an end. So too do our lives, and it is likely for this reason that we resonate so deeply with the structure of narrative. Every work project, job, hobby, relationship is in some sense a story.

The beginning of a story is full of hope, promise, and opportunity. We have the opening curtain, the launch party, the project kickoff. Characters spring to life, brimming with optimism and ambition. We learn their history, their origins, their aspirations. The beginning tells us where they came from and where they’re going. 

Then comes the middle. The middle is where dreams meet reality. It is full of the nitty-gritty details, the operations and the execution, the challenges and the hardships. People grow up. Things happen. Time moves on.

Finally there is the end. We have the closing ceremony, the final presentation, the project recap. We reflect on all that has transpired and, like any good story, what the lesson is. We walk away with winnings and rewards, or if none, experienced gained.

Periods in our life which do not follow the structure of narrative often do not feel right. Without a proper beginning, we lack context about why we’re here or where we’re going. We meander aimlessly, periodically retracing our steps, wondering if there’s some greater purpose we may have missed. If we spend too long in the middle, we become trapped in an eternal present, afflicted with a myopia condemning each day to be just like the last. Without a proper end, we never achieve closure, finality, and thus the distance required to extract the “why” of it all.

It was after my experience with the startup that I began to appreciate the importance of personal narrative. As I looked at the various areas of my life, I began to wonder what part of the story I was in. 

If I was at the beginning, I considered what my opening ceremony would be. If the middle, I debated whether I was still on the same path I had set out on, or instead if I had wandered away, given up, moved on. If at the end, I reflected on how I would tie up loose ends, what my closing ceremony would be, and what lessons I would take away.

I learned that the structure of narrative was not something that existed “out there”; indeed, it was all in my head. I could invent new beginnings as I pleased. A new year, a new quarter, a new work project. Each would become its own chapter, a story within a story, an episode with its own beginning, middle and end.

“Why does the total active user count seem so low?”, he asked.

I pulled up the dashboard and studied the number. Total active users was indeed too low. Obviously, undeniably, irredeemably low. Had I even looked at it?

Looking

Looking is a hard thing to do. Looking does not produce anything. It merely observes, reviews, and scrutinizes what has already been produced. It is an exercise in quality control. It does not create; it critiques.

Doing, on the other hand, is highly valued. There is a clear, tangible artifact. Excel models, Powerpoint presentations, Confluence specs, email memos, Python code, Tableau dashboards. There is no question about what you did, how you spent your time, or what you delivered.

Why spend 1 hour reviewing when you could instead spend 1 hour building?

So we build. We are incessant, frenetic builders, though we do not always look at what we built. How else can one explain the typo in the subject line of an email? Or memos with such discursive writing that readers are forced to read and re-read, as if deciphering hieroglyphics? Or inconsistently formatted presentations, the evidence of rushed copy-and-paste jobs between various templates? Or finally, as in my case above, glaring dashboard miscalculations that send viewers reeling before they even begin? 

Were these all dumb mistakes, or more perhaps aptly, hasty mistakes? One must inquire, in a very physical, tangible sense: did we even look?

Slowing down

As the world becomes increasingly connected, it too feels like everything is speeding up. Information arrives sooner and more frequently, we communicate non-stop with family and friends and colleagues, we feel we must always be moving, doing, producing.

All this rushing implicitly prioritizes quantity over quality.

We feel impelled to get things out the door - faster, more - so that we can move onto the next thing. We are constantly distracted by the next news article, the next phone notification, the next opportunity. We are in such a rush to reach the finish line that we forget whether we’re still holding the baton.

But what about this thing in front of us? What if we stared at it, defiantly refusing to shift our attention to the next thing?

We would reflect on it. We would wonder if it looks right, if it feels right. We would notice errors and inconsistencies and words that did not quite fit. We would probe it, reshape it, refine it. We would revisit it over and over again until it was complete. It would be a meditation on quality. 

What good is what we produce if it is not what we ultimately desire? We do not merely want “stuff”, but rather “good stuff”. This necessitates deep, effortful craftsmanship. We should not just “get things done”, but instead, “get quality things done”. The indelible mark of experience is having an intuition for what is good and what is not. It is exercising good judgment and having good taste.

To do this, we must slow down. Velocity is the mother of quantity, but patience is the mother of quality. As the old Zen teachings say: “If you are busy, meditate for an hour. If you are very busy, meditate for two hours.”

Design, build and test

Looking, while valuable in its own right, is not enough to produce great work. You must of course also produce.

In many disciplines, the roles of builder and reviewer are distinct, as if by necessity the reviewer must step back from the work, create distance, and evaluate it with the impartial eyes of an outside observer. When performed by the same person, there tend to be distinct phases of development.

When writing software, we typically begin with draft code before refactoring it into more modular, more parsimonious code. When building software systems, we typically separate software development from quality assurance (QA) and user acceptance testing (UAT). 

In civil engineering and manufacturing, we observe the same pattern: there is a stage of design, a stage of production, and finally a stage of review, collectively called design-build-test. Even in writing, many writers employ a similar methodology, my favorite of which is the madman, architect, carpenter, and judge.

As I write I flow through the different personas: he who creates, he who structures, he who fills in, and finally he who reviews. To only create is to produce without attention to detail; to only judge is to have nothing of substance to review.

All stages are necessary to produce great work. The madman will create wildly, generatively, uncontrollably. But without review, his studio of detours and missteps will be laid bare.

The judge must step in and revise. He pauses, takes a deep breath, and begins his work.

Perseveration is a clinical term referring to a wide range of functionless behaviors, such as repetitive speech, action or thought, which persist long after the original situation has passed. More colloquially, we might think of it as “getting stuck” or “doing the same thing over and over again to no effect.”

Perseveration is not quite the same thing as dwelling on something, as you can dwell on something that is nonetheless useful. It is not quite obsessing over something - although obsessive compulsive disorder is certainly comorbid with perseveration - as obsession does not capture the repetitive nature of perseveration. Thrashing comes close, though its violent, desperate nature overstates the droning, useless banality of perseveration.

Back in 2018, having recently quit my job to move abroad and start a business, I was living in Poland. After about 6 months of trial and error, I ended up mostly with error. I soon jettisoned my original business idea and searched anew. I recall the feeling of grasping around for something - anything - that would rescue me from the abyss of failure. I thought and thought and thought. My mind spun like the tires of a car stuck in mud.

During this time I felt a compulsion to be busy - furiously scribbling down notes, ruminating before bed - because action appeared to signify forward motion. I didn’t realize that my incessant thinking and incessant doing crowded out a more sober, more reflective, and ultimately more productive mindset. I had become the dreaded “busy loop” of programming: a while-loop with no break condition. I was perseverating.

2 months later, I packed my bags and returned home.

Once I became aware of my own perseveration, I began to notice it in other places - especially at work. I noticed coworkers who wrapped themselves in a suit of meetings in order to feel busy and productive. I noticed peers who, anxious about an upcoming deadline, upbraided and belittled coworkers, fulfilling the old trope that “the beatings will continue until morale improves”. I noticed leaders who changed strategic direction so frequently their clockwise rotations landed them precisely where they started. Were these too all forms of perseveration?

On the nature of perseveration

Perseveration appears to be an extreme case of otherwise completely routine, functional behavior. Throughout our lives, we exercise control over our environment in order to achieve our aims. We observe information from our environment, analyze it, form a decision, act, and ideally observe the outcome. These actions should, over time, bring us closer to our goals.

This “human control loop” is exceedingly simple. It is composed of:

1. An action
2. A result
3. An inference that the action caused the result

If I pick up a glass of water and drink the water, after which I feel sated, then I will infer that my drinking of the water caused my satiety. If I drop a ball from the air, then observe it hit the floor, I will infer that my dropping of the ball caused the ball to reach the floor.

All 3 aspects - the action, the result and the inference - are necessary to have a complete feedback loop. If I don’t observe my own action, for example the ball magically appears on the floor, then I will not be able to infer that it was my action which caused the ball to be there. If I don’t observe the result - there is no ball on the floor - then my action of dropping it certainly did not place it there. Finally, the causal inference itself can be suspect: if I drop a ball from the air, and then at some point later I observe a ball on the floor, it is not necessarily the case that I caused the ball to be on the floor.

Perseveration occurs when this feedback loop is broken or unclear. If we are not self-aware of our own behavior, or if we are not observant enough of outcomes, or finally if we do not reflect on the causal relationship between our behaviors and the subsequent outcomes, then we will be stuck in a cycle of action which does not produce its intended effect.

Often, we are so focused on feeling busy, on speaking, on working, on doing, that we do not lift our eyes to observe the environment around us. Other times, we are so focused on other things and other people that we do not observe our own behavior. And even if we observe both our actions and subsequent outcomes, it takes a healthy dose of reflection to infer which of our actions are causing which of the outcomes we observe.

On the economy of movement

From the outside, this type of reflection looks like a whole lot of nothing. Observing, inferring, thinking. But its plodding, methodical nature belies the immense power it wields. I refer to this as “the economy of movement”.

The economy of movement is the opposite of perseveration. It is the strategic deployment of energy only insofar as it produces desirable outcomes.

There is something deeply fundamental about the conservation of energy. Of course, it is the first law of thermodynamics. Given this physical law, it also manifests in biology as optimal foraging theory: organisms which do not allocate their energy productively are outcompeted by those which do.

The economy of movement looks like many things that we as a society regard highly. It looks like concision and parsimony, elegance and beauty, effortlessness and finesse, optimality and efficiency, or - my personal favorite - sprezzatura. Roger Federer playing tennis exemplifies the economy of movement.

The economy of movement does not reward blind action, as action without consideration of results may be wasteful. In the economy of movement, meditating on the relationship between cause and effect is just as important, if not more, than action itself.

The economy of movement looks like nothing instead of something. Less instead of more. Patience instead of impatience. Listening instead of speaking. It looks like the meticulous conservation of energy rather than the indulgent expenditure of it.

There is a time and place for action, and when it is necessary, it ought to be vigorous, effortful and intense. But action is not the only tool in our toolkit. Inaction is too. Because what appears to be inaction on the outside may instead be careful reflection on the inside.

Why is culture so hard to preserve as an organization scales?

Culture is the set of beliefs and practices that belong to a particular group of people - in this case, the group of people who work at the firm. The initial culture is woven by the founders, stitched together by what they say and what they do.

Among small groups of people, say up to 10, a single, coherent culture can persist without any explicit guidance. However, beyond such a point, implicit culture does not propagate well. Each additional hire can no longer meet individually with each of the original team members to emulate their way of working. Explicit guidance becomes necessary to overcome the constraints of space and time. This is invariably why all medium to large organizations codify their culture into mission and values statements.

The moment we codify our culture, however, we run into the general problem[0] that an implicit culture continues to exist in parallel, and the two can fall out of sync with one another. Even if a company claims to value “honesty and integrity”, it is not a foregone conclusion that every individual within the company will identify with or actualize it. 

Because people follow company values to varying degrees, the degree of fidelity between explicit values and implicit behavior is variable throughout the firm. A high degree of fidelity means that everyone is bought into the firm’s mission and values; a low degree means instead that people mock them.

How exactly do these two cultures fall out of sync? Were the explicit and implicit ever aligned, even at the very beginning? If so, when did they drift apart, and why? Most importantly, how do we keep them aligned so that implicit practices conform to explicit values, enabling the organization to broadly deliver on its stated objectives?

Codifying work standards into an explicit mission and values statements

As people perform work, they inevitably discover ways of working which are more effective for themselves and ways of working which are less effective. For example, when I work, I try to:

• Be responsive and overcommunicate
• Focus on outcomes, not implementation
• Focus on velocity, not perfection

To me, this is simply how I work. I don’t need to tell anyone this, and thus it is an implicit culture - a “culture of one”. 

Once I codify these practices into values or operating principles, they become explicit. One would hope that I, of all people, embody these values, but that is not necessarily the case. Leaders can practice “vacuous leadership”: saying one thing while doing another. In this case the fidelity between explicit culture and implicit behavior begins to degrade at the very inception of the organization.

Even if all original team members abide by their own values, there is nevertheless the problem that future team members will not, bringing us to the question of how such adherence can be widely adopted throughout the firm.

The alignment of explicit and implicit culture

In an ideal world, company values are general enough to be unobjectionable to the majority of employees, yet specific enough to delineate between better and worse behavior. For example, Amazon’s well-known value of “customer obsession” is unobjectionable insofar as it is not obviously wrong or vague, yet it is still able to trade off against other stakeholders like “shareholder obsession” or “media obsession”. In the world of Amazon, the most important stakeholder is unambiguously the customer.

Given a generally unobjectionable set of corporate values, it then follows that the majority of employees should adhere to them. In practice, this often does not occur. It may not occur because of self-interest, disagreement or, most commonly, inattention. Whatever the reason, company leaders should be sensitive to such a “normalization of deviance”, a tacit endorsement that people are free to do as they please and ignore company values.

In an informational sense, the normalization of deviance can be interpreted as “signal loss”. When you start out with 100% signal at the source, it can only deteriorate over time - to 98% to 95% to 92% and so on - until, after several hiring cycles, the resulting culture bears no resemblance to the original one. 

At work, I often request that my team provide a brief, written weekly update, which can be anywhere from 1-5 bullets. This serves as an operating principle for my team - a best practice if you will - because we invariably need to report upward on what we are doing each week. In crafting this, I solicited my team for a format which balanced both information value and convenience. We settled on just a few bullet points. Given majority agreement then, I expect it to be completed each week, the lack of which would constitute the normalization of deviance.

Promoting culture keepers into leadership positions

I, on my own, can encourage adherence to our corporate or team values, but this does not scale to large numbers of people. As the organization grows, this responsibility - like any other - must be delegated.

To whom should we delegate this? If we delegate it to people who are excellent at their jobs - rainmakers, marketing whizzes, software architects - but who are not necessarily good at following the company values, then what inevitably occurs is that the people they manage will also not adhere to the company values. If your goal is to preserve the company culture as the company expands, leaders must not only be good at their jobs; they must also adhere to and encourage the company values among those they work with. 

Just as there is a “Peter Principle” stating that people are promoted up to their level of incompetence, I propose there is similarly a “Repeater Principle” stating that those promoted into positions of leadership who are not “repeater nodes” will lead to the attenuation of company values. 

By this definition, being a leader is equivalent to exemplifying, instantiating, embodying and living the firm’s culture code. Leaders broadcast the culture, develop standards to further enrich it, adhere to it and encourage others to do so as well. 

Netflix famously popularized the “keeper test” for employees, which asked managers whether they would fight to keep an employee who was considering leaving the firm; if not, the manager should preemptively let the employee go. Perhaps an extension of this is the “cultural keeper test”: if new managers would not uphold the company culture in their new leadership positions, then they should not be promoted into leadership positions.

If instilling company culture into new hires is not prioritized by current and future leaders at the firm, then no one will, so new recruits are free to “bring their own”. That works up to a point - integrating new ideas is always useful, as we’ll see below - but too much cultural heterogeneity can also lead to fiefdoms, information silos, paralysis, and outright conflict between teams.

Cultural evolution within integrative versus retaliatory regimes

If leaders are responsible for exemplifying the company’s values and instilling it broadly among employees, does this effectively make leaders nothing more than loyalists, yes men and women, uncritical spokespeople and enforcers of company values?

Obviously such blind loyalty and subsequent coercion is undesirable. There is no guarantee the original culture set by the founders is optimal, and even if it was at company inception, there is no guarantee it will continue to be as the company scales. A company culture - that is, generally accepted “better ways of doing things” - must constantly adapt to new information and new environmental pressures.

If a company has less than optimal company values and practices, what are leaders within the organization to do?

Here, it really depends on what type of “regime” the company operates as: (1) a retaliatory regime or (2) an integrative regime. A retaliatory regime punishes public disagreement, rewarding those who toe the party line, while an integrative regime rewards public debate and ultimately consensus.[1]

Retaliatory regimes move quickly and dictatorially while integrative regimes move slowly and democratically. A retaliatory regime is extremely low-cost in time and energy, but suffers in the long run from private dissent, doubt, sabotage, and obscured information flows. An integrative regime is far more costly upfront in time and energy, but benefits in the long run from rewarding those who surface new information and those who debate it.

Both regimes feature failure modes: for the retaliatory, it is tyranny, and for the integrative, paralysis. There is no “right” regime, which is why we observe both in nature. A regime persists to the extent it is adaptive in a given environment.

When a company promotes its values to all of its employees, it must decide whether to pursue a retaliatory or an integrative approach. The retaliatory approach declares: “This is good. You must follow it.” The integrative one inquires: “Is this good? Should we follow it?”

Ultimately, the regime that any organization adopts starts from the top. If senior leadership punishes those who question them and rewards only those who agree, then retaliation and coercion will cascade down throughout the organization. On the other hand, if senior leadership rewards those who question them and those who surface new information, then the firm’s corporate values will evolve to incorporate these additional perspectives.

An integrative regime, which adapts to new information, is always more desirable, but in practice many leaders cannot or choose not to pay the cost.

All leaders, however, recognize the importance of maintaining cultural consistency throughout their organization. Codifying this culture and promoting those who embody it helps ensure the culture is preserved as the organization grows.

--

[0] Any explicit encoding will always produce a separation between “the encoding of the thing” and “the thing itself”. See: esoteric vs. exoteric, internal vs. external, reality vs. superficiality, map vs. the territory, Plato’s Cave.

[1] Here, I borrow heavily from Timur Kuran’s Private Truths, Public Lies, although he never uses these terms explicitly.

One skill I always try to practice on the job is to walk through walls.

Walls, in the metaphorical sense, could be explicit prohibitions, such as “thou shalt not do that”. They could be organizational walls, like “that team or committee is responsible, so they will take care of it.” Or technical walls, such as “we don’t know how to do that” or “we don’t have access to that system”. Or economic ones, like “that would be cost-prohibitive to do” or “we couldn’t possibly monetize that”. 

Especially at large organizations, walls are everywhere. Walls tell you what you cannot do. The wall is an obstacle, an injunction, a justification for inaction. The wall says no.

But why? Why are walls so prevalent?

On the nature of walls

Walls originate from the separation of responsibilities. As a firm grows, the complexity and scale of work-to-be-done becomes too much for one person, or a handful of people, to handle. The firm must federate responsibilities to different departments - sales, marketing, finance, product, etc. - in order to master the complexity of any one domain. Specialization is inevitable.

But specialization also creates boundaries - distinctions between “my work” and “your work”, or alternatively “my territory” and “your territory”. Often, we are advised to uncritically defer to the expertise of other teams, to not waste their time, to not ask questions. Rather than approaching their territory and getting our hands dirty, we keep a safe distance and sit on them instead.

While the separation of responsibility is designed to enable efficiency, it often produces an unintended side effect: the abdication of responsibility. When another department is responsible for the work, it is all too easy to say: “We are waiting on them.” Even if they are slow, busy, overworked, unresponsive, or incompetent. 

The most convenient solution to not solving your problems is to instead throw them over the wall and say “we are waiting on them.” When everybody participates in this game of hot potato, institutional paralysis ensues. Everyone waits, and nobody does the work.

I see this mentality often in large organizations. It’s as if, upon seeing a wall, people suddenly turn around and walk the opposite direction. The wall serves as a convenient excuse to look no further, to do no more. It becomes a sort of demilitarized zone, an area devoid of people and also, of progress. It is here then, at this junction, that the wall becomes one of the most interesting parts of the firm.

Wall inquiry as a state of mind

When you believe in walls, you default to asking: “In what ways can we not do it?”. When you don’t believe in them, you instead ask: “In what ways can we do it?” A convenient fiction then is to believe that the walls are not there. 

Once you stop believing in walls, when you see one it can be a little surprising. “What is this thing?” you ask. It is like walking up to a mirror and seeing your reflection. “Why is this here?” “Does it follow me if I move like this?” “Why is its right hand matched to my left?” You will inquire about this wall. You will not smash it, ignore it, change it - but you will certainly be curious about it.

Sometimes, a wall has a very clear KEEP OUT sign on it. That seems important; like Chesteron’s fence, it is probably there for good reason. But other times a wall is less forthcoming. It is there because so-and-so (who now switched teams) said we couldn’t 3 years ago, and no, we cannot find any documentation about this anywhere. Should we simply nod our heads and say: “Well so-and-so 3 years ago said we couldn’t?” Should we do nothing? Should we abdicate?

This wall, old and dilapidated, has few people come by anymore. The footpaths leading up to it have long since disappeared, evidence of those politely steered away earlier in their journey. 

If you walk up to the wall, you may find little details others long ago had overlooked. Did you see, for example, that the wall has fallen apart on the left side over there, and you can peer through it? Did you notice the friendly old lady who passes by each Sunday and invites you over for tea? Did you observe, far away near the end of the wall, others too are inspecting the wall, wondering why it is there?

Carefully, you creep through a narrow slit in a part that crumbled years ago. You walk forward a few paces, and already things feel different. The verdure seems vibrant and untrampled, there are birds you’ve never seen before, and in the distance there appear to be vehicles both smaller and older than those you’re used to. As you continue, you eventually come across people - welcoming people who willingly show you around - though they speak a different language, move slowly, and have an unusual gait. Things are certainly different over here.

You see some strange things too, like the crossing guard in the intersection who is manually directing traffic. Don’t they have technology here? You also see people going to work in overalls and boots. Don’t they value professionalism? It is concerning, but for the moment, you reserve judgment.

You had initially planned to stay only a few days, but days become weeks, and weeks become months. You make a few friends (including the tea lady from Sunday), and make a hardy attempt to learn the language. To really immerse yourself, you dress the same way, you speak the same way, you throw yourself into the things they seem to enjoy doing. You do not agree with everything they do over here, but darn it, they do have a handful of good ideas. In due time, they accept you, not as a foreigner passing through faraway lands, but as “one of us.”

Months later you return back to your side of the wall. Upon passing the wall you wonder again just what exactly its purpose is. Why obstruct us from what is on the other side? Is it for our own good, or for theirs? Who put it up here to begin with? Perhaps only its creator knew what was on the other side, and using the wall, they could play gatekeeper, controlling who could pass through and who could not.

Back on your side of the wall, you feel enlivened. There are a few neat tricks you discovered on the other side, and you share these learnings with your comrades. Sometimes, when your team has questions, you now answer, “I know who can help you with that” or “I know someone who knows”. 

You realize that sometimes when you did not have access to things before, it is not because you were not allowed to, but simply because you had never asked. You see that sometimes certain problems that were particularly thorny before were not truly intractable, but whose solution lay just on the other side of the wall. You see that sometimes the only reason a wall is still there is because no one ever bothered to take it down.

The parable concludes

On the other side of a wall, there is almost always useful information. Information is the lifeblood of any organization, and access to it is its currency. Information must flow quickly and liberally, otherwise the arteries atrophy and activity ceases. While walls enable specialization, they too can invite ossification.

Just as Google unlocked the world’s information with the search index, it is equally important to unlock an organization’s information trapped behind departmental walls. Information allows us to know who to go to for help, know what is permitted and not permitted, and know what is a priority versus what isn’t.

Walking through walls does not take technical skill or leadership skill. Most often it simply requires being open and inquisitive. Some things I try to practice are empathizing with people’s problems and needs, immersing oneself in the way they do things, being non-judgmental of their work, being useful, and being curious.

In the past, I used to speak to business teams using engineering terms because “that’s how things actually work”, I thought (and perhaps less admirably, because I wanted to show how smart I was). A few confused meetings later, I learned my lesson. I would use business speak when talking about my work, I would try to learn why the business stuff was just as important as the engineering stuff, and I would attempt to genuinely be interested during their meetings. 

The more I approached the wall, the more they did too, and the more productive we as a group became. Had I never crossed over to the other side, I’m sure they would have deflected me away, not because they are spiteful, but because it is simply more convenient to find someone else who makes an effort “to get it”.

Most departments do not have iron walls. Most people are not intentionally trying to shield you from their work. Often, they are simply too busy to roll out the red carpet for you, so you must find your own path in. There are even people who will gladly share information and tell you all about their work, though frequently they are not senior enough to formalize the entrypoint. They are your guides. They, the ones closest to the details and mechanics of how things actually work, hold the keys to the organization. When you find a wall, look for the shepherd.

Once you can walk through walls, it will look like a superpower. You can see things that others cannot and avert otherwise paralyzing obstacles. Of course there is no real trick here, no superpower to disclose. You come with keys. And once you collect enough of them, you can lead others to places they haven’t been before - even if they’re just a hallway away.

When it comes to individual productivity, I’ve generally found Andy Grove’s maxim to be relevant: “either they can’t, or they won’t”. Issues related to productivity generally stem from (a) a lack of competence or (b) a lack of motivation.

When someone first joins a company, they do not yet have enough familiarity with the job, the organization, the firm and the industry to be productive on day 1. And while most new hires do begin a new job with a certain optimism and eagerness, that goodwill can quickly erode by week 1 or 2 after a few mishaps during onboarding. 

Therefore, I see the first 90 days for a new hire as particularly important in giving them all the context they need to be productive, as well as ensuring they have a positive experience with the company so that they remain motivated to perform.

Competence

The first screen of competence is during the hiring process. Ideally you design the interview process in such a way as to accurately assess someone’s raw abilities.

But even after finding and hiring qualified people in general, you must train them to be competent in this particular role. Common frictions when adjusting to a new role include:

• Not being proficient in day-to-day tools and tech stack
• Not being permissioned to access certain tools, information or data
• Not knowing what work is valuable to the organization, and thus how work should be prioritized
• Not knowing who or where to go to for information
• Not knowing the industry, product lines, market participants, distribution models, etc.
• Not knowing what is “expected”, including communication norms, social norms, or work standard norms
• Not knowing what is prohibited, typically related to security or compliance

The goal of onboarding then is to take someone’s raw abilities and “fill in the gaps” with the knowledge and skill sets relevant to this particular role.

Many managers do not perform training. They expect new hires to pick it up on their own without any work on the manager’s part. These managers do not assemble relevant materials or teach on relevant tooling, and perhaps most importantly, they do not set aside time for training. If you do not explicitly allocate time for training, then the only alternative is time for working.

The work that untrained employees produce is simply substandard work. Like in an apprenticeship, the entire point of training is to produce “work to standard”. Without training, there is no concept of what good work looks like and what bad work looks like. Instead, untrained employees simply produce “stuff”, as producing something is better than nothing. But what is delivered is often hard to follow, ignores standard conventions, prioritizes the wrong things, looks bad, runs bad, and solves the problem the hard way instead of the easy way. Whenever I see a lot of subpar work in an organization, I end up instead noticing a systematic lack of training.

The longer people go untrained, the more likely they are to remain untrained. That’s because the longer you’re at a company, the more you’re expected to know. Asking a “dumb question” after 6 months or a year would expose one’s lack of knowledge, so instead people opt to stay quiet, resigned to a state of ignorance. But rather than wondering: “Why do I not know this?”, one must instead ask: “Why was I not trained on this?”

By the end of training, new hires should have a good sense of what work is valuable, who to ask when they have questions, what kind of communication is expected, how to use the tools at their disposal, and how to navigate the organizational landscape. They know what the standard of work quality is because they’ve seen it all around them, and further, they’ve been trained on how to build things the same way. 

For new hires I often put together a mini-orientation to familiarize them with our industry and organization, the outline of which roughly follows:

• Industry overview
  • History, evolution and innovations
  • Market structure and market participants
  • Economics
  • Product lines
  • Regulatory regime
• Company overview
  • History
  • Culture
  • Business model
  • Org chart and relevant departments
  • Product lines
  • Growth areas & strategic projects
  • Competitors
• Team overview
  • Mission and objectives
  • Key projects & relevant stakeholders
  • Measures of success
  • Culture
  • Team members and skill sets
  • Rituals (e.g. standups, weekly updates)
• Technology & tooling overview
  • Relevant systems/tools/software (plus getting access)
  • Conceptual overview of key tools
  • Hands-on projects using key tools

Generally, I dedicate about 1 hour a day to lecturing on these topics, getting as far as I can in each within the allotted hour. Sometimes it takes a week to go through the orientation, sometimes two. I realize this does not “scale well” - recordings would be much more efficient - but I think it’s important to signal active investment into your new hires and adapt each training individually to them.

For the technology portion, I like the training to be more hands-on. You can only learn so much from listening to someone else speak; sometimes you need to watch them work (shadowing) or play with things for yourself. 

As a result, I will often, separate from the lectures, share the entirety of my screen for about an hour a day. This way, new hires can see where my eyes go, what I click on, how I flip between email and chat apps and my IDE, what shortcuts I use, how I approach certain problems, how I behave in meetings, and so on. Often as a new hire, you don’t really know how the sausage gets made, but if you can literally see everything that goes on on my screen, you will know.

Motivation

Once new hires are trained to be productive in their role, there is no guarantee they will actually be productive. That’s because it’s easy to become checked out at work if you don’t have a positive experience with the company. Enormous amounts of productivity are lost by businesses each year, not because workers are incapable, but because they are unmotivated.

Although keeping your team motivated never ends, the first impression for how you treat others as a manager is certainly solidified during the first 90 days. When someone starts on their first day, I always ask: “What really drives this person? What would keep them motivated?”

Many a book have proposed motivators - money, flexibility, ownership, autonomy, mission, impact, balance, stability, talented colleagues, learning - and all of these are good. To find out which of these are most motivating for someone, I typically ask a bunch of questions:

• What do you like doing in your free time? What do you enjoy doing outside (or inside) work?
• What skills do you want to develop?
• What were some good experiences you had in the past with colleagues?
• What kind of projects have you enjoyed working on most in the past?
• What kind of career growth and responsibilities are you looking for?
• Which projects, and with whom, have you enjoyed working on so far the most?
• What about our industry, firm, company, team or technology would you like to learn more about?

Although my job is to build a personal relationship with every new hire, I also want them to build “lateral connections” with their team members. After about 3-4 direct reports, I don’t want everyone asking me individually for everything. Instead I want them asking each other. To facilitate this, I’ll schedule a series of events for new hires to build relationships, not just with me, but with the broader team, such as:

• Coffee chats
• Buddy system
• Lunch & learns
• Team projects
• Standups, daily or every other day

Finally, I’ll schedule 30-, 60- and 90-day check-ins to follow up on how their experience with the firm is progressing.

Conclusion

To me, the first 90 days for a new hire are not about the work. Any work here must necessarily be inconsequential, otherwise the firm is placing mission-critical work on untrained hires, which reflects a much larger problem.

Taking an interest in what allows employees to be their most productive and their most content goes a long way. I find that regularly revisiting the questions of “how can I train and motivate better” pays long-term dividends in how people perform on the job.

The job of training is to shrink the productivity gap between the most experienced (tenured employees) and the least experienced (new hires). If this does not lessen over time - if new employees remain trapped in the mire of unproductivity - this points to a lack of training. The goal of the first 90 days then is to be a force multiplier in taking one’s original, raw productivity and leveling it up to that of existing staff.

When we think of data, we typically think of structured data. Data which fits cleanly into a data table, neatly organized into rows and columns. Data which is readily amenable to calculating totals, averages and distinct counts. Data which is easy to summarize and, in turn, understand.

We don’t typically think of unstructured data: text, images, videos, PDFs and other proprietary formats, which collectively comprise the bulk of information in the world. In fact, the universe of unstructured data must necessarily be larger than that of structured data: all structured data ultimately derives from unstructured sources.

Real life is unstructured. When customers walk into and out of a retail store, video cameras do not store data points like {“customer_id”: “fcfbd2e1da00573f”, “direction”: “ENTRY”, “timestamp”: “2023-08-04T14:03:27.215”}. 

Instead, they record raw sensor data which must be converted into records and fields, rows and columns. To produce the data point above, we must classify a customer, match it to existing customers (or not), classify the direction, and finally record the timestamp. Classifying a customer is itself no trivial task: we need a machine learning model trained on a large volume of historical data that associates image data with “people”, and further, people with historical “customers”.

Even a simpler case of unstructured data - the business email - can be instructive:

Hi Jane,

Our team met last week to discuss the Jan. 24 email campaign, and although the overall performance was good, we had a few lingering questions. Could you provide the underlying campaign data from MailChimp when you get a chance?

Thanks,
Sam

From this unstructured data, we might extract the total number of characters in the email (LEN(email_body)), the number of lines (LEN(STRING_TO_ARRAY(email_body, ‘\n’))), and whether or not the email contains the word “data” (CASE WHEN CONTAINS(‘data’, email_body) THEN TRUE ELSE FALSE END).

These are simple extractions. What if we wanted to know whether the email is_marketing_related or not? 

It’s a useful property to know, but not one that is easily parsed from the email text alone. Here, we would again need a machine learning model trained on a large corpus of text to calculate if this email is “similar” enough to a set of pre-classified, marketing-related emails.

Such a machine learning model does not come cheaply. With structured data, we are off to the races, but with unstructured data, each classifier requires investment into a new model. 

Absent a clear business case to invest in such a model, most unstructured data lives on the periphery of data analysis, and by extension, comprehension.

LLMs as no-code machine learning classifiers

Since the launch of ChatGPT in 2022, the cost of encoding unstructured data has dramatically fallen. 

No longer must we build a bespoke machine learning model to classify whether an email is_marketing_related or not, we simply put the email into the LLM and ask. No longer must we ask if an image contains a dog or not, we simply put it into the LLM and ask. If we want to change the encoding to classify cats instead of dogs, it is as simple as updating one’s prompt. 

LLMs have ushered in the era of no-code classifiers. Once the data is encoded, it is immediately consumable in data analysis, statistical analysis and even further machine learning.

The mad dash for unstructured data

As the cost of encoding unstructured data falls, the demand for unstructured data will rise. From where will companies source this unstructured data?

Data, whether structured or unstructured, is typically sourced from three main channels:

1. It is collected
2. It is scraped
3. It is bought (or bartered)

Facebook, for example, collects vast amounts of behavioral data from the users of its platforms. Surveillance systems, such as camera and video tracking systems, collect footage of people who enter within certain premises. Websites collect profile information from users who seek to register to the platform.

In virtually every case of collection, there is a quid pro quo: a service must be provided in return for user consent. If companies want to collect their own unstructured data, they must then provide a valuable service to get it.

Scraping data is what occurs when consent is not expressly obtained. We can, for example, build a crawler to scrape LinkedIn public profiles or, if you’re Google, the entire Internet. However, precisely because the data is necessarily open to the general public, it is often not as valuable as data which is collected privately.

The final route is to buy data or barter for it, for example by offering a bi-directional data integration. Data sets for sale frequently suffer from the same drawback as scraped data: end users typically do not consent to having their data used by anyone except the service providers they directly interface with, and so any remaining data they do consent to share is of low quality. Nevertheless, for data sets which do not contain user information (such as economic or financial data), purchasing data is a common approach for acquiring data.

A data strategy for unstructured data

In the future, companies will put more thought into how they expand their data footprint. Unstructured data will no longer be considered prima facie inaccessible: with an LLM encoder, unstructured data can be cheaply encoded. The challenge will be to procure it.

Companies will be increasingly creative and nimble in how they acquire data. They will build free apps to collect data from users, crawlers to scrape repositories of public text (such as website DOMs), and relationships with data brokers who amass large stores of unstructured data. They will be just as thoughtful in how they acquire unstructured data tomorrow as how they acquire structured data today.