Every case I've written about so far shares one assumption I want to challenge directly: that Brand Drift is something that happens on the way to a customer. It isn't only that. Some of the most consequential drift I've come across never reaches a customer at all — it happens entirely inside the organization, in the systems deciding who gets hired, who gets promoted, and who gets a second look, and it can do just as much damage to what a brand actually is, even if no customer ever sees a trace of it.
The recruiting engine that quietly learned to prefer men
Starting in 2014, Amazon's machine-learning team built an experimental tool to automate part of its hiring process — scoring job applicants on a one-to-five scale, the way Amazon shoppers rate products, with the aim of surfacing top technical candidates faster than a human reviewer could. By 2015, the team discovered the tool wasn't rating candidates for technical roles in a gender-neutral way. The model had been trained on ten years of resumes submitted to the company, and because the tech industry those resumes came from was heavily male-dominated, the system taught itself, entirely on its own, that male candidates were preferable. It penalized resumes containing the word "women's" — as in "women's chess club captain" — and downgraded graduates of at least two all-women's colleges. Amazon edited the tool to neutralize those specific terms, but lost confidence that the system wasn't finding other, subtler ways to discriminate, and scrapped the project in 2017.
Why this is Brand Drift, not just an algorithmic bias story
This case gets told constantly in AI ethics circles, and for good reason — it's one of the clearest documented examples of biased training data producing a biased model. But I think the framing usually stops one step short of the point that matters most for how I think about brands specifically. Amazon's stated values, at the level any company states such things, presumably included some version of fair, merit-based evaluation of candidates. The tool's actual behavior drifted sharply away from that stated value, silently, for at least a year, before anyone caught it — and the reason it eventually surfaced wasn't an external scandal or a viral screenshot. It surfaced because Amazon's own team happened to notice the pattern during internal review, which makes this one of the rare cases in my research where sensing genuinely worked as intended, catching the drift before it did lasting external damage, rather than an outsider catching it first.
That's worth sitting with, because it means Amazon's story is simultaneously an example of drift and an example of the sensing capability actually functioning — which makes it a more useful teaching case than most of the ones I write about, precisely because it shows what catching this early actually looks like in practice, rather than only showing what missing it looks like.
Why internal-facing drift is easier to miss than customer-facing drift
Here's the structural reason I think internal AI tools deserve their own separate governance attention, rather than being folded into the same review process as customer-facing content. A drifting customer service chatbot generates a complaint, a screenshot, a public data point almost immediately, because a customer experiencing something off-brand has both the motivation and the platform to say so loudly. An internal hiring tool that's quietly disadvantaging a category of candidate generates no complaint at all from the people it's filtering out, because those candidates never make it far enough into the process to know they were evaluated unfairly in the first place, and the people who do see the tool's outputs — internal recruiters — have far less reason to suspect a pattern than an outside journalist actively looking for a story would.
This means internal-facing AI tools carry a structurally higher risk of running undetected for longer, not because the underlying failure mode is different, but because the external feedback loop that eventually catches customer-facing drift simply doesn't exist for anything that never reaches a customer or the public at all.
What this means for how a brand's values actually get lived, not just stated
I think this case also pushes back, usefully, on a framing I've used elsewhere that treats Brand Drift purely as a question of what customers experience. A brand's identity isn't only what it says to the outside world — it's also what its internal systems actually do to the people inside and around the organization, including the candidates it never hires. A company that states a public commitment to fair opportunity while running an internal tool that silently filters out qualified women isn't drifting in some narrow, technical, HR-department sense. It's drifting in exactly the same identity-versus-output gap I've described in every other case in this research, just facing inward instead of outward, where almost nobody but an unusually attentive internal team is positioned to ever notice.
Why I don't think this was actually a clean sensing success, on closer inspection
I want to complicate the mostly positive read I gave Amazon's response a moment ago, because I think the full picture is more instructive than a simple "they caught it, well done" story. The tool ran for at least a year before the bias was noticed, and Amazon's own account suggests the underlying problem — that ten years of male-dominated hiring data would inevitably teach a pattern-matching system to prefer male candidates — was arguably predictable from the moment the project was conceived, not just from the moment the skewed outputs were finally observed. That's a sensing failure at the design stage, even though it's a sensing success at the review stage. The team built the tool, trained it on data with an obvious and foreseeable skew, and only caught the resulting bias after the fact, through outcome review, rather than anticipating it structurally before the tool ever touched a real candidate's resume.
This distinction matters for how I think about building sensing capability more broadly. There are two different places sensing can happen: at the point of designing a system, where you ask what biases the training data is likely to encode before you ever deploy it, and at the point of reviewing a system's outputs after deployment, where you catch the bias empirically once it's already been operating on real people. Amazon's case is a genuine success at the second kind of sensing and a genuine failure at the first, and I think most organizations building AI tools today, whether for hiring, customer service, or anything else, are making the identical trade — skipping the harder, more speculative design-stage sensing in favor of hoping the review-stage sensing catches whatever slips through, which is a strictly riskier bet, because review-stage sensing only works if someone actually runs the review, on a schedule, with the specific intent to look for exactly this kind of pattern.
A wider point about what "internal" actually includes
I'd also push back gently on treating "internal tools" as a narrow category limited to hiring software specifically, because the same structural risk applies to internal knowledge assistants answering employee questions about company policy, internal AI systems drafting performance review language, and any other AI-mediated process that shapes how people inside an organization are treated, evaluated, or represented to each other. None of these produce a customer-facing complaint either, for the identical structural reason a biased hiring tool doesn't: the people affected by a drifting internal system are rarely in a position to notice the drift themselves, still less to raise it publicly, which makes internal AI governance, across every one of these categories, a genuinely under-resourced blind spot relative to how much organizational and human damage it can quietly do.
What actually closes this specific gap
The fix here looks slightly different from the customer-facing version, because the audience doing the checking has to be internal by necessity — there's no outside journalist who's going to stumble onto your hiring tool's training data. That means the sensing responsibility falls entirely on whoever owns the tool internally, and it has to be built as deliberately as any customer-facing check: a periodic audit of the tool's actual pattern of recommendations, broken out by the categories a stated value claims to protect, run by someone with both the technical access to see the pattern and the standing to act on it if the pattern looks wrong. Amazon's team happened to have both. Plenty of smaller organizations building or buying similar tools today have neither, which is precisely the condition under which this exact failure mode runs the longest before anyone notices it at all.
Wondering whether your own internal tools are quietly drifting from what your brand claims to stand for, in places no customer will ever see? Get Your Free Brand Audit →
References
Dastin, J. (2018, October 10). Amazon scraps secret AI recruiting tool that showed bias against women. Reuters.
MIT Technology Review. (2018, October 10). Amazon ditched AI recruitment software because it was biased against women.