Most discussions about AI eventually collapse into the same question:
Will it take our jobs?
That question is understandable. AI can already write software, analyze documents, summarize enormous amounts of information, identify patterns, generate possible solutions, and increasingly interact with computers and other tools autonomously.
Many tasks that once required trained professionals can now be completed, at least partially, by machines. So the fear is obvious. If AI can perform more and more of the work, what happens to the people who used to perform it? The answer depends on a distinction that is often missed. AI is becoming extraordinarily good at execution: producing code, classifications, analyses, recommendations, reports, and possible solutions. But producing an answer and knowing whether that answer is correct, relevant, safe, or important are not the same thing. As execution becomes cheaper, the bottleneck moves. And as answers become cheaper and abundant, knowing which questions matter may become considerably more valuable.
A Bank With Too Many Complaints
Imagine a large bank with a customer complaint database that has become almost unmanageable. Thousands of complaints are waiting to be processed: failed transfers, disputed transactions, account access problems, unexpected fees, possible fraud, incorrect balances, and countless variations of the same recurring problems.
Before AI, the only practical way to deal with the backlog was painfully simple. Someone had to read the complaints one by one, understand the issue, decide where it belonged, investigate it, and determine the appropriate response. That does not scale particularly well. The bank may simply be unable to process complaints as quickly as customers generate them, so the backlog continues to grow.
Now imagine two experienced analysts are assigned to solve the same problem using AI. They do not simply use AI to read individual complaints faster. Instead, they change the way the problem itself is approached. They ask AI to classify thousands of complaints into recurring categories: failed transfers, card disputes, account lockouts, fraud reports, unexpected fees, loan servicing issues, privacy concerns, and so on. The AI summarizes each category, identifies patterns, and proposes possible resolution strategies.
Suddenly, the analysts are no longer spending most of their time reading individual complaints. Their job has moved up a level. They begin reviewing the proposed solutions instead.
One analyst looks at a large group of failed-transfer complaints.
“The AI thinks most of these can be handled by automatically checking the transaction state and either retrying the transfer or issuing a reversal.”
The other replies:
“That sounds reasonable. But what happens if the receiving bank has already accepted the transaction before the reversal is triggered?”
They move to disputed card transactions.
“For low-value disputes, it suggests issuing temporary credits automatically.”
“That could clear thousands of cases quickly. But if customers learn that small disputes are refunded automatically, have we just created a fraud incentive?”
The discussion continues with the AI proposing an answer and the analysts examining its consequences. Sometimes they approve the approach. Sometimes they modify it. Sometimes they reject it because of regulatory, fraud, accounting, information-security, privacy, or operational implications that were not obvious from the complaint itself.
Then, after working this way for some time, one of them raises a different kind of question.
“Are we sure these categories are actually the right way to organize the problem?”
The other pauses.
“What do you mean?”
“We asked the AI to group complaints by similarity. That is exactly what it did. But two complaints that look almost identical to a customer might have completely different root causes.”
A failed transfer might result from a temporary network error, an incorrect account number, insufficient funds, fraud controls, or a failure at another financial institution. Treating all of them as one problem class may produce a solution that works beautifully for some cases and makes others worse.
Then the second analyst notices another assumption.
“We are also measuring success by how quickly we can reduce the backlog.”
“Right.”
“But clearing complaints and fixing the problems that create those complaints aren’t necessarily the same thing.”
Now the nature of the discussion has changed. The analysts are no longer only evaluating the complaints. They are no longer only evaluating the AI’s proposed solutions. They are beginning to evaluate their own way of thinking about the problem.
Perhaps the AI should classify complaints not only by what the customer reports, but also by probable root cause, potential harm, regulatory significance, urgency, and systemic relevance. Perhaps samples from each category should be manually reviewed to test whether the categories themselves make sense. Perhaps the objective should not be “reduce the complaint backlog” at all. Perhaps the real objective should be “identify and eliminate the recurring causes of customer harm.” That transition is important.
The old bottleneck was: Can we process all these complaints?
Then it became: Can we trust the solution we are about to apply to thousands of customers?
And eventually: Are we sure we have designed the right way to solve the problem in the first place?
That final question takes us somewhere deeper.
When Success Creates a New Kind of Risk
There is another reason this matters. Real-world systems are open-ended. A solution can succeed according to the objective it was given and still produce consequences nobody intended. Consider a different banking example.
Suppose a bank has millions of old customer records containing inconsistent or duplicated addresses, perhaps because of past mergers or different systems capturing the same customer’s address at different times. It can be difficult to know which address is current. Some contain spelling mistakes; others use old formats. It’s a mess.
The bank launches a large data-cleaning project. Automated matching techniques compare names, addresses, account information using similarity scores to identify records believed to belong to the same person. The project appears highly successful. Duplicates disappear. Addresses become standardized. Records are consolidated. Data quality improves. Then an unusual case appears.
Two records that look like duplicates are automatically merged. From the system’s perspective, the reconciliation is correct. But there was a legitimate reason those records had been kept separate. Sensitive financial correspondence that previously went to one address now goes to another household address, where someone else sees information that was never intended for them. The data-cleaning system succeeded technically. The harm appeared somewhere outside the problem it had been asked to solve. This is what makes real-world automation difficult.
The obvious questions are about matching accuracy:
-
- Did we identify the same customer correctly?
-
- Was the similarity threshold appropriate?
-
- How many false positives did the algorithm generate?
But a more difficult question is:
-
- What could happen if we are right?
That sounds strange, but it matters. Sometimes success itself creates the next risk. A system may correctly identify two records as belonging to the same person while still being wrong to consolidate them automatically. The difficult part is recognizing that the problem contains dimensions that are not represented in the matching algorithm at all: privacy, family circumstances, legal restrictions, personal safety, confidentiality, or other unusual situations that may be nearly impossible to enumerate in advance.
An AI system could certainly be prompted to brainstorm such risks. It might identify many of them. But that raises a more important question:
Would it know, on its own, that this is the moment when it should stop optimizing the immediate objective and begin questioning the assumptions behind that objective?
That question becomes central as AI is given greater responsibility.
The Bottleneck Moves
Discussions about AI displacement often assume a fixed amount of work. The implicit model is simple. There are 10,000 tasks. AI can now perform 8,000 of them. Therefore, fewer people are required.
Sometimes that may indeed happen. But when execution becomes dramatically cheaper and available at enormous scale, organizations can also attempt things that were previously too expensive, too slow, or simply impossible.
Instead of reviewing a sample of customer complaints, a bank can analyze all of them. Instead of performing a security assessment every few months, an organization may continuously examine every software release. Instead of producing three strategic alternatives, a team may generate fifty. Instead of manually examining a fraction of transactions, analysts may inspect millions.
But increased execution creates a new problem. More output means more results to verify. More alternatives mean more decisions between them. More automation means that a bad assumption can be propagated faster and at much greater scale.
The bottleneck moves from producing possible solutions toward deciding which solutions deserve to be trusted and implemented.
The question changes from:
-
- Can we do this?
to:
-
- Should we do this?
and eventually:
-
- Are we even solving the right problem?
Why Software Engineering Is Particularly Exposed
Software development provides an unusually favorable environment for AI because much of the work generates rapid, machine-readable feedback.
An AI system can write code and quickly discover whether it compiles, whether the application executes, whether an API returns the expected response, whether tests pass, whether a page renders correctly, or whether a database contains the expected result.
When something fails, the error can often be fed directly back into the model. The AI modifies the code, runs it again, observes what happened, and tries another approach.
This creates a cheap and rapid verification loop. The loop is certainly not perfect. Software that compiles can still be wrong. Tests can be incomplete. Security assumptions can be flawed. The specification itself may be incorrect. An AI system that writes both the implementation and inadequate tests may even create the illusion that its work has been successfully verified.
Nevertheless, compared with many other professions, software provides imperfect but exceptionally rich feedback. The AI does not need to produce the correct answer on its first attempt. It can generate, test, correct, and repeat. The important advantage is therefore not merely that AI can write code. It is that the environment can often tell the AI when its code is wrong.
The Verification Gap
Now consider cybersecurity. Suppose an AI identifies a vulnerability. The weakness exists. The exploit works. The result is reproducible. At first glance, this looks similar to software engineering. There is a clear technical outcome that can be tested.
But cybersecurity introduces another question:
-
- How much does this vulnerability actually matter?
That question is considerably harder. An exploit may technically succeed while requiring conditions that are extremely unlikely in practice. A vulnerability may expose sensitive information but require privileges that already give an attacker much greater capabilities. A finding may look severe when viewed in isolation but become relatively unimportant once compensating controls, architecture, user behavior, deployment conditions, and the actual threat model are considered.
The reverse can also occur. A technically unremarkable weakness may become serious because of the specific way an organization operates. The question therefore changes from “Can this be exploited?” to “Who can exploit it, under what conditions, with what likelihood, against which assets, and with what consequences?” Technical verification alone cannot always answer those questions.
A Pentest That Changed How I Think About AI
I encountered this distinction directly during a mobile application penetration test. I was using an AI coding agent as part of the investigation. The test environment included an Android emulator with root privileges, and the agent had access to ADB and Frida while investigating how the application protected sensitive information.
The process was technically impressive. The AI encountered errors, tried alternative approaches, modified its commands, adjusted its instrumentation, and continued working until it eventually succeeded in extracting and decrypting protected application data. I was not manually providing every individual exploitation step. The AI was genuinely navigating the technical problem.
If the question had simply been “Can this data be extracted under these test conditions?”, the answer was clearly yes. Then I asked the AI to document the finding and assign a severity rating. It rated the issue Critical. That was where the more interesting failure occurred. The technical success was real. The interpretation was weak. The attack depended on a critical condition: the AI had root-level access to the device. That should have immediately affected the threat model. For an ordinary attacker to reproduce the same attack against a real user, the target device would already need to be rooted, compromised, or operating under similarly unusual conditions. The AI had successfully used root access as an operational tool. But it had failed to treat that same root access as an analytically important constraint.
The AI Should Have Interrogated Its Own Success
This is where metacognition becomes important. Metacognition is often described as “thinking about thinking.” That definition is correct, but too vague to explain why it matters here.
In practical terms, metacognition means examining your own reasoning.
-
- What assumptions am I making?
-
- What conditions allowed me to reach this conclusion?
-
- What information might be missing?
-
- How confident should I be?
-
- What evidence would change my conclusion?
-
- Have I solved the real problem, or merely a convenient version of it?
An experienced penetration tester does not stop when an exploit succeeds. They ask why it succeeded.
-
- What conditions made the attack possible?
-
- Which of those conditions belong to the vulnerability itself, and which belong to the privileged testing environment?
-
- Would a realistic attacker normally possess those capabilities?
-
- If not, how should that change the likelihood and severity assessment?
The AI should have interrogated its own success. It should effectively have asked:
-
- I succeeded. Why?
-
- I had root access.
-
- Would the attacker normally have root access?
-
- If not, how should that change my conclusion?
That step was missing. The AI had solved the technical problem but had not adequately examined the assumptions surrounding its own solution.
Once I explicitly pointed out the root requirement and asked it to reconsider the finding under a realistic threat model, it reduced the severity. The lesson is not that AI cannot reason. The technical investigation demonstrated substantial reasoning ability. Nor would it be accurate to claim that AI cannot perform metacognitive reasoning. Current systems can often critique their own conclusions, examine assumptions, reconsider an answer, and identify weaknesses when asked to do so. The more difficult question is this:
Can we trust AI to notice when metacognitive reasoning is required, perform it correctly, recognize its own blind spots, and escalate when necessary?
At present, I do not think we can reliably assume that. And that distinction matters enormously.
Problem Formulation and Metacognition
Problem formulation is closely related to metacognition, but it is not quite the same thing. Problem formulation asks:
-
- What exactly is the problem we should be solving?
Metacognition asks:
-
- Am I thinking about that problem correctly?
Suppose the pentest objective is implicitly formulated as: “Find any possible way to extract this data.”
That creates one problem space. But the real security question may be: “Can a realistic attacker extract this data under plausible conditions?”
Those are not equivalent. The first formulation permits almost any laboratory condition. The second introduces a threat model. This leads to an important distinction:
-
- Poor problem formulation can create bad reasoning boundaries. Poor metacognition prevents the system from noticing those boundaries.
The same distinction appears in the bank complaint example. At first, the analysts formulate the problem as:
-
- “How do we process this backlog?”
Then:
-
- “How do we safely resolve each category?”
Eventually they question the formulation itself:
-
- “Should reducing the backlog even be our primary objective?”
The thinking has moved from solving the problem to examining the structure of the problem.
What Happens to White-Collar Expertise?
For decades, much professional value came from knowing how to produce an artifact.
-
- A developer produced software.
-
- An analyst produced reports.
-
- A lawyer produced legal arguments.
-
- A penetration tester performed technical tests.
AI is reducing the cost of producing many of these outputs. That does not necessarily mean expertise becomes worthless. More likely, expertise moves elsewhere. The scarce skill increasingly becomes the ability to determine what we are actually trying to achieve, what assumptions are hidden inside an approach, what evidence would disprove a conclusion, what context is missing, what could happen if the solution succeeds, and what happens when that solution is deployed at scale.
This is where the two bank analysts become important. They are not valuable merely because they can check individual AI outputs. If that were all they did, AI might eventually absorb much of that work too. Their higher-level function is to supervise the reasoning system itself.
-
- Are we asking the right question?
-
- Are we using the right categories?
-
- Are our success metrics misleading us?
-
- What kinds of failure have we not considered?
-
- Could a locally correct answer produce a globally bad outcome?
-
- When should automation stop and escalation begin?
At present, metacognitive oversight is one of the areas where human involvement remains particularly valuable, not because AI cannot reflect on its own reasoning, but because we cannot yet rely on it to know when, where, and how deeply that reflection is required. That may become an important function of white-collar work.
Human Work Moves Up the Stack
None of this means AI will leave white-collar employment untouched. Quite the opposite.
A penetration tester who refuses to use AI may eventually compete with another tester who can operate several AI agents simultaneously, analyze vastly more data, explore many more hypotheses, and automate large portions of repetitive investigation.
Developers, lawyers, researchers, auditors, engineers, analysts, and many others may face similar changes. Some mechanical work will become cheaper. Some tasks may disappear altogether. But much of the remaining work may move upward in abstraction. Instead of manually executing every step, professionals increasingly define objectives, frame problems, challenge assumptions, evaluate evidence, identify second-order consequences, decide where automation should stop, integrate knowledge from multiple domains, and take responsibility for the final judgment.
AI becomes increasingly capable of answering questions. Human expertise increasingly concentrates on deciding which questions should be asked, which answers should be trusted, what the system may have failed to consider, and which consequences actually matter.
Breadth Still Matters
This shift may also increase the value of broad contextual knowledge. Deep expertise remains essential because without understanding a field properly, it is difficult to recognize when an AI-generated conclusion is subtly wrong. But breadth matters because the consequences of a solution frequently sit outside the domain in which the solution was generated.
-
- A banking solution may be operationally efficient while creating a regulatory problem.
-
- A technically correct data merge may create a privacy problem.
-
- A security control may reduce one risk while creating another.
-
- An elegant software optimization may produce an unacceptable business consequence.
Recognizing those interactions requires some ability to cross disciplinary boundaries. Deep expertise and broad contextual awareness therefore complement each other. The professional does not need to be an expert in every neighboring discipline, but needs enough breadth to recognize when another domain changes the meaning of the problem.
Conclusion
The most useful question about AI may therefore not be whether it will replace white-collar workers. That question is too broad. AI will automate some tasks aggressively, transform others, and almost certainly change the economic value of many existing skills. A more useful question is:
Where does the bottleneck move when execution becomes cheap?
As AI capability increases, the bottleneck can move from execution to verification, consequence assessment, problem formulation, and eventually the reasoning process itself. AI becomes extraordinarily powerful where generation can be combined with rapid, inexpensive, objective feedback. The harder problems begin where success depends on incomplete information, hidden assumptions, human behavior, second-order consequences, uncertain probabilities, or context that was never represented in the original task.
Future AI systems will almost certainly improve at these things too. They may become far better at questioning their own assumptions, identifying hidden consequences, and deciding when to escalate. We should not assume that metacognition will remain a uniquely human advantage. But neither should we confuse an AI’s ability to produce reflective reasoning when prompted with the ability to reliably recognize when that reflection is necessary. That difference matters. The world may soon have an abundance of answers.
The scarce resource may increasingly become the ability to decide which questions deserve to be asked, which assumptions deserve to be challenged, which answers deserve to be trusted, and when apparent success itself should trigger another question.
As AI makes execution cheaper, knowing when to question apparent success becomes more valuable.