← All Weekly Query issues
NOV 17, 2026 The Weekly Query

The Weekly Query: How do you think critically when AI does the analysis?

Critical thinking, AI analytics news, impact, data jobs, and upcoming sessions.

Hai Guan
Hai Guan Co-founder, AI Analyst Lab

Hey folks, and welcome to another edition of The Weekly Query.

This week, Hai gets into how to keep thinking critically when AI is doing more of the analysis. We've also got eight data jobs asking for that judgment, recordings from last week's sessions, and a couple of free lessons coming up.

This week's query

How do you think critically when AI does the analysis?

By Hai Guan

Doing, thinking, understanding - and which one you can hand off

On Nov 9th, PwC US announced it is rolling out a Critical Thinking & Judgment Certificate for 85,000+ partners and staff starting in December. In other words, a firm that spent millions of dollars upskilling its people on AI is now spending more than $100M teaching them to question it.

What does this $100M+ program say about the state of affairs with people using AI in their professional work? It says that using AI and thinking critically are fundamentally different skills, and the second one doesn’t come free with the first.

In a separate study, Anthropic found that 50 professional analysts sided with the AI on 86% of judgment calls. These are calls where the AI's opinion was secretly set by a coin flip, and most people then explained the AI’s choice as their own.

We’re seeing more and more delegation of tasks and decisions as people either feel the pressure from the top or feel the urge within to maximize AI usage. In fact, throughout 2026, it felt like there’s this notion that the more one can defer work to AI, the more favorably they’re viewed as employees of the future.

I do not agree with that. Humans should always stay in control and use AI to amplify their work and ambition. And it seems like companies are starting to realize that too.

A few weeks ago I wrote about the gradient of reliance on AI and where things get dangerous.

Hai's note on doing, thinking, and understanding.
Hai's note on doing, thinking, and understanding. Click image to enlarge.

In this issue, we’ll take a look at the common ways things go wrong and what to do about it, in the context of doing analysis.

If we’re not careful, I fear that we’re headed down a dark path.

Three ways it goes wrong

I really like some of these comments from Substack on the topic and I’ll group them into everyday observations that you may have seen.

Will Preble on how surrender can feel like accepting a helpful default.
Will Preble on how surrender can feel like accepting a helpful default. Click image to enlarge.
Harish Gupta on using AI to challenge your thinking.
Harish Gupta on using AI to challenge your thinking. Click image to enlarge.
Harjaap on accepting a sensible-looking default.
Harjaap on accepting a sensible-looking default. Click image to enlarge.

There are three main ways that a data professional can fall into the trap of giving up their critical thinking.

The pass through - Handing over the output as is.

This is probably a common one. For an analyst or data scientist, this could look like: they use AI to run a query, get some numbers back, and those numbers go straight into a writeup that gets shared out.

A couple of months ago, I remember going into an executive meeting, where the presenter presented a deck that was obviously created by AI. There’s nothing wrong with that in and of itself; a tool is a tool if it helps with speed and quality. What actually happened was the presenter was walking the group through the AI content, deciphering the meaning of the slides together with the audience as if he had never seen it before! He started defending AI’s conclusions and attempted to justify why AI made choices that were obviously flawed. I can’t tell you how many times he said “Claude probably meant <a questionable claim> here”. Needless to say, his brand image took a dramatic hit and trust was completely lost (probably worse than if he had made a mechanical error by hand).

While this is an extreme story, I want to highlight that this happens more often than you'd think.

“Claude probably meant…”
“Claude probably meant…” Click image to enlarge.

No one ever wants to see that outcome.

The surrender - Letting AI make the call on what good is.

Another way it goes wrong is quieter. Nothing is broken and there's no error to catch.

People go with the default answer without double-checking or challenging the answers to make sure those are right and sound.

At the beginning of this article, we highlighted the Anthropic study where they found expert analysts default to an AI’s opinion 86% of the time even though the AI judgment was shaped by a coin flip.

Wharton’s Steven Shaw and Gideon Nave call it “cognitive surrender”: accepting the AI’s answer instead of reasoning your own way to one. In their study, the researchers gave people a set of reasoning puzzles. Some worked alone; others could ask an AI chatbot that, unknown to them, was sometimes rigged to give wrong answers. When the chatbot was right, the people using it beat the group working alone. When it was wrong, they did worse than that group, because they went along with its bad answers. And using the chatbot made them feel more sure of themselves, even when it had steered them in the wrong way.

The fold - You were right, but AI sounded surer, and you gave in.

This one is more subtle (probably the most alarming), and it’s only human nature to second-guess when our AI advisor is so confident in its opinion.

I’ve seen countless experts, who know what good looks like better than 99% of the world, doubt themselves when AI’s default answer goes against their opinion, formed after years of training and expertise. It is not that working with AI to sharpen thinking or get a second opinion is bad. It’s bad only if there are no criteria for drilling deeper into the AI’s answers.

When something feels off, even a little bit, it most likely is.

What can we actually do about it?

Before we can get into techniques to improve critical thinking, let’s define what it means first. I really like Google’s definition: “Critical thinking is the active, disciplined process of analyzing, questioning, evaluating, and synthesizing information rather than accepting it at face value.”

The critical-thinking definition quoted above.
The critical-thinking definition quoted above. Click image to enlarge.

Notice the words “active” and “disciplined”. Some people would say “those people are lazy” if they don’t do this, but it’s more nuanced than that.

As humans, we like to exercise as little of our brain as possible so we can conserve energy for survival. Daniel Kahneman called it the difference between System 1, the fast gut reaction, and System 2, the slow, effortful thinking our brains avoid whenever possible. Psychologists call us “cognitive misers”: we only think hard when we have to. AI gives us one more reason not to, so we’re just being human.

Here are five techniques to help with critical thinking:

  • Write your calls down first
  • Go through possible options and have AI argue them
  • Check against evidence you already have
  • Explain the choice back without AI
  • Use AI to understand, not just to produce

There are certainly more than I can cover fully here, so let me know in the comments what else you've seen work well.

Technique 1: Write your calls down first

Before using AI, how would you reason about the task at hand? Write that down first.

In the Anthropic study, even analysts who had already said what they’d pick switched 60% of the time when the AI disagreed. Committing first won't make you immune, but it's what lets you notice you switched at all and a Harvard study found it reduces over-reliance.

To generalize this further, one thing I like to ask people to do is anticipate the range of possible answers, and write down what their call would be for each one.

That way it is much harder to come up with justifications for the answer after you've seen it. As I’m sure many of you can relate to, it drives us mad when stakeholders use data to back up their decision after the fact.

“The first principle is that you must not fool yourself - and you are the easiest person to fool” - Richard Feynman, 1974

Technique 2: Go through possible options and have AI argue them

AI is good at coming up with an answer, but the work of comparing multiple alternatives then arriving at the best answer is what trains someone to be critical of any given one.

With people, someone is much more credible when they’re informed, meaning they’ve considered multiple solutions and multiple angles before making a decision. This shows deliberate thought instead of just a lucky first guess.

Same thing with AI, where it’s actually easier to ask for alternatives, adversarial cases, and critical blind spots you may have missed. So do all of that - put yourself in the shoes of a decision maker and ask what it would take to answer all your questions. In fact, ask AI to review and critique all the possibilities, once you or the AI come up with them.

Here’s an example prompt that could be helpful.

"Here's an analysis decision I need to make: [e.g. how to define an active user for this retention analysis].

1. List every reasonable option, including ones I haven't mentioned.

2. For each option, make the strongest case FOR it and the strongest case AGAINST it. Same effort for each. Don't tell me which one you prefer.

3. Tell me what I'd need to know about the business or the data to pick between them.

4. Then ask me which one I pick and why. Don't give your recommendation until I've answered."

As people like to say, we’re becoming managers of agents, so we need to be able to check if they’re doing a good job instead of letting them tell us they did.

Technique 3: Check against evidence you already have

This comes naturally for data professionals, but it is easy to miss when the context is seemingly outside of data.

Let me explain.

We’re trained to double-check and be rigorous about the numbers we present. That they should be right, they should match, and the spider sense should go off when some things don’t tie out. We go through dashboards, prior analyses, or other sources to ground our baseline with something verified, and that’s how we get so good and trusted when handling objective artifacts like data.

What I’ve seen is that when it comes to more subjective calls, like what the data is telling you, what to recommend, or what to do about it, that same rigor seems to disappear, because there's no obvious number to check against.

Our trained analytical thinking actually extends far beyond numbers and data. Perhaps it’s past judgment from the same person, maybe it’s how consistent the decisions are, or it could be whether the logic follows if certain choices were made - there’s always a benchmark to ground your baseline around.

So exercise the rigor you’ve been professionally trained in, and always watch for whether an AI answer checks out against other references you know.

Checking a recommendation against evidence you already have.
Checking a recommendation against evidence you already have. Click image to enlarge.

Technique 4: Explain the choice back without AI

Having done techniques 1-3, you should be able to explain your choice back without using AI.

This is where things get alarming in the Anthropic study, where 77% of analysts repeated the AI’s arguments as their own reasoning, sometimes word for word, and only 15% credited the AI.

An example of the study's opposing AI recommendations.
An example of the study's opposing AI recommendations. Click image to enlarge.

So if you change your mind, or if you think AI’s answer is truly better, write the “why” in your own words. That shows you’ve internalized its meaning and are truly convinced it is better.

Do this as much as you can, and over time you’ll recognize where AI sounds a bit fishy or where things don’t make sense even though it’s always confident.

Take agency to reject AI’s recommendations and answers, and work critically with it.

At the end of the day, you never know if you’re part of an experiment where the AI’s answers are manipulated for internal experiments or some other purpose.

Technique 5: Use AI to understand, not just to produce

Finally, the biggest unlock for critical thinking is to use AI as a thought partner to learn, not to outsource the understanding.

Challenge its assumptions, ask where it got the numbers from, question how it made certain connections in the data and why it reasoned the way it did. If you’re not fully buying into its preferences or opinions, don’t just skip over it, because everything else compounds from that baseline.

Next time you find yourself having AI do everything, insert yourself in the process to make sure you understand what’s happening, not just that the work is done.

What does this mean for everyone

As data professionals, we’re trained to be truth seeking and critical of evidence (in the form of data). This is not necessarily happening with the general public or people in other professions.

MIT researchers measured the brain activity of 54 people as they wrote essays, some using ChatGPT, some using a search engine, and some using nothing but their own heads. The ChatGPT users showed the weakest brain connectivity of the three groups and struggled to quote what they had just written.

So we’re seeing early evidence of AI’s impact on our thinking.

I believe everything we went through here can be useful, regardless of profession. If you find these helpful, share them around.

New in data this week

By Hai Guan

Claude Dashboards is now Claude Data

Anthropic made Claude Dashboards generally available on Nov 10 and renamed it Claude Data, alongside two new betas: Claude Sheets and Claude Whiteboards. We covered Dashboards in our first issue, back when it was still in beta.

Claude Data connects to sources like BigQuery, Redshift and ClickHouse (it needs a data connector to work), and lets people ask questions, build dashboards and share them live. The part I'm most interested in is certified metric definitions and certified queries, which let the data team mark the definitions and queries everyone should be using.

Databricks' Genie agent now works on its own

On Nov 11, Databricks unveiled an updated version of Genie, its AI agent, which can now take on multistep jobs that run for a while and, according to the company, complete them unsupervised. CEO Ali Ghodsi told TechCrunch he's been using it as "a source of unvarnished truth about what wasn't going well" (by helping him analyze employee satisfaction survey) since people in a 12,000-person company don't always bring bad news to the CEO.

The 30-second lesson: explaining your impact

By Hai Guan

There are 3 ways to think about how to articulate impact of your work:

  • You moved a business metric - your work led to more users, more revenue, better retention, etc.
  • You changed the roadmap - your insight sparked a new direction or stopped a bad investment.
  • You improved a process - your work made the team faster, smarter, or more efficient.

If your project didn’t do one of these things, or you can’t show how it did, then the work had no impact. It doesn’t mean the output wasn’t good, but no one could see how it mattered.

So track your work, follow up on what happened, document the results, because you want to tie the outcome back to your output.

Jobs this week

The same judgment Hai is talking about shows up in this week's job descriptions. Figma wants someone who can use AI-assisted tools, validate the output, and own the result. Chime asks for the judgment to distinguish a predictive relationship from evidence of causation. Mirage expects people using Codex and Claude Code to validate the work and explain its limits.

These roles ask you to use AI while taking responsibility for the question, the method, and what you recommend doing about the answer.

Here are eight open roles asking for that combination:

Staff Data Scientist - Ads Marketplace at Nextdoor

US remote. Listed San Francisco salary: US$190,000–$283,000/year; other locations vary.

Nextdoor expects people to use Claude, Gemini, ChatGPT, and Glean to challenge their thinking and take ownership of AI-assisted outputs. This role also requires designing experiments that account for marketplace interference, spillovers, and long-term effects.

Technical Operations & Insights Analyst at Figma

US hubs or remote US. Listed base salary: US$140,000–$245,000/year; the offer depends on location, level, and experience.

Figma wants this person to validate AI-assisted work, explain the assumptions and limits of the evidence, and recommend whether to proceed with an investment, test it, defer it, or stop.

Senior/Staff Data Scientist at Render

San Francisco or remote US/Canada. Listed salary: US$195,000–$268,000/year, plus equity; confirm the applicable offer for your location.

Render wants someone using Codex, Cursor, or Claude Code who can take an ambiguous business question, choose the right analytical approach, and challenge assumptions before turning the findings into a recommendation.

Staff Data Scientist, Product Marketing at Chime

San Francisco. Listed base salary: US$152,000–$210,000/year.

Chime asks this person to use AI to accelerate analysis while validating outputs and keeping the work reproducible. The role includes checking forecasts, quantifying uncertainty, and separating predictive relationships from causal evidence.

Data Scientist, Risk at Gusto

San Francisco hybrid. Listed cash compensation: US$186,000–$230,000/year. The posting describes the role as Staff-equivalent.

Gusto wants this person to use Claude Code and AI-assisted analytics while turning ambiguous stakeholder questions into testable analyses, choosing appropriate methods, and separating signal from noise.

Senior Data Scientist, Brand Analytics at DoorDash

San Francisco, Seattle, New York, Los Angeles, Chicago, Austin, or Washington, D.C. Listed US base pay spans US$124,100–$271,000/year across levels I4–I6; the applicable band depends on level.

DoorDash wants someone using AI coding tools such as Claude Code or Cursor who can define brand measurement, validate findings through hypothesis testing or causal inference, and turn the results into investment decisions.

Data Scientist, Product at Mirage

New York City, in person. Listed salary: US$120,000–$180,000/year, plus equity.

Mirage uses Codex and Claude Code in its everyday workflow and explicitly expects this person to validate their work. The role includes choosing appropriately simple methods, debugging incorrect outputs, and explaining uncertainty before recommending an action.

Staff Data Scientist, Marketplace Analytics at Zocdoc

Listed base salary: US$200,000–$270,000/year. The posting lists New York and US remote, but the application also asks about NYC hybrid attendance; confirm the location requirements with the recruiter.

Zocdoc wants someone who frames the problem before starting the analysis, uses causal inference when a standard A/B test isn't feasible, and explains trade-offs and uncertainty. Integrating generative AI into daily analytical work is also part of the role.

What's happening at AI Analyst Lab

Last week

We ran two lightning lessons on how data work changes when anyone can ask AI for an analysis. Both recordings are now on YouTube:

The New Data Job: What AI Does, What You Own

Watch the recording · The New Data Job: What AI Does, What You Own
Watch the recording · The New Data Job: What AI Does, What You Own

What should AI take on, what needs review, and what should data people lead? We worked through how to route questions before the data team becomes responsible for checking everyone else's AI output. This is one of the questions we'll go deeper on in our Enterprise AI Analytics for Data Leaders workshop in December.

Decide which data questions AI answers

Watch the recording · Decide which data questions AI answers
Watch the recording · Decide which data questions AI answers

We looked at how the cost of a wrong answer and the person's ability to recognize it affect which questions you hand over to AI. This also feeds into the rollout decisions we'll work through in the Data Leaders workshop.

Context management with Hex

Thanks to everyone who joined me, Hai, and Rachel from Hex for Enterprise Context Management for Analytics on Saturday. We worked through building and managing the context an AI analyst needs to understand your company and use your data correctly.

If you couldn't make it but want to work on this at your company, we've put up two resources you can use:

This week

Build a Promotion Case That Gets Approved

View the session · Build a Promotion Case That Gets Approved
View the session · Build a Promotion Case That Gets Approved

Tuesday, November 17 · 1–2 p.m. Pacific · Free

If Hai's lesson on explaining your impact has you thinking about your own work, join us this afternoon. We'll look at how promotion decisions get made, how to map your work to the next level, and how to explain your impact when you're not in the room.

Check if your data is ready for AI

View the session · Check if your data is ready for AI
View the session · Check if your data is ready for AI

Wednesday, November 18 · 10–11 a.m. Pacific · Free

Take a question your company asks often and look at the data it needs. We'll work through what needs fixing first and what can wait, so you can make a realistic plan for your first AI analytics use case.

Coming up

Enterprise AI Analytics for Data Leaders

View the session · Enterprise AI Analytics for Data Leaders
View the session · Enterprise AI Analytics for Data Leaders

Saturday, December 12 · 8–11 a.m. Pacific · Hai and Shane · US$380

If you're responsible for rolling AI analytics out across your company, join Hai and me for this half-day workshop. We'll work through where to start, who owns the definitions and context, how to evaluate the tools you already have, and what needs to happen before you expand to more teams. You'll start building a 90/180-day plan for your company during the session.

Join us for Enterprise AI Analytics for Data Leaders.

Would you join an AI evals workshop?

We're considering a one-time, two-day workshop on AI evals for AI analytics, probably around $90. If that's something you'd be interested in, reply and let me know. I'd like to get a feel for whether we should run it.

Thanks for reading. See you next week.

Shane and Hai

Share this post

Hai Guan
Written by
Hai Guan
Co-founder, AI Analyst Lab

Ran data at Nextdoor, LinkedIn, Pinterest and Meta. 16+ years teaching product people to decide with data.