Skill.re
Chain-of-Thought and Multi-Step Reasoning for Technical Problems
Thinking Out Loud Works Better
You're debugging a weird race condition. You ask the AI: "Why is this happening?" It gives you a plausible answer. You apply the fix. It doesn't work.
You ask differently: "Let's think through this step-by-step. What could cause a race condition here? What are all the possibilities? Which is most likely given the code structure?" Now the AI walks through its reasoning. Halfway through, you both realize something the first answer missed entirely.
This is chain-of-thought reasoning. It's not a trick. It's making the AI actually think instead of pattern-matching on your question.
When you ask a simple question, the AI gives a simple answer based on pattern-matching. It's fast but often misses the complexity. When you ask the AI to reason step-by-step, to show its work, to explore the space before concluding. You get dramatically better results on difficult problems.
Why Chain-of-Thought Actually Works
This is worth understanding because it changes how you use AI. Large language models are fundamentally predicting the next token. When you ask a question, they predict what the answer token should be. With complex problems, the first answer that seems plausible is what they generate.
But when you ask them to show their work, to write out "Step 1: [reason] → Step 2: [reason] → Step 3: [conclusion]". They're forced to generate intermediate steps. Those intermediate steps are constraints that prevent them from jumping to wrong conclusions. They're much more likely to catch their own errors when they have to reason step-by-step.
Example: Debugging Performance
Without chain-of-thought:
"This endpoint is slow. Here's the code: [code]. How do I fix it?"
You get something like: "The issue is probably the database query. Add an index on the user_id column."
You add the index. The endpoint is still slow. You're confused because the AI seemed confident.
With chain-of-thought:
This endpoint is slow. Let's think through why, step-by-step.
Here's the code:
[code]
Step 1: What does this code do? (Walk through line by line)
Step 2: What are all the ways this could be slow? (List every possibility)
Step 3: Which slow-downs are most likely? (Give reasoning)
Step 4: How would we verify each? (What would we check?)
Step 5: Once we know the problem, what's the fix?
Now the AI walks through it. Maybe it realizes the code makes 100 separate database queries in a loop (N+1 problem). That wouldn't show up in a superficial analysis. Or maybe it's a network call that times out. Or maybe it's algorithmic complexity that's fine for 100 items but catastrophic for 10,000.
Chain-of-thought forces the AI to find the real problem instead of guessing.
Example: Architecture Decisions
Without chain-of-thought:
"Should we migrate to microservices?"
Answer: "Microservices offer better scalability and independent deployment. You should do it."
You start a six-month migration. Halfway through, you realize microservices created operational complexity that's killing your team's velocity. The answer was technically correct but missing the context of your actual constraints.
With chain-of-thought:
We're considering migrating to microservices. Let's think through this carefully.
Our current state:
- 20-person engineering team
- Node.js monolith
- Running on 12 servers
- Processing 50K requests/second
- Deployment takes 30 minutes (builds + tests)
Let's think this through:
- What specific problems would microservices solve?
2. What new problems would they create?
3. What's our actual bottleneck? (Deployment speed? Scalability? Team coordination?)
4. Are there simpler solutions to that specific bottleneck?
5. What's the actual cost? (Development time, operational complexity, infrastructure)
6. What's the cost of NOT doing it?
Only then: Given all this, should we do it?
The AI reasons through it. Maybe it realizes: "Your actual bottleneck is deployment speed, not scalability. Microservices would add team coordination overhead that doesn't help with deployment. Better options: implement faster testing, CI/CD pipeline improvements, or better monitoring."
You avoid a six-month detour because you forced reasoning instead of accepting pattern-matched answers.
The Iterative Refinement Approach
Chain-of-thought is powerful for single-shot analysis. But for complex problems, iterative refinement is where the magic happens.
Instead of asking one big question and getting one answer, you have a conversation. You propose a direction. The AI responds. You refine based on what you learn. You go deeper into branches that matter. You discard dead ends quickly.
Example: Designing a Rate Limiter
Step 1 (You): "We need to design a rate limiter for our API. 1000 requests per minute per customer, distributed across 5 regions. What should we consider?"
Step 2 (AI): "Key considerations: (1) Distributed state, must sync across regions, (2) Accuracy, what's acceptable tolerance? (3) Failure mode, do we block or allow on failure? (4) Performance, must be <1ms overhead. Here are approaches: Redis with Lua scripts, in-memory with gossip protocol, centralized decision service."
Step 3 (You): "We use Redis already. But here's the constraint: one region occasionally loses Redis briefly. What should happen then? And how do we handle clock skew between servers?"
Step 4 (AI): "If you lose Redis: either block aggressively (safe, but frustrating) or allow everything (risky). Clock skew: use server time from Redis, not local time. Here's a design: [specific proposal]"
Step 5 (You): "That works. Now here's real code. Implement it. Here's our existing Redis wrapper and error handling patterns, follow those."
Step 6 (AI): "[Implementation that actually works in your system]"
Notice what happened: you didn't ask for a rate limiter and get a generic answer. You iterated. You revealed constraints. You refined. You ended up with something tailored to your system, not a textbook example.
When to Use Chain-of-Thought vs. Direct
Chain-of-thought isn't always necessary. For simple tasks, "write a function that sorts this array", direct requests work fine. Reserve chain-of-thought for situations where the problem is complex, the right answer isn't obvious, or you're exploring trade-offs.
Use chain-of-thought when:
- You're debugging something where the cause isn't obvious.
- You're making architectural decisions with trade-offs.
- You're exploring a design space and want to see options.
- You need to understand someone else's complex code.
- You're worried the AI might miss something important.
Use direct requests when:
- The task is straightforward.
- You need something quick and you're okay with good-enough.
- You're generating code for a clear spec.
- You're familiar with the problem domain and just need execution.
Combining Chain-of-Thought with Context
The most powerful approach combines chain-of-thought with rich context. You give the AI enough context to reason well, then ask it to think step-by-step about your specific situation.
Our system:
- Node.js microservices
- PostgreSQL (primary), Redis (cache)
- 500K users, 10K online concurrent
- AWS deployment
We have a bug: Users occasionally see stale data after login.
Step 1: What could cause stale data reads?
Step 2: Given our architecture, which is most likely?
Step 3: How would we reproduce it?
Step 4: What data would we log to prove our hypothesis?
Step 5: Once we find the root cause, what's the fix?
The AI knows your architecture. It reasons about your specific system. It's not offering generic debugging advice. It's reasoning about your situation.
Documenting Your Reasoning
Here's a practice that pays dividends: save the chain-of-thought reasoning, not just the conclusion. When you ask the AI to debug something, capture the step-by-step reasoning. Why? Two reasons:
First, when the fix doesn't work, you can see where the reasoning went wrong and correct it. Second, that reasoning becomes training material for your team. Junior engineers read how you think through problems. They learn your decision-making process.
Common Mistakes with Chain-of-Thought
Being too vague about the problem: "This is slow" doesn't help. "This endpoint takes 2 seconds when processing 1000 items" does. The AI can't reason well without specific information.
Not letting the AI actually reason: If you give it the answer in your prompt ("I think it's a database issue"), the AI just confirms what you said. Let it explore independently.
Too many steps: 20-step reasoning becomes unwieldy. 5-8 steps is usually right. Balance depth with readability.
Skipping the "why": If you ask each step but don't ask the AI to explain its reasoning, you miss the insight. Always ask "why" or "what's your reasoning."
Real World Example: A CTO's team was debugging a webhook delivery system that occasionally lost events. They asked the AI: "Why are we losing webhooks?" Answer: "Message queue is full." Wrong. With chain-of-thought, breaking down what happens when webhooks arrive, how they're queued, how they're processed, what happens on failure. They discovered the real issue: message queue was fine, but failed deliveries weren't being retried because a typo broke the retry logic. Chain-of-thought forced the AI to trace through the system instead of guessing.
Case Study: Chain-of-Thought Catching Production Bugs
A financial services company had an intermittent bug: payment confirmations were sometimes delayed by hours. Their engineering team asked the AI: "Why are confirmations delayed?" The immediate answer: "Database queries are slow. Add indexes." They added indexes. Confirmations were still delayed. Total wasted engineering time: 6 hours investigating, applying fix, testing, rolling back.
Then they tried chain-of-thought. They asked: "Let's reason through payment confirmation flow. What happens when a payment is processed? (Step 1) Payment service receives request. (Step 2) Payment is processed. (Step 3) Confirmation event is generated. (Step 4) Confirmation service processes event. (Step 5) Email is sent. Which step could cause delays?" The AI reasoned: "Step 5 could delay if email service is overloaded. But delay would be visible in logs. Let's assume that's not it. Step 4: confirmation service processes event, does it process synchronously or asynchronously? If asynchronously, there could be queue delays. Let's check that." This directed the team to look at the confirmation service queue. They found it: a bug in the queue consumer was causing it to crash on certain payment types, so confirmations for those types were stuck in the queue until manual intervention.
The team fixed the actual bug in 1 hour. Total time saved vs. the index approach: 7 hours. More importantly: they understood the system better afterward because they'd reasoned through it step-by-step.
When This Goes Wrong: Overthinking Simple Problems
A team asked the AI to debug a simple issue ("imports are failing") using chain-of-thought with 10 steps. Step 1-10 explored every theoretical possibility (circular imports, module path issues, caching, environment variables, etc.). The actual answer: they'd forgotten to add __init__.py to a directory. Chain-of-thought was overkill. Lesson: use chain-of-thought proportional to problem complexity. For simple problems, simple debugging is enough.
Key Insight
Chain-of-thought is the difference between AI guessing and AI reasoning. On complex problems, make it show its work. You'll catch errors before they become expensive problems.
What to Do Monday Morning
- Pick the hardest problem you're debugging this week. Ask the AI to think through it step-by-step instead of jumping to an answer.
- For your next architectural decision, write out the chain-of-thought prompt. Make the AI reason through trade-offs before making a recommendation.
- Next complex feature you're designing, use iterative refinement. Propose a direction. Get feedback. Refine. Iterate toward a solution.
- When the AI gives you an answer you're not sure about, ask "how did you arrive at that conclusion?" Make it show its reasoning. You'll catch bad logic faster.
- Save one chain-of-thought reasoning session. Share it with your team. Show them how it differs from direct requests.
Frequently Asked Questions
Q: Doesn't chain-of-thought take longer than just asking directly?
Sometimes yes, sometimes no. If you ask a simple question and get the wrong answer, iterating to the right answer takes longer than step-by-step reasoning would have. On complex problems where the right answer isn't obvious, chain-of-thought usually saves time because you catch errors early instead of discovering them later in implementation.
Q: Can I use chain-of-thought for code generation or just analysis?
Both. For code: "Think through how this should work before writing code. Step 1: What are the inputs and outputs? Step 2: What are the main operations? Step 3: What could go wrong? Now write code that handles all of it." For analysis: same pattern. For design: definitely use it.
Q: What if the AI's step-by-step reasoning disagrees with my intuition?
That's often valuable. Your intuition can be wrong, especially on complex problems. Read through the AI's reasoning. Where does it diverge from your thinking? Maybe it's right. Maybe it's missing context you haven't explained. Push back: "That doesn't seem right because..." and see what the AI says. Sometimes you'll realize your intuition was wrong.
Q: How do I know if the AI's step-by-step reasoning is actually correct?
You verify it. Don't accept it blindly. Read through the logic. Ask follow-up questions. "Why is that true?" "What if [edge case]?" "How would we test this?" Use chain-of-thought to make the AI's reasoning visible so you can evaluate it, not just accept it.
Q: Should I always use chain-of-thought or just sometimes?
Use it when it matters. Simple, low-stakes problems: direct request is fine. Complex, high-stakes, unclear problems: chain-of-thought pays for itself. If you're unsure which category a problem falls into, chain-of-thought is safe. Worst case, it takes a bit longer. Best case, it saves you from a bad mistake.
Q: What if the AI goes off track in the middle of chain-of-thought reasoning?**
Interrupt. "That step isn't right because [reason]. Let's reconsider." The AI can then self-correct. One of the benefits of chain-of-thought is that it makes errors visible so you can fix them mid-stream rather than discovering them at the end.
Q: Can I combine chain-of-thought with other prompt patterns?**
Yes. Combine chain-of-thought with examples ("Here's an example of good reasoning for a similar problem"). Combine with role-playing ("You're a systems architect. Think through this design decision"). Combine with RAG (give it context, ask it to reason through a problem with that context). These combinations are very powerful.