Skill.re
AI for Log Analysis and Observability
Logs At Scale Are Overwhelming
A typical system produces millions of log entries daily. A single transaction might generate 20 log entries (user logged in, API called, database queried, response returned, error logged, retry triggered, success logged). Scale this to 1M transactions and you have 20M log entries. Humans can't read 20M entries. You need automated analysis to find signals in the noise.
The challenge: logs are unstructured ("User login failed: timeout waiting for database"), inconsistent (different services log differently), and noisy (most entries are routine). Finding the signal (what caused the outage?) in the noise (millions of routine entries) requires sophisticated analysis.
AI helps by automatically parsing logs, extracting patterns, detecting anomalies, and correlating events. When something goes wrong, instead of manual log investigation (hours of grepping and searching), the AI highlights the problem immediately.
What AI is Good At
- Parsing unstructured logs into structured data
- Extracting key information (error type, affected service, user ID)
- Identifying error patterns (which errors happen most frequently)
- Detecting anomalies (unusual error spikes, unexpected patterns)
- Correlating logs across services (if service A fails, what happens to service B?)
- Explaining error messages in plain language
- Finding root causes (what triggered the chain of errors?)
- Suggesting fixes based on error patterns
What it's not good at:
- Understanding business impact (this error matters, that error doesn't)
- Deciding what to alert on (needs human judgment about what's worth notifying)
- Understanding complex system behavior (sometimes errors are expected)
- Preventing false positives (it might flag harmless events as problems)
- Understanding semantic meaning (the logs say X happened, but you know Y actually happened)
Log Parsing and Structuring
Logs are messy. Different systems, different formats, different levels of detail. Before analysis, you need structured data.
"Analyze these logs and extract structure:
[Raw logs]
2024-04-09 15:23:45 ERROR [PaymentService] user_id=12345 txn_id=abc123 msg='Payment processing failed: Gateway timeout' retry_count=2
2024-04-09 15:23:46 WARN [PaymentService] user_id=12345 txn_id=abc123 msg='Retrying payment attempt 3'
2024-04-09 15:23:47 ERROR [PaymentService] user_id=12345 txn_id=abc123 msg='Payment processing failed: Gateway timeout' retry_count=3
...
Extract:
- timestamp
- service (PaymentService, UserService, etc.)
- level (ERROR, WARN, INFO)
- user_id
- transaction_id
- error_type
- retry_count
Show frequency:
- How many errors per minute?
- Which errors are most common?
- Which services are failing?
- User impact (how many users affected?)"
The AI parses unstructured logs into structured data. It extracts fields (timestamp, service, error type, affected user). It aggregates (100 payment failures in the last 5 minutes) and correlates (these failures all have the same error: "Gateway timeout").
Error Pattern Identification
Logs show what happened. Patterns explain why. The AI groups similar errors and identifies themes.
"Last hour, we had 450 errors. Group them:
Group 1: Gateway timeout (300 errors)
- Affect: PaymentService
- Pattern: All requests to external payment provider timing out
- Timeline: Started at 14:52, ongoing
- User impact: All payment attempts failing
Group 2: Database connection pool exhaustion (100 errors)
- Affect: UserService
- Pattern: Can't get database connection, queue full
- Timeline: Started at 14:55, ongoing
- Related: UserService has been slower → using more connections → pool exhausted
Group 3: Cache miss, slow fallback (50 errors)
- Affect: ProductService
- Pattern: Cache is empty, falling back to database, database is slow
- Timeline: Started at 14:50, improved at 15:10 (cache warmed up)
- Root cause: Cache restart, took time to rebuild
Summary:
- Primary issue: Payment gateway is slow/down
- Secondary: Gateway slowness cascaded to other services
- Tertiary: Cache restart affected product lookups
- Recommendation: Investigate payment gateway. Scale up database connections. Monitor cache health."
The AI automatically groups related errors and identifies root causes. Instead of "450 errors, what happened?", you get "these 300 are gateway issues, these 100 are pool exhaustion caused by gateway slowness, these 50 are cache-related." Much clearer.
Anomaly Detection and Alerting
The goal is to detect problems before they impact customers. Anomaly detection learns normal behavior, then alerts when behavior deviates.
"Set up anomaly detection for our system:
Metrics to monitor:
- Error rate: normally 0.01% (1 error per 10,000 requests). Alert if >0.1% (10x normal).
- Response latency: normally p99=200ms. Alert if p99>500ms.
- Database slow queries: normally 20 per minute.
- Memory usage: normally 60% average. Alert if >85%.
- Queue depth: normally 10,000 messages.
- External API latency: normally p99=100ms. Alert if p99>500ms.
Learning phase:
- Baseline for 2 weeks on normal traffic
- Learn what's normal (seasonal patterns, time-of-day effects)
- Set thresholds
Detection phase:
- Continuously monitor metrics
- Alert when metric exceeds threshold
- Correlate alerts (if payment gateway latency is high AND error rate is high, they're related)
- Escalate based on severity (low: log it, medium: notify on-call, high: page immediately)"
The AI learns normal patterns and alerts on abnormalities. This catches problems hours before customers notice them.
Root Cause Analysis and Correlation
When something breaks, you need to know why. Root cause analysis connects dots: what changed? What failed? What's the chain of causation?
"Error rate spiked at 3:15pm. Help me understand what happened:
[Metrics and logs from 3:00-3:30pm]
Timeline:
- 3:10pm: Database slow query count increases (from 5/min to 20/min)
- 3:12pm: UserService response latency increases (p99 goes from 150ms to 400ms)
- 3:15pm: Error rate spikes (0.01% to 1%)
- 3:16pm: Payment transactions start failing (Gateway timeout)
- 3:18pm: Cache miss rate increases (cache evictions?)
- 3:20pm: Situation stabilizes
What changed at 3:10pm?
- At 3:10pm, a deployment happened (checked change log)
- New code makes more database queries
- Queries are slower than expected
- Database connection pool fills up
- Services start timing out
- Error rate increases
- Payment service gives up on database, returns timeout
- Cascade effect
Root cause: New deployment made database queries slower. Scaling issues appeared under load.
Recommendation:
- Revert deployment or optimize queries
- Increase database connection pool
- Add circuit breaker to database calls
- Monitor query latency in pre-production"
The AI traces the chain of events, identifies the root cause (new code is slow), and suggests fixes. Without this analysis, you'd manually grep logs and might miss the connection.
Incident Response Automation
When an incident occurs, time is critical. The AI can automate response:
"Alert: Error rate exceeded threshold (1.5% vs. threshold 0.1%)
Automated response:
1. Analyze logs to identify root cause
2. Check recent deployments (was anything deployed in last 30 min? Yes, at 3:10pm)
3. Check metrics to identify affected systems (PaymentService, UserService)
4. Check if this is a known issue (similar errors happened last Tuesday, caused by X)
5. Suggest actions:
- Revert last deployment (high confidence it's the cause)
- Scale up database connections (error pattern matches connection exhaustion)
- Clear cache (might help with recovery)
6. Page on-call engineer with summary
7. Continue monitoring and update status"
Automation doesn't solve the incident, but it dramatically speeds incident response by diagnosing the problem immediately.
Observability Principle: You can't fix what you can't see. Good logging and analysis let you detect problems early and fix them fast. Log the right things, analyze them, alert appropriately.
When This Goes Wrong: Failure Modes in Log Analysis
Blind Spot: The Missing Signal**
You're not logging the right thing. A critical bug is happening but it's not in your logs. Example: Your application has a memory leak. It doesn't throw errors (it just slowly uses more RAM). Your logs don't mention memory. Your anomaly detection doesn't monitor memory. The system crashes at 3am because OOM, and you have no early warning. Solution: instrument comprehensive metrics. For each critical resource (CPU, memory, database connections, network bandwidth), log and monitor. Memory should have triggered an alert at 80%, 90%, 95% before it hit 100%.
The Alert Storm**
Your anomaly detection generates 100 alerts per day. Each one triggers a page to on-call. Nobody is sleeping. Most alerts are false positives. On-call burns out and stops responding to alerts. When a real issue occurs, it gets lost in the noise. Solution: use composite alerting. Don't alert on one signal. Require 2-3 correlated signals (error rate AND latency AND database slow queries all spiking). This cuts false positives 90% while keeping true incident detection.
The Investigation Black Hole**
You find an error in logs ("Connection timeout"). You investigate. No obvious cause. You grep logs for 2 hours. Finally realize: the error happened at 2:15am, the same second a deployment completed. But that connection wasn't in your deployment logs. So you missed the connection. Solution: correlate all events (logs, deployments, config changes, scale changes) into one timeline. When something goes wrong, first question: "What changed in the last 10 minutes?" Answer that and 80% of incidents are solved.
The Expensive Log Spiral**
You log every API request. 1M requests/day = 1M log entries. Storage costs $100/month. You scale to 10M requests/day. Log storage is now $1K/month. You scale to 100M/day. Now it's $10K/month and growing. You can't afford to keep logs anymore. But you need them for debugging. Solution: be selective from the start. Log errors (always). Log business-critical events (100%). Log routine operations (sample 1%). This keeps costs linear even as you scale.
The Timestamp Trap**
Your application logs with timestamps. Your database has timestamps. Your external API has timestamps. All in different timezones or with different precision. You try to correlate events and the timestamps don't line up. You waste hours. Solution: standardize all timestamps to UTC. Use millisecond or microsecond precision. When you look at logs from your app, database, and API, timestamps should align perfectly.
Case Study: E-Commerce Platform at Scale
A fast-growing e-commerce platform handled 50K transactions/day. During a holiday sale, traffic jumped to 500K transactions/day (10x increase). Their logging infrastructure, designed for 50K/day, collapsed.
What Happened:** Logs went from 5M entries/day to 50M entries/day. Log shipping infrastructure got overwhelmed. Logs started dropping (being discarded to reduce load). The system started failing, but they had no visibility because logs weren't being recorded. They went dark for 2 hours while debugging blindly.
The Fix:** They implemented sampling and filtering. Instead of logging every request, they logged: all errors (100%), all failed transactions (100%), all slow queries (>500ms), random successful requests (1% sample). This brought log volume to 8M entries/day even at 500K transaction/day, totally manageable. They added sampling metadata so they could estimate totals from the sample ("we saw 1% of successful requests, so 1M successful requests happened").
The Second Issue:** During incidents, they needed more detail. They couldn't diagnose problems from 1% samples alone. Solution: dynamic sampling. In production, use low sampling (1%). When an error is detected, increase sampling for that specific transaction/user/service to 100% for the next 10 minutes. This gives you detail when you need it without collecting everything always.
Result:** By handling 500K transactions/day with selective logging, they maintained observability while keeping costs linear. When incidents occurred, they could increase sampling dynamically for targeted debugging.
Log Volume Management
Logs can become expensive (storage, analysis, infrastructure). You need to be selective about what to log:
Log this (100% sample):
- Errors and warnings (debugging crashes)
- Important business events (user signup, payment, refund, account deletion)
- Performance outliers (slow queries, API calls >1s, high error rate)
- Security events (failed logins, permission changes, suspicious activity)
Log this (sample rate 1-10%):
- Successful API requests (need some visibility but not all)
- Normal operations (helps with performance analysis)
- Cache hits/misses (understand cache efficiency)
Don't log this (0% sample):
- Every connection opened/closed (too noisy)
- Routine operations (request started, response sent)
- Sensitive data (passwords, credit cards, never log these)
- Debug info (temporary variables, internal state, unless investigating)
Cost math: Logging 1M events/day costs ~$100-500/month depending on retention and infrastructure. At 10M events/day, that's $1000-5000/month. At 100M/day, it's $10K-50K/month. Be intentional. Estimate log volume before implementing. If you'll log 100M events/day, that's probably $15K+/month in costs. Make sure the observability value justifies that spend.
Key Insight
Good observability comes from selective logging, effective parsing, and actionable analysis. Observe what matters, detect problems early, and fix them fast.
What to Do Monday Morning
- Audit your logging. In your system, what are you currently logging? What's the daily volume? What are you paying? Is it giving you useful debugging information?
- Identify information gaps. When was the last time you had an incident? Could AI analysis of logs have diagnosed it faster? What information was missing?
- Set baselines for anomaly detection. For error rate, latency (p50/p95/p99), database connection pool usage, cache hit ratio, memory usage, measure normal behavior for 1 week. Set alert thresholds at 2-3x normal.
- Design sampling strategy. What will you log at 100%? What at 10%? What at 1%? Estimate daily volume. Make sure costs are reasonable.
- Ask the AI to analyze recent logs. Feed your logs to Claude. "Analyze these logs. What patterns appear? What errors are common? What's worth investigating?"
- Set up structured logging. Logs should be parseable. Use JSON format or standard structured format (not free-form text). This makes automated analysis possible.
- Create incident response playbooks. When error rate spikes 10x, what do you check first? Check recent deployments? Check database performance? Document the process. Automate what you can.
FAQ
Q: How much log data should we keep?
A: Operational logs (errors, critical events): 30 days minimum. Audit logs (security): 1 year+. Debug logs: 7 days. Depends on compliance and analysis needs. Keep what you need to diagnose problems and comply with regulations. Delete what you don't need.
Q: Should logs be centralized?
A: Yes. Logs from all services in one place (using ELK, Datadog, etc.). This lets you correlate events across services. Without centralization, investigating incidents means checking 10 different systems.
Q: How do we prevent false positives in anomaly detection?
A: False positives happen when you alert on harmless deviations. Solutions: (1) Tune thresholds to reduce noise, (2) Learn seasonality (Monday looks different from Friday), (3) Require multiple signals before alerting (if both error rate AND latency spike, it's probably real), (4) Review and suppress known causes.
Q: Can we use AI to prevent incidents?
A: Partially. AI can detect early warning signs (error rate trending up, latency increasing slightly) and alert before the situation becomes critical. It's predictive alerting rather than reactive. This gives you time to investigate and fix before users see outages.
Q: How do we balance log verbosity with observability?
A: Log the important stuff always. Log detailed debug info only when investigating. Use log levels (ERROR, WARN, INFO, DEBUG) and sample strategically. In production, run on INFO level (errors, important events). When investigating, increase to DEBUG temporarily. Budget: you should spend 1-3% of your infrastructure cost on logging/observability. More than that and you're over-logging.
Q: How should we structure logs for AI analysis?
A: Use structured logging (JSON or key=value format). Include: timestamp (UTC, milliseconds), service name, log level (ERROR/WARN/INFO), user_id if relevant, transaction_id if relevant, error message, error type if it's an exception, stack trace if available. Avoid free-form text like "Something went wrong." Instead: "Database query timeout: query_id=abc, table=users, timeout_ms=5000".
Q: What's the right trade-off between sampling and observability?
A: Sample errors at 100% (see all failures). Sample successful operations at 1-5% (estimate totals from sample). This gives you complete visibility of failures with minimal cost. At 100M requests/day with 0.1% error rate, you have 100K errors (all logged) and 99.9M successful requests (sample 1% = 1M logged). Total: 1.1M log entries for 100M requests.