Agents and Skills

techhadoop ai

The fundamental distinction between Agents and Skills comes down to who drives the task (runtime) versus what knowledge or tooling is applied (capability).

Core Comparison

DimensionAI Agent (The “Actor”)AI Skill (The “Asset”)
Primary RoleThe autonomous decision-maker / runtime engine.Modular package of capability, domain knowledge, or tooling.
Question It Answers“Who is doing the task, and how do they reason?”“What step-by-step instructions or tools are needed?”
Control FlowDynamic loop: evaluates state, plans next steps, decides when to stop.Passive/Declarative: loaded on demand to perform a specific function.
ScopeGoal-oriented across multiple steps or tools.Task-bound (e.g., parse PDF, format JSON, query SQL).
ReusabilityLow to medium; typically bound to a specific workflow/persona.High; can be attached to any generalist agent on demand.

Key Concepts

1. Agent (Runtime & Reasoning)

An Agent is the orchestrator. It manages memory, maintains state across turns, plans sub-tasks, and evaluates its own progress.

  • Example: A “DevOps Engineer Agent” that monitors server health, investigates alerts, and decides whether to restart a service or escalate to a human.

2. Skill (Knowledge & Execution)

A Skill is a self-contained module containing domain guidelines, scripts, API calls, or instructions loaded dynamically when needed.Skills prevent you from overloading an Agent’s system prompt with information irrelevant to the current step.

  • Example:A docker-log-parser skill or a jira-ticket-creator skill.

How They Work Together

Rather than building a complex fleet of specialized agents (which can lead to “agent sprawl”), modern AI architectures favor using a generalist agent powered by a library of skills:

┌─────────────────────────────────────────────────────────┐
│                      AI AGENT                           │
│  (Maintains goal, plans steps, reasons over outputs)   │
└──────────────────────────┬──────────────────────────────┘
                           │
             Reads context & loads on demand
                           │
       ┌───────────────────┼───────────────────┐
       ▼                   ▼                   ▼
┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│  PDF Parser  │    │ SQL Query    │    │ Chart Visual │
│    Skill     │    │    Skill     │    │    Skill     │
└──────────────┘    └──────────────┘    └──────────────┘
  1. User Request: “Analyze this quarterly PDF report and update the SQL database.”
  2. Agent Action: Recognizes the plan requires two distinct tasks.
  3. Skill Loading: Dynamically pulls the pdf-parsing skill into its active context to extract text. Once complete, drops that context and loads the sql-database skill to run queries.

The Agent Skills standard (pioneered by Anthropic and adopted across tools like GitHub Copilot, Cursor, and Claude Code) relies on a standardized, file-based architecture centered around a SKILL.md manifest.

This format leverages progressive disclosure: the host system only loads the metadata (~100 tokens) at boot, and only reads the full SKILL.md instructions when a user request triggers the description.

1. Directory Blueprint

A production-grade skill is self-contained inside a directory named after the skill.

Plaintext

my-agent-skill/
├── SKILL.md              # REQUIRED: Metadata + Primary System Prompt
├── scripts/              # Executable helpers run by the agent (e.g. Python/Bash)
│   └── helper.py
├── references/           # Supplemental context (loaded by agent on demand)
│   └── api_docs.md
└── assets/               # Fixed files (e.g., templates, schemas)
    └── report_template.json

2. Anatomy of SKILL.md

SKILL.md is composed of two main parts: YAML Frontmatter (determines when the agent activates the skill) and the Markdown Body (tells the agent how to execute the task).

Markdown

---
name: sql-database-analyst
description: Generates, validates, and runs analytical SQL queries against PostgreSQL databases. Use when the user asks to query metrics, build reports, or inspect database schemas.
when_to_use:
  - User asks for SQL query generation or database investigation
  - User asks for key performance metrics (ARR, Churn, Active Users)
  - Do NOT use for: Data migration scripts or schema mutation (DDL) tasks
allowed-tools:
  - bash
  - read_file
  - execute_sql
arguments:
  - name: database_url
    description: Target PostgreSQL connection string
    required: false
---
# SQL Database Analyst Skill
## System Role & Instructions
You are an expert PostgreSQL DBA and Data Analyst. When assigned a querying task, follow this exact protocol:
1. **Schema Check:** Inspect the database tables using `references/schema_map.md` or by running `\dt` using the `execute_sql` tool.
2. **Query Draft:** Write standard ANSI-compliant PostgreSQL queries. Avoid non-standard extensions.
3. **Safety Check:** Ensure all queries are read-only (`SELECT`). Do not issue `UPDATE`, `INSERT`, `DELETE`, or `DROP`.
---
## Tool Execution Guide
If you need to calculate statistical distributions, execute the python calculation script rather than calculating manually:
```bash
python scripts/helper.py --input-data ./data.json

Output Format Requirements

Always format your response to the user with the following template:

Markdown

### Key Insights
- [Bullet points summarizing findings]
### SQL Query Used
```sql
[Generated Query]

---

## Common Edge Cases & Rules
- **Null Values:** Always use `COALESCE(column, 0)` for financial sums.
- **Timezones:** Convert all timestamp columns to `UTC` before aggregation.

3. Frontmatter Field Breakdown

FieldRequired?Purpose & Best Practices
nameYesLowercase identifier matching directory name. (Max 64 chars, no spaces/underscores).
descriptionYesThe most important field. Describes what the skill does and includes trigger phrases. The agent reads this to decide if the skill should fire.
when_to_useOptionalGranular activation criteria, explicit trigger phrases, and negative constraints (“Do NOT use for…”).
allowed-toolsOptionalRestricts which agent tools (e.g., bash, read_file) this skill is authorized to invoke.
argumentsOptionalDefines structured variables required or accepted by the skill.

4. How the Agent Loads and Runs the Skill

┌────────────────────────────────────────────────────────┐
│                   Agent App Startup                    │
│ Agent scans directory and reads frontmatter (~100 tokens)│
└──────────────────────────┬─────────────────────────────┘
                           │
             User prompt matches 'description'
                           │
                           ▼
┌────────────────────────────────────────────────────────┐
│                   Skill Activation                     │
│  Agent dynamically loads entire SKILL.md into context   │
└──────────────────────────┬─────────────────────────────┘
                           │
                  Needs deeper context?
                           │
                           ▼
┌────────────────────────────────────────────────────────┐
│                   Selective Execution                  │
│ Agent reads references/ or executes scripts/ as needed │
└────────────────────────────────────────────────────────┘
  1. Zero Context Overhead at Boot: System prompts stay slim because only the frontmatter description is loaded into memory.
  2. Context Window Protection: Detailed documentation or large JSON schemas are placed in /references or /assets so the agent only pulls them if a specific sub-task demands it.
  3. Execution, Not Context Inflation: If complex logic (like data parsing or API transformation) is required, place it inside a script in /scripts. The agent executes the script using terminal tools and only receives the output, preserving token limits.

Here is a complete, production-grade SKILL.md file designed specifically for an AI Agent performing Log Parsing, Normalization, and AI-Powered Anomaly Detection.

You can save this structure inside a project directory as .claude/skills/log-parser/SKILL.md or .agents/skills/log-parser/SKILL.md.

File Path: log-parser/SKILL.md

Markdown

---
name: log-parser
description: Auto-detects, parses, normalizes, and analyzes log files (JSON, Syslog, Apache/Nginx, Log4j, Stacktraces). Use when a user asks to parse raw logs, extract metrics, detect anomalies, or clean log streams before indexing.
when_to_use:
  - User provides raw unformatted log lines or stack traces
  - User asks to convert unstructured text logs into structured JSON/ECS format
  - User asks to extract top errors, IP addresses, or failure patterns from log dumps
  - Do NOT use for: Database performance queries (use `sql-database-analyst` skill)
allowed-tools:
  - bash
  - read_file
  - write_file
arguments:
  - name: log_format
    description: Known format (e.g., json, syslog, nginx, log4j, auto). Defaults to auto-detect.
    required: false
---
# Log Parsing & Anomaly Analysis Skill
## Role & Mission
You are a Principal Observability & Site Reliability Engineer (SRE). Your objective is to ingest raw log data, automatically identify the underlying log schema, extract key fields into Elastic Common Schema (ECS) standard key-value pairs, and report critical failure vectors.
---
## Step 1: Format Detection & Ingestion
Examine the first 10 lines of the provided log data to identify the schema format:
| Pattern Recognized | Target Format | Parser Strategy |
| :--- | :--- | :--- |
| `{"timestamp": ...}` | **Structured JSON** | Direct extraction |
| `<13>1 2026-10-08T...` | **RFC 5424 Syslog** | Regex pattern match |
| `127.0.0.1 - - [08/Oct/2026...]` | **Nginx / Apache Common** | Standard Web Log Regex |
| `2026-10-08 14:02:11,102 [main] ERROR` | **Log4j / Python Logging** | Multiline Grok extraction |
*Note: If multiline exception stack traces are detected (e.g., Python `Traceback` or Java `Exception in thread`), group all indented child lines into a single log record before parsing.*
---
## Step 2: Normalization Strategy (ECS Format)
Convert extracted raw attributes into standard **Elastic Common Schema (ECS)** field names:
* `@timestamp` $\rightarrow$ Standard ISO 8601 UTC timestamp string.
* `log.level` $\rightarrow$ Normalized string in lowercase (`debug`, `info`, `warn`, `error`, `critical`, `fatal`).
* `service.name` $\rightarrow$ Name of the service or application emitting the log.
* `host.ip` or `client.ip` $\rightarrow$ Extracted IPv4 / IPv6 addresses.
* `http.response.status_code` $\rightarrow$ Integer status codes (if web traffic).
* `message` $\rightarrow$ Clean textual message string stripped of timestamps and levels.
* `error.stack_trace` $\rightarrow$ Preserved full multiline exception payload (if present).
---
## Step 3: Execution Helpers
If processing large log dumps (over 500 lines), run the automated Python parser script located in this skill package rather than manually reading lines into context:
```bash
python scripts/parse_logs.py --input ./raw_logs.log --format auto --output ./parsed_logs.json

Step 4: Output Presentation Protocol

Always structure your analysis output to the user using the following layout:

Markdown

### Log Analysis Overview
- **Total Lines Parsed:** [Count]
- **Detected Format:** [JSON / Syslog / Grok / Unstructured]
- **Primary Severity Distribution:** `ERROR`: X | `WARN`: Y | `INFO`: Z
### Critical Anomalies & Root Cause
1. **[Error Pattern 1 Name]** (Occurred X times)
   - **First Seen:** [Timestamp]
   - **Sample Log:** `[Raw snippet]`
   - **AI Analysis:** [1-sentence explanation of what failed and how to remediate]
### Normalized Sample (ECS JSON Format)
```json
[
  {
    "@timestamp": "2026-10-08T14:02:11Z",
    "log.level": "error",
    "service.name": "user-auth-service",
    "message": "Connection back-off failure, system.db.Pool empty.",
    "error": {
      "code": "DB_POOL_EMPTY"
    }
  }
]

---

## Safety Constraints & Rules
* **Data Anonymization:** Automatically redact sensitive tokens, authorization headers (`Bearer eyJ...`), passwords, and PII (Social Security numbers, Credit Card numbers) replaced with `[REDACTED]`.
* **Timestamp Handling:** If no year is specified in the log, assume the current system year (`2026`). Always convert relative timestamps into standard UTC.

Complementary Directory Structure

To deploy this skill completely, place it in the agent directory alongside its helper scripts:

Plaintext

log-parser/
├── SKILL.md
├── scripts/
│   └── parse_logs.py      # Python script that executes fast regex/grok batch processing
└── references/
    └── ecs_cheatsheet.md  # Standard Elastic Common Schema field reference guide

Leave a Reply