README.md (view raw)
1<h1 align="center">llm_aggregator</h1>
2
3<p align="center">
4 <img src="assets/logo.svg" alt="LLM Aggregator logo" width=500>
5 <br>
6 <strong>A CLI tool to aggregate RSS feeds and summarise them with LLMs</strong>
7</p>
8
9
10
11
12---
13
14## What is `llm_aggregator`?
15
16`llm_aggregator` is a command‑line utility that fetches articles from multiple
17RSS feeds, filters and processes the content, and sends it to an LLM through
18OpenAI-compatible API to generate a concise summary or analysis. It’s designed
19for keeping up with news and articles from your favourite sources without
20having to read dozens or hundreds of individual posts.
21
22## How do I use `llm_aggregator`?
23
24 llm_aggregator --feeds-file FEEDS-FILE --prompt PROMPT [OPTIONS]...
25
26By default, `llm_aggregator` reads a list of RSS feed URLs, fetches the
27articles, filters them by date and keywords, and sends a summary request to
28the LLM. The resulting output is printed to the terminal in your chosen format
29(text, markdown, or JSON).
30
31### Basic Options
32
33 -f, --feeds-file FILE Path to file containing RSS feed URLs (one per line) [required]
34 -p, --prompt PROMPT User prompt for summarisation/analysis [required]
35 --api-key KEY API key (default: read from $LLM_AGGREGATOR_API_KEY)
36 -m, --model MODEL Model to use (default: deepseek-chat)
37 --base-url URL API base URL (default: https://api.deepseek.com)
38 --max-tokens N Maximum tokens in response (default: 4000)
39 -P, --plain Output only the raw LLM response without formatting or metadata
40 -o, --output FORMAT Output format: text, markdown, or json (default: text)
41 --output-file FILE Write output to FILE instead of STDOUT
42 -t, --tui Enable TUI interface with progress bar
43 -D, --dry-run Validate config, show article statistics, and exit without making LLM API calls
44 -v, --verbose Enable verbose logging
45 -h, --help Show this help message and exit
46 --version Show version information and exit
47
48### Filtering & Processing
49
50 -n, --max-articles-per-feed N Maximum articles to fetch from each feed (default: 10)
51 -d, --max-days-old N Only include articles from the last N days (0 for all) (default: 7)
52 --max-total-articles N Maximum total articles to process (default: 20)
53 -i, --include-keywords LIST Comma-separated list of keywords to include (case‑insensitive)
54 -e, --exclude-keywords LIST Comma-separated list of keywords to exclude (case‑insensitive)
55 --include-articles Include original articles in JSON output
56
57### LLM Configuration
58
59 --temperature VALUE Sampling temperature (0.0 to 1.0) (default: 0.7)
60 --system-prompt TEXT Custom system prompt for LLM
61
62### Examples
63
64```bash
65# Basic usage: summarise tech news from a list of feeds
66llm_aggregator -f feeds.txt -p "What are the latest AI-related trends in free software?"
67
68# With TUI progress bar
69llm_aggregator -f feeds.txt -p "Summarise tech news" -t
70
71# Output to a JSON file with included articles
72llm_aggregator -f feeds.txt -p "Analyse AI developments" \
73 -o json --output-file analysis.json --include-articles
74
75
76# Filter by keywords (only include articles about Linux or open source)
77llm_aggregator -f feeds.txt -p "Linux news" \
78 -i linux,opensource -d 3
79
80# Use a custom model and higher token limit
81llm_aggregator -f feeds.txt -p "Code analysis" \
82 -m deepseek-reasoner --max-tokens 8000
83
84# Use a custom API endpoint (e.g., local Ollama)
85llm_aggregator -f feeds.txt -p "Summarise news" \
86 --base-url "http://localhost:11434/v1" -m llama3
87
88# Show version information
89llm_aggregator --version
90
91# Show help message
92llm_aggregator --help
93```
94
95## How does `llm_aggregator` work?
96
97`llm_aggregator` performs the following steps for each run:
98
991. Parse command‑line arguments
1002. Read the feeds file: a plain text file containing one RSS/Atom feed URL per
101 line.
1023. **Fetch and parse feeds concurrently**: RSS, Atom, and JSON Feed formats are
103 supported. Feeds are fetched in parallel with rate limiting to maximise
104 throughput while avoiding server overload.
1054. **Extract article content**: for each feed entry, the tool extracts the
106 title, link, publication date, author, and description. If the feed provides
107 only a snippet, it can optionally fetch the full webpage using `goquery` to
108 extract the main content.
1095. **Filter and sort articles**: articles are filtered by age (configurable
110 with `--max-days-old`), optionally filtered by keywords (include/exclude),
111 and sorted by date, title, or source.
1126. Prepare the prompt with selected articles, formatted into a context
113 string that is sent to the LLM along with the user’s custom prompt.
1147. Call the OpenAI API via the `openai‑go` client.
1158. **Format and output the result**: the AI’s response is printed in the chosen
116 format (plain text, GitHub‑flavoured markdown, or JSON). If JSON output is
117 selected, the original articles can be included alongside the summary.
118
119
120
121When the `--tui` flag is used, the entire process is wrapped in a `bubbletea`
122TUI that shows a colourful progress bar, live article counters, and elapsed
123time. The TUI renders Markdown content from the LLM with proper styling for
124headers, bold, italic, code blocks, and lists. The TUI supports keyboard
125navigation (j/k, arrows, space, b, g/G) and mouse wheel scrolling for
126browsing long summaries.
127
128Feeds are fetched concurrently for optimal performance, with rate limiting to
129avoid overwhelming feed servers.
130
131## Configuration
132
133`llm_aggregator` supports multiple configuration sources with the following precedence order (highest to lowest):
134
1351. **Command‑line arguments** – Override everything
1362. **Environment variables** – Start with `LLM_AGGREGATOR_` prefix
1373. **Configuration file** – `~/.config/llm_aggregator/config.toml`
1384. **Built‑in defaults**
139
140### Configuration file
141
142Create a TOML file at `~/.config/llm_aggregator/config.toml` with the following structure:
143
144```toml
145# Feed aggregation options
146max_articles_per_feed = 10
147max_days_old = 7
148max_total_articles = 20
149
150# Content filtering (comma-separated keywords)
151# include_keywords = "linux,opensource"
152# exclude_keywords = "windows,microsoft"
153
154# LLM API options
155# api_key = "your_api_key_here" # Can also be set via LLM_AGGREGATOR_API_KEY env var
156# base_url = "https://api.deepseek.com" # Optional custom API endpoint
157model = "deepseek-chat"
158max_tokens = 4000
159temperature = 0.7
160
161# System prompt for LLM API
162system_prompt = """You are an expert analyst and summariser.
163You analyse content from multiple sources and provide
164concise, insightful summaries based on user requests.
165Focus on key points, trends, and important information."""
166
167# Output options
168output = "text" # Options: text, json, markdown
169# output_file = "" # Optional output file path
170include_articles = false
171```
172
173### Environment variables
174
175All configuration options can also be set via environment variables with the `LLM_AGGREGATOR_` prefix:
176
177| Variable | Description |
178|----------|-------------|
179| `LLM_AGGREGATOR_API_KEY` | LLM API key |
180| `LLM_AGGREGATOR_BASE_URL` | API base URL (default: "https://api.deepseek.com") |
181| `LLM_AGGREGATOR_MODEL` | Model name (default: "deepseek-chat") |
182| `LLM_AGGREGATOR_MAX_TOKENS` | Maximum tokens in response (default: 4000) |
183| `LLM_AGGREGATOR_TEMPERATURE` | Sampling temperature (default: 0.7) |
184| `LLM_AGGREGATOR_SYSTEM_PROMPT` | Custom system prompt |
185| `LLM_AGGREGATOR_MAX_ARTICLES_PER_FEED` | Maximum articles per feed (default: 10) |
186| `LLM_AGGREGATOR_MAX_DAYS_OLD` | Maximum article age in days (default: 7) |
187| `LLM_AGGREGATOR_MAX_TOTAL_ARTICLES` | Maximum total articles (default: 20) |
188| `LLM_AGGREGATOR_INCLUDE_KEYWORDS` | Comma‑separated include keywords |
189| `LLM_AGGREGATOR_EXCLUDE_KEYWORDS` | Comma‑separated exclude keywords |
190| `LLM_AGGREGATOR_OUTPUT` | Output format (default: "text") |
191| `LLM_AGGREGATOR_OUTPUT_FILE` | Output file path |
192| `LLM_AGGREGATOR_INCLUDE_ARTICLES` | Include articles in JSON output (true/false) |
193| `LLM_AGGREGATOR_PLAIN` | Plain output without metadata (true/false) |
194
195The API key can be provided via `--api‑key`, `LLM_AGGREGATOR_API_KEY` environment variable, or in the configuration file.
196
197## Example feeds file
198
199Create a file named `feeds.txt` with your favourite RSS feeds, one per line.
200For example:
201
202 https://news.ycombinator.com/rss
203 https://lwn.net/headlines/newrss
204 https://opensource.com/feed
205 https://www.phoronix.com/rss.php
206
207Then run:
208
209 llm_aggregator -f feeds.txt -p "Summarise the top tech stories"
210
211## Dependencies
212
213`llm_aggregator` is written in Go and uses the following libraries:
214
215| Library | Description |
216|---------|-------------|
217| [`gofeed`](https://github.com/mmcdole/gofeed) | Robust RSS/Atom/JSON feed parser |
218| [`openai‑go`](https://github.com/openai/openai-go) | Official OpenAI API library for Go |
219| [`bubbletea`](https://github.com/charmbracelet/bubbletea) | TUI framework for terminal applications |
220| [`lipgloss`](https://github.com/charmbracelet/lipgloss) | Library for styling terminal output |
221| [`glamour`](https://github.com/charmbracelet/glamour) | Markdown rendering for terminal (used in TUI mode) |
222| [`go‑arg`](https://github.com/alexflint/go-arg) | Struct‑based argument parsing |
223| [`tiktoken-go`](https://github.com/pkoukk/tiktoken-go) | OpenAI's tiktoken BPE tokeniser |
224| [`viper`](https://github.com/spf13/viper) | Configuration management |
225| [`goquery`](https://github.com/PuerkitoBio/goquery) | jQuery‑like HTML scraping |
226
227## How do I build `llm_aggregator`?
228
229`llm_aggregator` can be built with a standard Go toolchain:
230
231 go build ./cmd/llm_aggregator.go
232
233For information about the project's test suite, see
234[docs/TESTING.md](docs/TESTING.md).
235
236## Licence
237
238This project is licensed under [European Union Public Licence
2391.2](https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12).