README.md (view raw)
1<h1 align="center">llm_aggregator</h1>
2
3<p align="center">
4 <img src="assets/logo.svg" alt="LLM Aggregator logo" width=500>
5 <br>
6 <strong>A CLI tool to aggregate RSS feeds and summarise them with LLMs</strong>
7</p>
8
9
10
11
12---
13
14## What is `llm_aggregator`?
15
16`llm_aggregator` is a command‑line utility that fetches articles from multiple
17RSS feeds, filters and processes the content, and sends it to an LLM through
18OpenAI-compatible API to generate a concise summary or analysis. It’s designed
19for keeping up with news and articles from your favourite sources without
20having to read dozens or hundreds of individual posts.
21
22## How do I use `llm_aggregator`?
23
24 llm_aggregator --feeds-file FEEDS-FILE --prompt PROMPT [OPTIONS]...
25
26By default, `llm_aggregator` reads a list of RSS feed URLs, fetches the
27articles, filters them by date and keywords, and sends a summary request to
28the LLM. The resulting output is printed to the terminal in your chosen format
29(text, markdown, or JSON).
30
31### Basic Options
32
33 --feeds-file FILE Path to file containing RSS feed URLs (one per line)
34 --prompt PROMPT User prompt for summarisation/analysis
35 --api-key KEY API key (default: read from $LLM_AGGREGATOR_API_KEY)
36 --model MODEL Model to use (default: deepseek-chat)
37 --dry-run Validate config, show article statistics, and exit without making LLM API calls.
38 --base-url URL API base URL (default: https://api.deepseek.com)
39 --max-tokens N Maximum tokens in response (default: 4000)
40 --output FORMAT Output format: text, markdown, or json (default: text)
41 --output-file FILE Write output to FILE instead of stdout
42 --tui Enable TUI interface with progress bar
43 --verbose, -v Enable verbose logging
44 --help, -h Show this help message and exit
45 --version Show version information and exit
46
47### Filtering & Processing
48
49 --max-articles-per-feed N Maximum articles to fetch from each feed (default: 10)
50 --max-days-old N Only include articles from the last N days (0 for all) (default: 7)
51 --max-total-articles N Maximum total articles to process (default: 20)
52 --include-keywords LIST Comma-separated list of keywords to include (case‑insensitive)
53 --exclude-keywords LIST Comma-separated list of keywords to exclude (case‑insensitive)
54 --include-articles Include original articles in JSON output
55
56### LLM Configuration
57
58 --temperature VALUE Sampling temperature (0.0 to 1.0) (default: 0.7)
59 --system-prompt TEXT Custom system prompt for LLM
60
61<div align="center">
62 <img src="assets/demo.svg" alt="LLM Aggregator in action (SVG animation)" width=900>
63</div>
64
65### Examples
66
67```bash
68# Basic usage: summarise tech news from a list of feeds
69llm_aggregator --feeds-file feeds.txt \
70--prompt "What are the latest AI-related trends in free software?"
71
72# With TUI progress bar
73llm_aggregator --feeds-file feeds.txt --prompt "Summarise tech news" --tui
74
75# Output to a JSON file with included articles
76llm_aggregator --feeds-file feeds.txt --prompt "Analyse AI developments" \
77 --output json --output-file analysis.json --include-articles
78
79# Filter by keywords (only include articles about Linux or open source)
80llm_aggregator --feeds-file feeds.txt --prompt "Linux news" \
81 --include-keywords linux,opensource --max-days-old 3
82
83# Use a custom model and higher token limit
84llm_aggregator --feeds-file feeds.txt --prompt "Code analysis" \
85 --model deepseek-reasoner --max-tokens 8000
86
87# Use a custom API endpoint (e.g., local Ollama)
88llm_aggregator --feeds-file feeds.txt --prompt "Summarise news" \
89 --base-url "http://localhost:11434/v1" --model llama3
90
91# Show version information
92llm_aggregator --version
93
94# Show help message
95llm_aggregator --help
96```
97
98## How does `llm_aggregator` work?
99
100`llm_aggregator` performs the following steps for each run:
101
1021. Parse command‑line arguments
1032. Read the feeds file: a plain text file containing one RSS/Atom feed URL per
104 line.
1053. **Fetch and parse feeds concurrently**: RSS, Atom, and JSON Feed formats are
106 supported. Feeds are fetched in parallel with rate limiting to maximise
107 throughput while avoiding server overload.
1084. **Extract article content**: for each feed entry, the tool extracts the
109 title, link, publication date, author, and description. If the feed provides
110 only a snippet, it can optionally fetch the full webpage using `goquery` to
111 extract the main content.
1125. **Filter and sort articles**: articles are filtered by age (configurable
113 with `--max-days-old`), optionally filtered by keywords (include/exclude),
114 and sorted by date, title, or source.
1156. Prepare the prompt with selected articles, formatted into a context
116 string that is sent to the LLM along with the user’s custom prompt.
1177. Call the OpenAI API via the `openai‑go` client.
1188. **Format and output the result**: the AI’s response is printed in the chosen
119 format (plain text, GitHub‑flavoured markdown, or JSON). If JSON output is
120 selected, the original articles can be included alongside the summary.
121
122When the `--tui` flag is used, the entire process is wrapped in a `bubbletea`
123TUI that shows a colourful progress bar, live article counters, and elapsed
124time. The TUI supports keyboard navigation (j/k, arrows, space, b, g/G) and
125mouse wheel scrolling for browsing long summaries.
126
127Feeds are fetched concurrently for optimal performance, with rate limiting to
128avoid overwhelming feed servers.
129
130## Configuration
131
132`llm_aggregator` supports multiple configuration sources with the following precedence order (highest to lowest):
133
1341. **Command‑line arguments** – Override everything
1352. **Environment variables** – Start with `LLM_AGGREGATOR_` prefix
1363. **Configuration file** – `~/.config/llm_aggregator/config.toml`
1374. **Built‑in defaults**
138
139### Configuration file
140
141Create a TOML file at `~/.config/llm_aggregator/config.toml` with the following structure:
142
143```toml
144# Feed aggregation options
145max_articles_per_feed = 10
146max_days_old = 7
147max_total_articles = 20
148
149# Content filtering (comma-separated keywords)
150# include_keywords = "linux,opensource"
151# exclude_keywords = "windows,microsoft"
152
153# LLM API options
154# api_key = "your_api_key_here" # Can also be set via LLM_AGGREGATOR_API_KEY env var
155# base_url = "https://api.deepseek.com" # Optional custom API endpoint
156model = "deepseek-chat"
157max_tokens = 4000
158temperature = 0.7
159
160# System prompt for LLM API
161system_prompt = """You are an expert analyst and summariser.
162You analyse content from multiple sources and provide
163concise, insightful summaries based on user requests.
164Focus on key points, trends, and important information."""
165
166# Output options
167output = "text" # Options: text, json, markdown
168# output_file = "" # Optional output file path
169include_articles = false
170```
171
172### Environment variables
173
174All configuration options can also be set via environment variables with the `LLM_AGGREGATOR_` prefix:
175
176| Variable | Description |
177|----------|-------------|
178| `LLM_AGGREGATOR_API_KEY` | LLM API key |
179| `LLM_AGGREGATOR_BASE_URL` | API base URL (default: "https://api.deepseek.com") |
180| `LLM_AGGREGATOR_MODEL` | Model name (default: "deepseek-chat") |
181| `LLM_AGGREGATOR_MAX_TOKENS` | Maximum tokens in response (default: 4000) |
182| `LLM_AGGREGATOR_TEMPERATURE` | Sampling temperature (default: 0.7) |
183| `LLM_AGGREGATOR_SYSTEM_PROMPT` | Custom system prompt |
184| `LLM_AGGREGATOR_MAX_ARTICLES_PER_FEED` | Maximum articles per feed (default: 10) |
185| `LLM_AGGREGATOR_MAX_DAYS_OLD` | Maximum article age in days (default: 7) |
186| `LLM_AGGREGATOR_MAX_TOTAL_ARTICLES` | Maximum total articles (default: 20) |
187| `LLM_AGGREGATOR_INCLUDE_KEYWORDS` | Comma‑separated include keywords |
188| `LLM_AGGREGATOR_EXCLUDE_KEYWORDS` | Comma‑separated exclude keywords |
189| `LLM_AGGREGATOR_OUTPUT` | Output format (default: "text") |
190| `LLM_AGGREGATOR_OUTPUT_FILE` | Output file path |
191| `LLM_AGGREGATOR_INCLUDE_ARTICLES` | Include articles in JSON output (true/false) |
192
193The API key can be provided via `--api‑key`, `LLM_AGGREGATOR_API_KEY` environment variable, or in the configuration file.
194
195## Example feeds file
196
197Create a file named `feeds.txt` with your favourite RSS feeds, one per line.
198For example:
199
200 https://news.ycombinator.com/rss
201 https://lwn.net/headlines/newrss
202 https://opensource.com/feed
203 https://www.phoronix.com/rss.php
204
205Then run:
206
207 llm_aggregator --feeds-file feeds.txt --prompt "Summarise the top tech stories"
208
209## Dependencies
210
211`llm_aggregator` is written in Go and uses the following libraries:
212
213| Library | Description |
214|---------|-------------|
215| [`gofeed`](https://github.com/mmcdole/gofeed) | Robust RSS/Atom/JSON feed parser |
216| [`openai‑go`](https://github.com/openai/openai-go) | Official OpenAI API library for Go |
217| [`bubbletea`](https://github.com/charmbracelet/bubbletea) | TUI framework for terminal applications |
218| [`lipgloss`](https://github.com/charmbracelet/lipgloss) | Library for styling terminal output |
219| [`go‑arg`](https://github.com/alexflint/go-arg) | Struct‑based argument parsing |
220| [`tiktoken-go`](https://github.com/pkoukk/tiktoken-go) | OpenAI's tiktoken BPE tokeniser |
221| [`viper`](https://github.com/spf13/viper) | Configuration management |
222| [`goquery`](https://github.com/PuerkitoBio/goquery) | jQuery‑like HTML scraping |
223
224## How do I build `llm_aggregator`?
225
226`llm_aggregator` can be built with a standard Go toolchain:
227
228 go build ./cmd/llm_aggregator.go
229
230For information about the project's test suite, see
231[docs/TESTING.md](docs/TESTING.md).
232
233## Licence
234
235This project is licensed under [European Union Public Licence
2361.2](https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12).