README.md (view raw)
1<h1 align="center">llm_aggregator</h1>
2
3<p align="center">
4 <strong>A CLI tool to aggregate RSS feeds and summarise them with LLMs</strong>
5</p>
6
7
8
9
10---
11
12## What is `llm_aggregator`?
13
14`llm_aggregator` is a command‑line utility that fetches articles from multiple
15RSS feeds, filters and processes the content, and sends it to an LLM through
16OpenAI-compatible API to generate a concise summary or analysis. It’s designed
17for keeping up with news and articles from your favourite sources without
18having to read dozens or hundreds of individual posts.
19
20## How do I use `llm_aggregator`?
21
22 llm_aggregator --feeds-file FEEDS-FILE --prompt PROMPT [OPTIONS]...
23
24By default, `llm_aggregator` reads a list of RSS feed URLs, fetches the
25articles, filters them by date and keywords, and sends a summary request to
26the LLM. The resulting output is printed to the terminal in your chosen format
27(text, markdown, or JSON).
28
29### Basic Options
30
31 --feeds-file FILE Path to file containing RSS feed URLs (one per line)
32 --prompt PROMPT User prompt for summarisation/analysis
33 --api-key KEY API key (default: read from $LLM_AGGREGATOR_API_KEY)
34 --model MODEL Model to use (default: deepseek-chat)
35 --base-url URL API base URL (default: https://api.deepseek.com)
36 --max-tokens N Maximum tokens in response (default: 4000)
37 --output FORMAT Output format: text, markdown, or json (default: text)
38 --output-file FILE Write output to FILE instead of stdout
39 --tui Enable TUI interface with progress bar
40 --verbose, -v Enable verbose logging
41 --help, -h Show this help message and exit
42 --version Show version information and exit
43
44### Filtering & Processing
45
46 --max-articles-per-feed N Maximum articles to fetch from each feed (default: 10)
47 --max-days-old N Only include articles from the last N days (0 for all) (default: 7)
48 --max-total-articles N Maximum total articles to process (default: 20)
49 --include-keywords LIST Comma-separated list of keywords to include (case‑insensitive)
50 --exclude-keywords LIST Comma-separated list of keywords to exclude (case‑insensitive)
51 --include-articles Include original articles in JSON output
52
53### LLM Configuration
54
55 --temperature VALUE Sampling temperature (0.0 to 1.0) (default: 0.7)
56 --system-prompt TEXT Custom system prompt for LLM
57
58### Examples
59
60```bash
61# Basic usage: summarise tech news from a list of feeds
62llm_aggregator --feeds-file feeds.txt \
63--prompt "What are the latest AI-related trends in free software?"
64
65# With TUI progress bar
66llm_aggregator --feeds-file feeds.txt --prompt "Summarise tech news" --tui
67
68# Output to a JSON file with included articles
69llm_aggregator --feeds-file feeds.txt --prompt "Analyse AI developments" \
70 --output json --output-file analysis.json --include-articles
71
72# Filter by keywords (only include articles about Linux or open source)
73llm_aggregator --feeds-file feeds.txt --prompt "Linux news" \
74 --include-keywords linux,opensource --max-days-old 3
75
76# Use a custom model and higher token limit
77llm_aggregator --feeds-file feeds.txt --prompt "Code analysis" \
78 --model deepseek-reasoner --max-tokens 8000
79
80# Use a custom API endpoint (e.g., local Ollama)
81llm_aggregator --feeds-file feeds.txt --prompt "Summarise news" \
82 --base-url "http://localhost:11434/v1" --model llama3
83
84# Show version information
85llm_aggregator --version
86
87# Show help message
88llm_aggregator --help
89```
90
91## How does `llm_aggregator` work?
92
93`llm_aggregator` performs the following steps for each run:
94
951. Parse command‑line arguments
962. Read the feeds file: a plain text file containing one RSS/Atom feed URL per
97 line.
983. **Fetch and parse feeds concurrently**: RSS, Atom, and JSON Feed formats are
99 supported. Feeds are fetched in parallel with rate limiting to maximise
100 throughput while avoiding server overload.
1014. **Extract article content**: for each feed entry, the tool extracts the
102 title, link, publication date, author, and description. If the feed provides
103 only a snippet, it can optionally fetch the full webpage using `goquery` to
104 extract the main content.
1055. **Filter and sort articles**: articles are filtered by age (configurable
106 with `--max-days-old`), optionally filtered by keywords (include/exclude),
107 and sorted by date, title, or source.
1086. Prepare the prompt with selected articles, formatted into a context
109 string that is sent to the LLM along with the user’s custom prompt.
1107. Call the OpenAI API via the `openai‑go` client.
1118. **Format and output the result**: the AI’s response is printed in the chosen
112 format (plain text, GitHub‑flavoured markdown, or JSON). If JSON output is
113 selected, the original articles can be included alongside the summary.
114
115When the `--tui` flag is used, the entire process is wrapped in a `bubbletea`
116TUI that shows a colourful progress bar, live article counters, and elapsed
117time. The TUI supports keyboard navigation (j/k, arrows, space, b, g/G) and
118mouse wheel scrolling for browsing long summaries.
119
120Feeds are fetched concurrently for optimal performance, with rate limiting to
121avoid overwhelming feed servers.
122
123## Configuration
124
125`llm_aggregator` supports multiple configuration sources with the following precedence order (highest to lowest):
126
1271. **Command‑line arguments** – Override everything
1282. **Environment variables** – Start with `LLM_AGGREGATOR_` prefix
1293. **Configuration file** – `~/.config/llm_aggregator/config.toml`
1304. **Built‑in defaults**
131
132### Configuration file
133
134Create a TOML file at `~/.config/llm_aggregator/config.toml` with the following structure:
135
136```toml
137# Feed aggregation options
138max_articles_per_feed = 10
139max_days_old = 7
140max_total_articles = 20
141
142# Content filtering (comma-separated keywords)
143# include_keywords = "linux,opensource"
144# exclude_keywords = "windows,microsoft"
145
146# LLM API options
147# api_key = "your_api_key_here" # Can also be set via LLM_AGGREGATOR_API_KEY env var
148# base_url = "https://api.deepseek.com" # Optional custom API endpoint
149model = "deepseek-chat"
150max_tokens = 4000
151temperature = 0.7
152
153# System prompt for LLM API
154system_prompt = """You are an expert analyst and summariser.
155You analyse content from multiple sources and provide
156concise, insightful summaries based on user requests.
157Focus on key points, trends, and important information."""
158
159# Output options
160output = "text" # Options: text, json, markdown
161# output_file = "" # Optional output file path
162include_articles = false
163```
164
165### Environment variables
166
167All configuration options can also be set via environment variables with the `LLM_AGGREGATOR_` prefix:
168
169| Variable | Description |
170|----------|-------------|
171| `LLM_AGGREGATOR_API_KEY` | LLM API key |
172| `LLM_AGGREGATOR_BASE_URL` | API base URL (default: "https://api.deepseek.com") |
173| `LLM_AGGREGATOR_MODEL` | Model name (default: "deepseek-chat") |
174| `LLM_AGGREGATOR_MAX_TOKENS` | Maximum tokens in response (default: 4000) |
175| `LLM_AGGREGATOR_TEMPERATURE` | Sampling temperature (default: 0.7) |
176| `LLM_AGGREGATOR_SYSTEM_PROMPT` | Custom system prompt |
177| `LLM_AGGREGATOR_MAX_ARTICLES_PER_FEED` | Maximum articles per feed (default: 10) |
178| `LLM_AGGREGATOR_MAX_DAYS_OLD` | Maximum article age in days (default: 7) |
179| `LLM_AGGREGATOR_MAX_TOTAL_ARTICLES` | Maximum total articles (default: 20) |
180| `LLM_AGGREGATOR_INCLUDE_KEYWORDS` | Comma‑separated include keywords |
181| `LLM_AGGREGATOR_EXCLUDE_KEYWORDS` | Comma‑separated exclude keywords |
182| `LLM_AGGREGATOR_OUTPUT` | Output format (default: "text") |
183| `LLM_AGGREGATOR_OUTPUT_FILE` | Output file path |
184| `LLM_AGGREGATOR_INCLUDE_ARTICLES` | Include articles in JSON output (true/false) |
185
186The API key can be provided via `--api‑key`, `LLM_AGGREGATOR_API_KEY` environment variable, or in the configuration file.
187
188## Example feeds file
189
190Create a file named `feeds.txt` with your favourite RSS feeds, one per line.
191For example:
192
193 https://news.ycombinator.com/rss
194 https://lwn.net/headlines/newrss
195 https://opensource.com/feed
196 https://www.phoronix.com/rss.php
197
198Then run:
199
200 llm_aggregator --feeds-file feeds.txt --prompt "Summarise the top tech stories"
201
202## Dependencies
203
204`llm_aggregator` is written in Go and uses the following libraries:
205
206| Library | Description |
207|---------|-------------|
208| [`gofeed`](https://github.com/mmcdole/gofeed) | Robust RSS/Atom/JSON feed parser |
209| [`openai‑go`](https://github.com/openai/openai-go) | Official OpenAI API library for Go |
210| [`bubbletea`](https://github.com/charmbracelet/bubbletea) | TUI framework for terminal applications |
211| [`lipgloss`](https://github.com/charmbracelet/lipgloss) | Library for styling terminal output |
212| [`go‑arg`](https://github.com/alexflint/go-arg) | Struct‑based argument parsing |
213| [`tiktoken-go`](https://github.com/pkoukk/tiktoken-go) | OpenAI's tiktoken BPE tokeniser |
214| [`viper`](https://github.com/spf13/viper) | Configuration management |
215| [`goquery`](https://github.com/PuerkitoBio/goquery) | jQuery‑like HTML scraping |
216
217## How do I build `llm_aggregator`?
218
219`llm_aggregator` can be built with a standard Go toolchain:
220
221 go build ./cmd/llm_aggregator.go
222
223For information about the project's test suite, see
224[docs/TESTING.md](docs/TESTING.md).
225
226## Licence
227
228This project is licensed under [European Union Public Licence
2291.2](https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12).