1<h1 align="center">llm_aggregator</h1>
  2
  3<p align="center">
  4    <img src="assets/logo.svg" alt="LLM Aggregator logo" width=500>
  5    <br>
  6    <strong>A CLI tool to aggregate RSS feeds and summarise them with LLMs</strong>
  7</p>
  8
  9![Codeberg Release](https://img.shields.io/gitea/v/release/maxwelljensen/llm_aggregator?gitea_url=https%3A%2F%2Fcodeberg.org&style=for-the-badge)
 10![Codeberg License](assets/eupl-12-badge.svg)
 11
 12---
 13
 14## What is `llm_aggregator`?
 15
 16`llm_aggregator` is a command‑line utility that fetches articles from multiple
 17RSS feeds, filters and processes the content, and sends it to an LLM through
 18OpenAI-compatible API to generate a concise summary or analysis. It’s designed
 19for keeping up with news and articles from your favourite sources without
 20having to read dozens or hundreds of individual posts.
 21
 22## How do I use `llm_aggregator`?
 23
 24    llm_aggregator --feeds-file FEEDS-FILE --prompt PROMPT [OPTIONS]...
 25
 26By default, `llm_aggregator` reads a list of RSS feed URLs, fetches the
 27articles, filters them by date and keywords, and sends a summary request to
 28the LLM. The resulting output is printed to the terminal in your chosen format
 29(text, markdown, or JSON).
 30
 31### Basic Options
 32
 33    -f, --feeds-file FILE    Path to file containing RSS feed URLs (one per line)
 34        --stdin              Read a single RSS/Atom feed from stdin (combinable with -f)
 35    -p, --prompt PROMPT      User prompt for summarisation/analysis                 [required]
 36    --api-key KEY            API key (default: read from $LLM_AGGREGATOR_API_KEY)
 37    -m, --model MODEL        Model to use (default: deepseek-chat)
 38    --base-url URL           API base URL (default: https://api.deepseek.com)
 39    --max-tokens N           Maximum tokens in response (default: 4000)
 40    -P, --plain              Output only the raw LLM response without formatting or metadata
 41    -o, --output FORMAT      Output format: text, markdown, or json (default: text)
 42    --output-file FILE       Write output to FILE instead of STDOUT
 43    -t, --tui                Enable TUI interface with progress bar
 44    -D, --dry-run            Validate config, show article statistics, and exit without making LLM API calls
 45    -v, --verbose            Enable verbose logging
 46    -h, --help               Show this help message and exit
 47    --version                Show version information and exit
 48
 49### Filtering & Processing
 50
 51    -n, --max-articles-per-feed N  Maximum articles to fetch from each feed (default: 10)
 52    -d, --max-days-old N           Only include articles from the last N days (0 for all) (default: 7)
 53    --max-total-articles N         Maximum total articles to process (default: 20)
 54    -i, --include-keywords LIST    Comma-separated list of keywords to include (case‑insensitive)
 55    -e, --exclude-keywords LIST    Comma-separated list of keywords to exclude (case‑insensitive)
 56    --include-articles             Include original articles in JSON output
 57
 58### LLM Configuration
 59
 60    --temperature VALUE       Sampling temperature (0.0 to 1.0) (default: 0.7)
 61    --timeout N               LLM request timeout in seconds (default: 300)
 62    --system-prompt TEXT      Custom system prompt for LLM
 63
 64### Examples
 65
 66```bash
 67# Basic usage: summarise tech news from a list of feeds
 68llm_aggregator -f feeds.txt -p "What are the latest AI-related trends in free software?"
 69
 70# Read RSS feed directly from stdin
 71curl -s https://example.com/feed.xml | llm_aggregator --stdin -p "Summarise"
 72
 73# Combine feeds file with stdin feed
 74llm_aggregator -f feeds.txt --stdin -p "Summarise all"
 75
 76# With TUI progress bar
 77llm_aggregator -f feeds.txt -p "Summarise tech news" -t
 78
 79# Output to a JSON file with included articles
 80llm_aggregator -f feeds.txt -p "Analyse AI developments" \
 81    -o json --output-file analysis.json --include-articles
 82
 83
 84# Filter by keywords (only include articles about Linux or open source)
 85llm_aggregator -f feeds.txt -p "Linux news" \
 86    -i linux,opensource -d 3
 87
 88# Use a custom model and higher token limit
 89llm_aggregator -f feeds.txt -p "Code analysis" \
 90    -m deepseek-reasoner --max-tokens 8000
 91
 92# Use a custom API endpoint (e.g., local Ollama)
 93llm_aggregator -f feeds.txt -p "Summarise news" \
 94    --base-url "http://localhost:11434/v1" -m llama3
 95
 96# Show version information
 97llm_aggregator --version
 98
 99# Show help message
100llm_aggregator --help
101```
102
103## How does `llm_aggregator` work?
104
105`llm_aggregator` performs the following steps for each run:
106
1071. Parse command‑line arguments
1082. Read the feeds file: a plain text file containing one RSS/Atom feed URL per
109   line.
1103. **Fetch and parse feeds concurrently**: RSS, Atom, and JSON Feed formats are
111   supported. Feeds are fetched in parallel with rate limiting to maximise
112   throughput while avoiding server overload.
1134. **Extract article content**: for each feed entry, the tool extracts the
114   title, link, publication date, author, and description. If the feed provides
115   only a snippet, it can optionally fetch the full webpage using `goquery` to
116   extract the main content.
1175. **Filter and sort articles**: articles are filtered by age (configurable
118   with `--max-days-old`), optionally filtered by keywords (include/exclude),
119   and sorted by date, title, or source.
1206. Prepare the prompt with selected articles, formatted into a context
121   string that is sent to the LLM along with the user’s custom prompt.
1227. Call the OpenAI API via the `openai‑go` client.
1238. **Format and output the result**: the AI’s response is printed in the chosen
124   format (plain text, GitHub‑flavoured markdown, or JSON). If JSON output is
125   selected, the original articles can be included alongside the summary.
126
127![LLM Aggregator action in GIF](./assets/demo.gif)
128
129When the `--tui` flag is used, the entire process is wrapped in a `bubbletea`
130TUI that shows a colourful progress bar, live article counters, and elapsed
131time. The TUI renders Markdown content from the LLM with proper styling for
132headers, bold, italic, code blocks, and lists. The TUI supports keyboard
133navigation (j/k, arrows, space, b, g/G) and mouse wheel scrolling for
134browsing long summaries.
135
136Feeds are fetched concurrently for optimal performance, with rate limiting to
137avoid overwhelming feed servers.
138
139## Configuration
140
141`llm_aggregator` supports multiple configuration sources with the following precedence order (highest to lowest):
142
1431. **Command‑line arguments** – Override everything
1442. **Environment variables** – Start with `LLM_AGGREGATOR_` prefix
1453. **Configuration file** – `~/.config/llm_aggregator/config.toml`
1464. **Built‑in defaults**
147
148### Configuration file
149
150Create a TOML file at `~/.config/llm_aggregator/config.toml` with the following structure:
151
152```toml
153# Feed aggregation options
154max_articles_per_feed = 10
155max_days_old = 7
156max_total_articles = 20
157
158# Content filtering (comma-separated keywords)
159# include_keywords = "linux,opensource"
160# exclude_keywords = "windows,microsoft"
161
162# LLM API options
163# api_key = "your_api_key_here"  # Can also be set via LLM_AGGREGATOR_API_KEY env var
164# base_url = "https://api.deepseek.com"  # Optional custom API endpoint
165model = "deepseek-chat"
166max_tokens = 4000
167temperature = 0.7
168
169# System prompt for LLM API
170system_prompt = """You are an expert analyst and summariser.
171You analyse content from multiple sources and provide
172concise, insightful summaries based on user requests.
173Focus on key points, trends, and important information."""
174
175# Output options
176output = "text"  # Options: text, json, markdown
177# output_file = ""  # Optional output file path
178include_articles = false
179```
180
181### Environment variables
182
183All configuration options can also be set via environment variables with the `LLM_AGGREGATOR_` prefix:
184
185| Variable | Description |
186|----------|-------------|
187| `LLM_AGGREGATOR_API_KEY` | LLM API key |
188| `LLM_AGGREGATOR_BASE_URL` | API base URL (default: "https://api.deepseek.com") |
189| `LLM_AGGREGATOR_MODEL` | Model name (default: "deepseek-chat") |
190| `LLM_AGGREGATOR_MAX_TOKENS` | Maximum tokens in response (default: 4000) |
191| `LLM_AGGREGATOR_TEMPERATURE` | Sampling temperature (default: 0.7) |
192| `LLM_AGGREGATOR_TIMEOUT` | LLM request timeout in seconds (default: 300) |
193| `LLM_AGGREGATOR_SYSTEM_PROMPT` | Custom system prompt |
194| `LLM_AGGREGATOR_MAX_ARTICLES_PER_FEED` | Maximum articles per feed (default: 10) |
195| `LLM_AGGREGATOR_MAX_DAYS_OLD` | Maximum article age in days (default: 7) |
196| `LLM_AGGREGATOR_MAX_TOTAL_ARTICLES` | Maximum total articles (default: 20) |
197| `LLM_AGGREGATOR_INCLUDE_KEYWORDS` | Comma‑separated include keywords |
198| `LLM_AGGREGATOR_EXCLUDE_KEYWORDS` | Comma‑separated exclude keywords |
199| `LLM_AGGREGATOR_OUTPUT` | Output format (default: "text") |
200| `LLM_AGGREGATOR_OUTPUT_FILE` | Output file path |
201| `LLM_AGGREGATOR_INCLUDE_ARTICLES` | Include articles in JSON output (true/false) |
202| `LLM_AGGREGATOR_PLAIN` | Plain output without metadata (true/false) |
203| `LLM_AGGREGATOR_STDIN` | Read RSS/Atom feed from stdin (true/false) |
204
205The API key can be provided via `--api‑key`, `LLM_AGGREGATOR_API_KEY` environment variable, or in the configuration file.
206
207## Example feeds file
208
209Create a file named `feeds.txt` with your favourite RSS feeds, one per line.
210For example:
211
212    https://news.ycombinator.com/rss
213    https://lwn.net/headlines/newrss
214    https://opensource.com/feed
215    https://www.phoronix.com/rss.php
216
217Then run:
218
219    llm_aggregator -f feeds.txt -p "Summarise the top tech stories"
220
221## Dependencies
222
223`llm_aggregator` is written in Go and uses the following libraries:
224
225| Library | Description |
226|---------|-------------|
227| [`gofeed`](https://github.com/mmcdole/gofeed) | Robust RSS/Atom/JSON feed parser |
228| [`openai‑go`](https://github.com/openai/openai-go) | Official OpenAI API library for Go |
229| [`bubbletea`](https://github.com/charmbracelet/bubbletea) | TUI framework for terminal applications |
230| [`lipgloss`](https://github.com/charmbracelet/lipgloss) | Library for styling terminal output |
231| [`glamour`](https://github.com/charmbracelet/glamour) | Markdown rendering for terminal (used in TUI mode) |
232| [`go‑arg`](https://github.com/alexflint/go-arg) | Struct‑based argument parsing |
233| [`tiktoken-go`](https://github.com/pkoukk/tiktoken-go) | OpenAI's tiktoken BPE tokeniser |
234| [`viper`](https://github.com/spf13/viper) | Configuration management |
235| [`goquery`](https://github.com/PuerkitoBio/goquery) | jQuery‑like HTML scraping |
236
237## How do I build `llm_aggregator`?
238
239`llm_aggregator` can be built with a standard Go toolchain:
240
241    go build ./cmd/llm_aggregator.go
242
243For information about the project's test suite, see
244[docs/TESTING.md](docs/TESTING.md).
245
246## Licence
247
248This project is licensed under [European Union Public Licence
2491.2](https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12).