1<h1 align="center">llm_aggregator</h1>
  2
  3<p align="center">
  4    <img src="assets/logo.svg" alt="LLM Aggregator logo" width=500>
  5    <br>
  6    <strong>A CLI tool to aggregate RSS feeds and summarise them with LLMs</strong>
  7</p>
  8
  9![Codeberg Release](https://img.shields.io/gitea/v/release/maxwelljensen/llm_aggregator?gitea_url=https%3A%2F%2Fcodeberg.org&style=for-the-badge)
 10![Codeberg License](assets/eupl-12-badge.svg)
 11
 12---
 13
 14## What is `llm_aggregator`?
 15
 16`llm_aggregator` is a command‑line utility that fetches articles from multiple
 17RSS feeds, filters and processes the content, and sends it to an LLM through
 18OpenAI-compatible API to generate a concise summary or analysis. It’s designed
 19for keeping up with news and articles from your favourite sources without
 20having to read dozens or hundreds of individual posts.
 21
 22## How do I use `llm_aggregator`?
 23
 24    llm_aggregator --feeds-file FEEDS-FILE --prompt PROMPT [OPTIONS]...
 25
 26By default, `llm_aggregator` reads a list of RSS feed URLs, fetches the
 27articles, filters them by date and keywords, and sends a summary request to
 28the LLM. The resulting output is printed to the terminal in your chosen format
 29(text, markdown, or JSON).
 30
 31### Basic Options
 32
 33    --feeds-file FILE         Path to file containing RSS feed URLs (one per line)
 34    --prompt PROMPT           User prompt for summarisation/analysis
 35    --api-key KEY             API key (default: read from $LLM_AGGREGATOR_API_KEY)
 36    --model MODEL             Model to use (default: deepseek-chat)
 37    --dry-run                 Validate config, show article statistics, and exit without making LLM API calls.
 38    --base-url URL            API base URL (default: https://api.deepseek.com)
 39    --max-tokens N            Maximum tokens in response (default: 4000)
 40    --output FORMAT           Output format: text, markdown, or json (default: text)
 41    --output-file FILE        Write output to FILE instead of stdout
 42    --tui                     Enable TUI interface with progress bar
 43    --verbose, -v             Enable verbose logging
 44    --help, -h                Show this help message and exit
 45    --version                 Show version information and exit
 46
 47### Filtering & Processing
 48
 49    --max-articles-per-feed N Maximum articles to fetch from each feed (default: 10)
 50    --max-days-old N          Only include articles from the last N days (0 for all) (default: 7)
 51    --max-total-articles N    Maximum total articles to process (default: 20)
 52    --include-keywords LIST   Comma-separated list of keywords to include (case‑insensitive)
 53    --exclude-keywords LIST   Comma-separated list of keywords to exclude (case‑insensitive)
 54    --include-articles        Include original articles in JSON output
 55
 56### LLM Configuration
 57
 58    --temperature VALUE       Sampling temperature (0.0 to 1.0) (default: 0.7)
 59    --system-prompt TEXT      Custom system prompt for LLM
 60
 61### Examples
 62
 63```bash
 64# Basic usage: summarise tech news from a list of feeds
 65llm_aggregator --feeds-file feeds.txt \
 66--prompt "What are the latest AI-related trends in free software?"
 67
 68# With TUI progress bar
 69llm_aggregator --feeds-file feeds.txt --prompt "Summarise tech news" --tui
 70
 71# Output to a JSON file with included articles
 72llm_aggregator --feeds-file feeds.txt --prompt "Analyse AI developments" \
 73    --output json --output-file analysis.json --include-articles
 74
 75# Filter by keywords (only include articles about Linux or open source)
 76llm_aggregator --feeds-file feeds.txt --prompt "Linux news" \
 77    --include-keywords linux,opensource --max-days-old 3
 78
 79# Use a custom model and higher token limit
 80llm_aggregator --feeds-file feeds.txt --prompt "Code analysis" \
 81    --model deepseek-reasoner --max-tokens 8000
 82
 83# Use a custom API endpoint (e.g., local Ollama)
 84llm_aggregator --feeds-file feeds.txt --prompt "Summarise news" \
 85    --base-url "http://localhost:11434/v1" --model llama3
 86
 87# Show version information
 88llm_aggregator --version
 89
 90# Show help message
 91llm_aggregator --help
 92```
 93
 94## How does `llm_aggregator` work?
 95
 96`llm_aggregator` performs the following steps for each run:
 97
 981. Parse command‑line arguments
 992. Read the feeds file: a plain text file containing one RSS/Atom feed URL per
100   line.
1013. **Fetch and parse feeds concurrently**: RSS, Atom, and JSON Feed formats are
102   supported. Feeds are fetched in parallel with rate limiting to maximise
103   throughput while avoiding server overload.
1044. **Extract article content**: for each feed entry, the tool extracts the
105   title, link, publication date, author, and description. If the feed provides
106   only a snippet, it can optionally fetch the full webpage using `goquery` to
107   extract the main content.
1085. **Filter and sort articles**: articles are filtered by age (configurable
109   with `--max-days-old`), optionally filtered by keywords (include/exclude),
110   and sorted by date, title, or source.
1116. Prepare the prompt with selected articles, formatted into a context
112   string that is sent to the LLM along with the user’s custom prompt.
1137. Call the OpenAI API via the `openai‑go` client.
1148. **Format and output the result**: the AI’s response is printed in the chosen
115   format (plain text, GitHub‑flavoured markdown, or JSON). If JSON output is
116   selected, the original articles can be included alongside the summary.
117
118When the `--tui` flag is used, the entire process is wrapped in a `bubbletea`
119TUI that shows a colourful progress bar, live article counters, and elapsed
120time. The TUI renders Markdown content from the LLM with proper styling for
121headers, bold, italic, code blocks, and lists. The TUI supports keyboard
122navigation (j/k, arrows, space, b, g/G) and mouse wheel scrolling for
123browsing long summaries.
124
125Feeds are fetched concurrently for optimal performance, with rate limiting to
126avoid overwhelming feed servers.
127
128## Configuration
129
130`llm_aggregator` supports multiple configuration sources with the following precedence order (highest to lowest):
131
1321. **Command‑line arguments** – Override everything
1332. **Environment variables** – Start with `LLM_AGGREGATOR_` prefix
1343. **Configuration file** – `~/.config/llm_aggregator/config.toml`
1354. **Built‑in defaults**
136
137### Configuration file
138
139Create a TOML file at `~/.config/llm_aggregator/config.toml` with the following structure:
140
141```toml
142# Feed aggregation options
143max_articles_per_feed = 10
144max_days_old = 7
145max_total_articles = 20
146
147# Content filtering (comma-separated keywords)
148# include_keywords = "linux,opensource"
149# exclude_keywords = "windows,microsoft"
150
151# LLM API options
152# api_key = "your_api_key_here"  # Can also be set via LLM_AGGREGATOR_API_KEY env var
153# base_url = "https://api.deepseek.com"  # Optional custom API endpoint
154model = "deepseek-chat"
155max_tokens = 4000
156temperature = 0.7
157
158# System prompt for LLM API
159system_prompt = """You are an expert analyst and summariser.
160You analyse content from multiple sources and provide
161concise, insightful summaries based on user requests.
162Focus on key points, trends, and important information."""
163
164# Output options
165output = "text"  # Options: text, json, markdown
166# output_file = ""  # Optional output file path
167include_articles = false
168```
169
170### Environment variables
171
172All configuration options can also be set via environment variables with the `LLM_AGGREGATOR_` prefix:
173
174| Variable | Description |
175|----------|-------------|
176| `LLM_AGGREGATOR_API_KEY` | LLM API key |
177| `LLM_AGGREGATOR_BASE_URL` | API base URL (default: "https://api.deepseek.com") |
178| `LLM_AGGREGATOR_MODEL` | Model name (default: "deepseek-chat") |
179| `LLM_AGGREGATOR_MAX_TOKENS` | Maximum tokens in response (default: 4000) |
180| `LLM_AGGREGATOR_TEMPERATURE` | Sampling temperature (default: 0.7) |
181| `LLM_AGGREGATOR_SYSTEM_PROMPT` | Custom system prompt |
182| `LLM_AGGREGATOR_MAX_ARTICLES_PER_FEED` | Maximum articles per feed (default: 10) |
183| `LLM_AGGREGATOR_MAX_DAYS_OLD` | Maximum article age in days (default: 7) |
184| `LLM_AGGREGATOR_MAX_TOTAL_ARTICLES` | Maximum total articles (default: 20) |
185| `LLM_AGGREGATOR_INCLUDE_KEYWORDS` | Comma‑separated include keywords |
186| `LLM_AGGREGATOR_EXCLUDE_KEYWORDS` | Comma‑separated exclude keywords |
187| `LLM_AGGREGATOR_OUTPUT` | Output format (default: "text") |
188| `LLM_AGGREGATOR_OUTPUT_FILE` | Output file path |
189| `LLM_AGGREGATOR_INCLUDE_ARTICLES` | Include articles in JSON output (true/false) |
190
191The API key can be provided via `--api‑key`, `LLM_AGGREGATOR_API_KEY` environment variable, or in the configuration file.
192
193## Example feeds file
194
195Create a file named `feeds.txt` with your favourite RSS feeds, one per line.
196For example:
197
198    https://news.ycombinator.com/rss
199    https://lwn.net/headlines/newrss
200    https://opensource.com/feed
201    https://www.phoronix.com/rss.php
202
203Then run:
204
205    llm_aggregator --feeds-file feeds.txt --prompt "Summarise the top tech stories"
206
207## Dependencies
208
209`llm_aggregator` is written in Go and uses the following libraries:
210
211| Library | Description |
212|---------|-------------|
213| [`gofeed`](https://github.com/mmcdole/gofeed) | Robust RSS/Atom/JSON feed parser |
214| [`openai‑go`](https://github.com/openai/openai-go) | Official OpenAI API library for Go |
215| [`bubbletea`](https://github.com/charmbracelet/bubbletea) | TUI framework for terminal applications |
216| [`lipgloss`](https://github.com/charmbracelet/lipgloss) | Library for styling terminal output |
217| [`glamour`](https://github.com/charmbracelet/glamour) | Markdown rendering for terminal (used in TUI mode) |
218| [`go‑arg`](https://github.com/alexflint/go-arg) | Struct‑based argument parsing |
219| [`tiktoken-go`](https://github.com/pkoukk/tiktoken-go) | OpenAI's tiktoken BPE tokeniser |
220| [`viper`](https://github.com/spf13/viper) | Configuration management |
221| [`goquery`](https://github.com/PuerkitoBio/goquery) | jQuery‑like HTML scraping |
222
223## How do I build `llm_aggregator`?
224
225`llm_aggregator` can be built with a standard Go toolchain:
226
227    go build ./cmd/llm_aggregator.go
228
229For information about the project's test suite, see
230[docs/TESTING.md](docs/TESTING.md).
231
232## Licence
233
234This project is licensed under [European Union Public Licence
2351.2](https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12).