README.md (view raw)
1<h1 align="center">llm_aggregator</h1>
2
3<p align="center">
4 <strong>A CLI tool to aggregate RSS feeds and summarise them with LLMs</strong>
5</p>
6
7
8
9
10---
11
12## What is `llm_aggregator`?
13
14`llm_aggregator` is a command‑line utility that fetches articles from multiple
15RSS feeds, filters and processes the content, and sends it to an LLM through
16OpenAI-compatible API to generate a concise summary or analysis. It’s designed
17for keeping up with news and articles from your favourite sources without
18having to read dozens or hundreds of individual posts.
19
20## How do I use `llm_aggregator`?
21
22 llm_aggregator --feeds-file FEEDS-FILE --prompt PROMPT [OPTIONS]...
23
24By default, `llm_aggregator` reads a list of RSS feed URLs, fetches the
25articles, filters them by date and keywords, and sends a summary request to
26the LLM. The resulting output is printed to the terminal in your chosen format
27(text, markdown, or JSON).
28
29### Basic Options
30
31 --feeds-file FILE Path to file containing RSS feed URLs (one per line)
32 --prompt PROMPT User prompt for summarisation/analysis
33 --api-key KEY API key (default: read from $LLM_AGGREGATOR_API_KEY)
34 --model MODEL Model to use (default: deepseek-chat)
35 --base-url URL API base URL (default: https://api.deepseek.com)
36 --max-tokens N Maximum tokens in response (default: 4000)
37 --output FORMAT Output format: text, markdown, or json (default: text)
38 --output-file FILE Write output to FILE instead of stdout
39 --tui Enable TUI interface with progress bar
40 --verbose, -v Enable verbose logging
41 --help, -h Show this help message and exit
42 --version Show version information and exit
43
44### Filtering & Processing
45
46 --max-articles-per-feed N Maximum articles to fetch from each feed (default: 10)
47 --max-days-old N Only include articles from the last N days (0 for all) (default: 7)
48 --max-total-articles N Maximum total articles to process (default: 20)
49 --include-keywords LIST Comma-separated list of keywords to include (case‑insensitive)
50 --exclude-keywords LIST Comma-separated list of keywords to exclude (case‑insensitive)
51 --include-articles Include original articles in JSON output
52
53### LLM Configuration
54
55 --temperature VALUE Sampling temperature (0.0 to 1.0) (default: 0.7)
56 --system-prompt TEXT Custom system prompt for LLM
57
58### Examples
59
60```bash
61# Basic usage: summarise tech news from a list of feeds
62llm_aggregator --feeds-file feeds.txt \
63--prompt "What are the latest AI-related trends in free software?"
64
65# With TUI progress bar
66llm_aggregator --feeds-file feeds.txt --prompt "Summarise tech news" --tui
67
68# Output to a JSON file with included articles
69llm_aggregator --feeds-file feeds.txt --prompt "Analyse AI developments" \
70 --output json --output-file analysis.json --include-articles
71
72# Filter by keywords (only include articles about Linux or open source)
73llm_aggregator --feeds-file feeds.txt --prompt "Linux news" \
74 --include-keywords linux,opensource --max-days-old 3
75
76# Use a custom model and higher token limit
77llm_aggregator --feeds-file feeds.txt --prompt "Code analysis" \
78 --model deepseek-reasoner --max-tokens 8000
79
80# Use a custom API endpoint (e.g., local Ollama)
81llm_aggregator --feeds-file feeds.txt --prompt "Summarise news" \
82 --base-url "http://localhost:11434/v1" --model llama3
83
84# Show version information
85llm_aggregator --version
86
87# Show help message
88llm_aggregator --help
89```
90
91## How does `llm_aggregator` work?
92
93`llm_aggregator` performs the following steps for each run:
94
951. Parse command‑line arguments
962. Read the feeds file: a plain text file containing one RSS/Atom feed URL per
97 line.
983. **Fetch and parse feeds concurrently**: RSS, Atom, and JSON Feed formats are
99 supported. Feeds are fetched in parallel with rate limiting to maximise
100 throughput while avoiding server overload.
1014. **Extract article content**: for each feed entry, the tool extracts the
102 title, link, publication date, author, and description. If the feed provides
103 only a snippet, it can optionally fetch the full webpage using `goquery` to
104 extract the main content.
1055. **Filter and sort articles**: articles are filtered by age (configurable
106 with `--max-days-old`), optionally filtered by keywords (include/exclude),
107 and sorted by date, title, or source.
1086. Prepare the prompt with selected articles, formatted into a context
109 string that is sent to the LLM along with the user’s custom prompt.
1107. Call the OpenAI API via the `openai‑go` client.
1118. **Format and output the result**: the AI’s response is printed in the chosen
112 format (plain text, GitHub‑flavoured markdown, or JSON). If JSON output is
113 selected, the original articles can be included alongside the summary.
114
115When the `--tui` flag is used, the entire process is wrapped in a `bubbletea`
116TUI that shows a colourful progress bar, live article counters, and elapsed
117time. The TUI supports keyboard navigation (j/k, arrows, space, b, g/G) and
118mouse wheel scrolling for browsing long summaries.
119
120Feeds are fetched concurrently for optimal performance, with rate limiting to
121avoid overwhelming feed servers.
122
123## Configuration
124
125`llm_aggregator` supports multiple configuration sources with the following precedence order (highest to lowest):
126
1271. **Command‑line arguments** – Override everything
1282. **Environment variables** – Start with `LLM_AGGREGATOR_` prefix
1293. **Configuration file** – `~/.config/llm_aggregator/config.toml`
1304. **Built‑in defaults**
131
132### Configuration file
133
134Create a TOML file at `~/.config/llm_aggregator/config.toml` with the following structure:
135
136```toml
137# Feed aggregation options
138max_articles_per_feed = 10
139max_days_old = 7
140max_total_articles = 20
141
142# Content filtering (comma-separated keywords)
143# include_keywords = "linux,opensource"
144# exclude_keywords = "windows,microsoft"
145
146# LLM API options
147# api_key = "your_api_key_here" # Can also be set via LLM_AGGREGATOR_API_KEY env var
148# base_url = "https://api.deepseek.com" # Optional custom API endpoint
149model = "deepseek-chat"
150max_tokens = 4000
151temperature = 0.7
152
153# System prompt for LLM API
154system_prompt = """You are an expert analyst and summariser.
155You analyse content from multiple sources and provide
156concise, insightful summaries based on user requests.
157Focus on key points, trends, and important information."""
158
159# Output options
160output = "text" # Options: text, json, markdown
161# output_file = "" # Optional output file path
162include_articles = false
163```
164
165### Environment variables
166
167All configuration options can also be set via environment variables with the `LLM_AGGREGATOR_` prefix:
168
169- `LLM_AGGREGATOR_API_KEY` – LLM API key
170- `LLM_AGGREGATOR_BASE_URL` – API base URL (default: "https://api.deepseek.com")
171- `LLM_AGGREGATOR_MODEL` – Model name (default: "deepseek-chat")
172- `LLM_AGGREGATOR_MAX_TOKENS` – Maximum tokens in response (default: 4000)
173- `LLM_AGGREGATOR_TEMPERATURE` – Sampling temperature (default: 0.7)
174- `LLM_AGGREGATOR_SYSTEM_PROMPT` – Custom system prompt
175- `LLM_AGGREGATOR_MAX_ARTICLES_PER_FEED` – Maximum articles per feed (default: 10)
176- `LLM_AGGREGATOR_MAX_DAYS_OLD` – Maximum article age in days (default: 7)
177- `LLM_AGGREGATOR_MAX_TOTAL_ARTICLES` – Maximum total articles (default: 20)
178- `LLM_AGGREGATOR_INCLUDE_KEYWORDS` – Comma‑separated include keywords
179- `LLM_AGGREGATOR_EXCLUDE_KEYWORDS` – Comma‑separated exclude keywords
180- `LLM_AGGREGATOR_OUTPUT` – Output format (default: "text")
181- `LLM_AGGREGATOR_OUTPUT_FILE` – Output file path
182- `LLM_AGGREGATOR_INCLUDE_ARTICLES` – Include articles in JSON output (true/false)
183
184The API key can be provided via `--api‑key`, `LLM_AGGREGATOR_API_KEY` environment variable, or in the configuration file.
185
186## Example feeds file
187
188Create a file named `feeds.txt` with your favourite RSS feeds, one per line.
189For example:
190
191 https://news.ycombinator.com/rss
192 https://lwn.net/headlines/newrss
193 https://opensource.com/feed
194 https://www.phoronix.com/rss.php
195
196Then run:
197
198 llm_aggregator --feeds-file feeds.txt --prompt "Summarise the top tech stories"
199
200## Dependencies
201
202`llm_aggregator` is written in Go and uses the following libraries:
203
204* [`gofeed`](https://github.com/mmcdole/gofeed): a robust RSS/Atom/JSON feed parser
205* [`openai‑go`](https://github.com/openai/openai-go): the official OpenAI API
206 library for Go
207* [`bubbletea`](https://github.com/charmbracelet/bubbletea): a TUI framework
208 for building terminal applications
209* [`lipgloss`](https://github.com/charmbracelet/lipgloss): a library for
210 styling terminal output (colours, borders, alignment)
211* [`go‑arg`](https://github.com/alexflint/go-arg): struct‑based argument
212 parsing with automatic help and version flags
213* [`tiktoken-go`](https://github.com/pkoukk/tiktoken-go): OpenAI's tiktoken
214 BPE tokeniser for accurate token counting
215* [`viper`](https://github.com/spf13/viper): configuration management with
216 support for TOML, environment variables, and flags
217* [`goquery`](https://github.com/PuerkitoBio/goquery): jQuery‑like HTML scraping
218 for extracting full article content when feed descriptions are minimal
219
220## How do I build `llm_aggregator`?
221
222`llm_aggregator` can be built with a standard Go toolchain:
223
224 go build ./cmd/llm_aggregator.go
225
226## Licence
227
228This project is licensed under [European Union Public Licence
2291.2](https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12).
230
231---
232
233For information about the project's test suite, see [docs/TESTING.md](docs/TESTING.md).