README.md (view raw)
1<h1 align="center">llm_aggregator</h1>
2
3<p align="center">
4 <strong>A CLI tool to aggregate RSS feeds and summarise them with LLMs</strong>
5</p>
6
7
8
9
10---
11
12## What is `llm_aggregator`?
13
14`llm_aggregator` is a command‑line utility that fetches articles from multiple
15RSS feeds, filters and processes the content, and sends it to an LLM through
16OpenAI-compatible API to generate a concise summary or analysis. It’s designed
17for keeping up with news and articles from your favourite sources without
18having to read dozens or hundreds of individual posts.
19
20## How do I use `llm_aggregator`?
21
22 llm_aggregator --feeds-file FEEDS-FILE --prompt PROMPT [OPTIONS]...
23
24By default, `llm_aggregator` reads a list of RSS feed URLs, fetches the
25articles, filters them by date and keywords, and sends a summary request to
26the LLM. The resulting output is printed to the terminal in your chosen format
27(text, markdown, or JSON).
28
29### Basic Options
30
31 --feeds-file FILE Path to file containing RSS feed URLs (one per line)
32 --prompt PROMPT User prompt for summarisation/analysis
33 --api-key KEY API key (default: read from $LLM_AGGREGATOR_API_KEY)
34 --model MODEL Model to use (default: deepseek-chat)
35 --max-tokens N Maximum tokens in response (default: 4000)
36 --output FORMAT Output format: text, markdown, or json (default: text)
37 --output-file FILE Write output to FILE instead of stdout
38 --tui Enable TUI interface with progress bar
39 --verbose, -v Enable verbose logging
40 --help, -h Show this help message and exit
41 --version Show version information and exit
42
43### Filtering & Processing
44
45 --max-articles-per-feed N Maximum articles to fetch from each feed (default: 10)
46 --max-days-old N Only include articles from the last N days (0 for all) (default: 7)
47 --max-total-articles N Maximum total articles to process (default: 20)
48 --include-keywords LIST Comma-separated list of keywords to include (case‑insensitive)
49 --exclude-keywords LIST Comma-separated list of keywords to exclude (case‑insensitive)
50 --include-articles Include original articles in JSON output
51
52### LLM Configuration
53
54 --temperature VALUE Sampling temperature (0.0 to 1.0) (default: 0.7)
55 --system-prompt TEXT Custom system prompt for LLM
56
57### Examples
58
59```bash
60# Basic usage: summarise tech news from a list of feeds
61llm_aggregator --feeds-file feeds.txt \
62--prompt "What are the latest AI-related trends in free software?"
63
64# With TUI progress bar
65llm_aggregator --feeds-file feeds.txt --prompt "Summarise tech news" --tui
66
67# Output to a JSON file with included articles
68llm_aggregator --feeds-file feeds.txt --prompt "Analyse AI developments" \
69 --output json --output-file analysis.json --include-articles
70
71# Filter by keywords (only include articles about Linux or open source)
72llm_aggregator --feeds-file feeds.txt --prompt "Linux news" \
73 --include-keywords linux,opensource --max-days-old 3
74
75# Use a custom model and higher token limit
76llm_aggregator --feeds-file feeds.txt --prompt "Code analysis" \
77 --model deepseek-reasoner --max-tokens 8000
78
79# Show version information
80llm_aggregator --version
81
82# Show help message
83llm_aggregator --help
84```
85
86## How does `llm_aggregator` work?
87
88`llm_aggregator` performs the following steps for each run:
89
901. Parse command‑line arguments
912. Read the feeds file: a plain text file containing one RSS/Atom feed URL per
92 line.
933. **Fetch and parse feeds concurrently**: RSS, Atom, and JSON Feed formats are
94 supported. Feeds are fetched in parallel with rate limiting to maximise
95 throughput while avoiding server overload.
964. **Extract article content**: for each feed entry, the tool extracts the
97 title, link, publication date, author, and description. If the feed provides
98 only a snippet, it can optionally fetch the full webpage using `goquery` to
99 extract the main content.
1005. **Filter and sort articles**: articles are filtered by age (configurable
101 with `--max-days-old`), optionally filtered by keywords (include/exclude),
102 and sorted by date, title, or source.
1036. Prepare the prompt with selected articles, formatted into a context
104 string that is sent to the LLM along with the user’s custom prompt.
1057. Call the OpenAI API via the `openai‑go` client.
1068. **Format and output the result**: the AI’s response is printed in the chosen
107 format (plain text, GitHub‑flavoured markdown, or JSON). If JSON output is
108 selected, the original articles can be included alongside the summary.
109
110When the `--tui` flag is used, the entire process is wrapped in a `bubbletea`
111TUI that shows a colourful progress bar, live article counters, and elapsed
112time. The TUI supports keyboard navigation (j/k, arrows, space, b, g/G) and
113mouse wheel scrolling for browsing long summaries.
114
115Feeds are fetched concurrently for optimal performance, with rate limiting to
116avoid overwhelming feed servers.
117
118## Configuration
119
120`llm_aggregator` supports multiple configuration sources with the following precedence order (highest to lowest):
121
1221. **Command‑line arguments** – Override everything
1232. **Environment variables** – Start with `LLM_AGGREGATOR_` prefix
1243. **Configuration file** – `~/.config/llm_aggregator/config.toml`
1254. **Built‑in defaults**
126
127### Configuration file
128
129Create a TOML file at `~/.config/llm_aggregator/config.toml` with the following structure:
130
131```toml
132# Feed aggregation options
133max_articles_per_feed = 10
134max_days_old = 7
135max_total_articles = 20
136
137# Content filtering (comma-separated keywords)
138# include_keywords = "linux,opensource"
139# exclude_keywords = "windows,microsoft"
140
141# LLM API options
142# api_key = "your_api_key_here" # Can also be set via LLM_AGGREGATOR_API_KEY env var
143model = "deepseek-chat"
144max_tokens = 4000
145temperature = 0.7
146
147# System prompt for LLM API
148system_prompt = """You are an expert analyst and summariser.
149You analyse content from multiple sources and provide
150concise, insightful summaries based on user requests.
151Focus on key points, trends, and important information."""
152
153# Output options
154output = "text" # Options: text, json, markdown
155# output_file = "" # Optional output file path
156include_articles = false
157```
158
159### Environment variables
160
161All configuration options can also be set via environment variables with the `LLM_AGGREGATOR_` prefix:
162
163- `LLM_AGGREGATOR_API_KEY` – LLM API key
164- `LLM_AGGREGATOR_MODEL` – Model name (default: "deepseek-chat")
165- `LLM_AGGREGATOR_MAX_TOKENS` – Maximum tokens in response (default: 4000)
166- `LLM_AGGREGATOR_TEMPERATURE` – Sampling temperature (default: 0.7)
167- `LLM_AGGREGATOR_SYSTEM_PROMPT` – Custom system prompt
168- `LLM_AGGREGATOR_MAX_ARTICLES_PER_FEED` – Maximum articles per feed (default: 10)
169- `LLM_AGGREGATOR_MAX_DAYS_OLD` – Maximum article age in days (default: 7)
170- `LLM_AGGREGATOR_MAX_TOTAL_ARTICLES` – Maximum total articles (default: 20)
171- `LLM_AGGREGATOR_INCLUDE_KEYWORDS` – Comma‑separated include keywords
172- `LLM_AGGREGATOR_EXCLUDE_KEYWORDS` – Comma‑separated exclude keywords
173- `LLM_AGGREGATOR_OUTPUT` – Output format (default: "text")
174- `LLM_AGGREGATOR_OUTPUT_FILE` – Output file path
175- `LLM_AGGREGATOR_INCLUDE_ARTICLES` – Include articles in JSON output (true/false)
176
177The API key can be provided via `--api‑key`, `LLM_AGGREGATOR_API_KEY` environment variable, or in the configuration file.
178
179## Example feeds file
180
181Create a file named `feeds.txt` with your favourite RSS feeds, one per line.
182For example:
183
184 https://news.ycombinator.com/rss
185 https://lwn.net/headlines/newrss
186 https://opensource.com/feed
187 https://www.phoronix.com/rss.php
188
189Then run:
190
191 llm_aggregator --feeds-file feeds.txt --prompt "Summarise the top tech stories"
192
193## Dependencies
194
195`llm_aggregator` is written in Go and uses the following libraries:
196
197* [`gofeed`](https://github.com/mmcdole/gofeed): a robust RSS/Atom/JSON feed parser
198* [`openai‑go`](https://github.com/openai/openai-go): the official OpenAI API
199 library for Go
200* [`bubbletea`](https://github.com/charmbracelet/bubbletea): a TUI framework
201 for building terminal applications
202* [`lipgloss`](https://github.com/charmbracelet/lipgloss): a library for
203 styling terminal output (colours, borders, alignment)
204* [`go‑arg`](https://github.com/alexflint/go-arg): struct‑based argument
205 parsing with automatic help and version flags
206* [`tiktoken-go`](https://github.com/pkoukk/tiktoken-go): OpenAI's tiktoken
207 BPE tokeniser for accurate token counting
208* [`viper`](https://github.com/spf13/viper): configuration management with
209 support for TOML, environment variables, and flags
210* [`goquery`](https://github.com/PuerkitoBio/goquery): jQuery‑like HTML scraping
211 for extracting full article content when feed descriptions are minimal
212
213## How do I build `llm_aggregator`?
214
215`llm_aggregator` can be built with a standard Go toolchain:
216
217 go build ./cmd/llm_aggregator.go
218
219## Licence
220
221This project is licensed under [European Union Public Licence
2221.2](https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12).