README.md (view raw)
1<h1 align="center">llm_aggregator</h1>
2
3<p align="center">
4 <strong>A CLI tool to aggregate RSS feeds and summarise them with LLMs</strong>
5</p>
6
7
8
9
10---
11
12## What is `llm_aggregator`?
13
14`llm_aggregator` is a command‑line utility that fetches articles from multiple
15RSS feeds, filters and processes the content, and sends it to an LLM through
16OpenAI-compatible API to generate a concise summary or analysis. It’s designed
17for keeping up with news and articles from your favourite sources without
18having to read dozens or hundreds of individual posts.
19
20The tool includes a WIP terminal user interface (TUI) built with
21`bubbletea` and `lipgloss` that shows a live progress bar, article counts, and
22elapsed time while the aggregation runs.
23
24## How do I use `llm_aggregator`?
25
26 llm_aggregator --feeds-file FEEDS-FILE --prompt PROMPT [OPTIONS]...
27
28By default, `llm_aggregator` reads a list of RSS feed URLs, fetches the
29articles, filters them by date and keywords, and sends a summary request to
30the LLM. The resulting output is printed to the terminal in your chosen format
31(text, markdown, or JSON).
32
33### Basic Options
34
35 --feeds-file FILE Path to file containing RSS feed URLs (one per line)
36 --prompt PROMPT User prompt for summarisation/analysis
37 --api-key KEY API key (default: read from $LLM_AGGREGATOR_API_KEY)
38 --model MODEL Model to use (default: deepseek-chat)
39 --max-tokens N Maximum tokens in response (default: 4000)
40 --output FORMAT Output format: text, markdown, or json (default: text)
41 --output-file FILE Write output to FILE instead of stdout
42 --tui Enable TUI interface with progress bar
43 --verbose, -v Enable verbose logging
44 --help, -h Show this help message and exit
45 --version Show version information and exit
46
47### Filtering & Processing
48
49 --max-articles-per-feed N Maximum articles to fetch from each feed (default: 10)
50 --max-days-old N Only include articles from the last N days (0 for all) (default: 7)
51 --max-total-articles N Maximum total articles to process (default: 20)
52 --include-keywords LIST Comma-separated list of keywords to include (case‑insensitive)
53 --exclude-keywords LIST Comma-separated list of keywords to exclude (case‑insensitive)
54 --include-articles Include original articles in JSON output
55
56### Deepseek Configuration
57
58 --temperature VALUE Sampling temperature (0.0 to 1.0) (default: 0.7)
59 --system-prompt TEXT Custom system prompt for Deepseek
60
61### Examples
62
63```bash
64# Basic usage: summarise tech news from a list of feeds
65llm_aggregator --feeds-file feeds.txt \
66--prompt "What are the latest AI-related trends in free software?"
67
68# With TUI progress bar
69llm_aggregator --feeds-file feeds.txt --prompt "Summarise tech news" --tui
70
71# Output to a JSON file with included articles
72llm_aggregator --feeds-file feeds.txt --prompt "Analyse AI developments" \
73 --output json --output-file analysis.json --include-articles
74
75# Filter by keywords (only include articles about Linux or open source)
76llm_aggregator --feeds-file feeds.txt --prompt "Linux news" \
77 --include-keywords linux,opensource --max-days-old 3
78
79# Use a custom model and higher token limit
80llm_aggregator --feeds-file feeds.txt --prompt "Code analysis" \
81 --model deepseek-reasoner --max-tokens 8000
82
83# Show version information
84llm_aggregator --version
85
86# Show help message
87llm_aggregator --help
88```
89
90## How does `llm_aggregator` work?
91
92`llm_aggregator` performs the following steps for each run:
93
941. Parse command‑line arguments
952. Read the feeds file: a plain text file containing one RSS/Atom feed URL per
96 line.
973. Fetch and parse each feed. RSS, Atom, and JSON Feed formats are supported.
984. **Extract article content**: for each feed entry, the tool extracts the
99 title, link, publication date, author, and description. If the feed provides
100 only a snippet, it can optionally fetch the full webpage using `goquery` to
101 extract the main content.
1025. **Filter and sort articles**: articles are filtered by age (configurable
103 with `--max-days-old`), optionally filtered by keywords (include/exclude),
104 and sorted by date, title, or source.
1056. Prepare the prompt with selected articles, formatted into a context
106 string that is sent to the LLM along with the user’s custom prompt.
1077. Call the OpenAI API via the `openai‑go` client.
1088. **Format and output the result**: the AI’s response is printed in the chosen
109 format (plain text, GitHub‑flavoured markdown, or JSON). If JSON output is
110 selected, the original articles can be included alongside the summary.
111
112When the `--tui` flag is used, the entire process is wrapped in a `bubbletea`
113TUI that shows a colourful progress bar, live article counters, and elapsed
114time (WIP).
115
116## Configuration
117
118`llm_aggregator` supports multiple configuration sources with the following precedence order (highest to lowest):
119
1201. **Command‑line arguments** – Override everything
1212. **Environment variables** – Start with `LLM_AGGREGATOR_` prefix
1223. **Configuration file** – `~/.config/llm_aggregator/config.toml`
1234. **Built‑in defaults**
124
125### Configuration file
126
127Create a TOML file at `~/.config/llm_aggregator/config.toml` with the following structure:
128
129```toml
130# Feed aggregation options
131max_articles_per_feed = 10
132max_days_old = 7
133max_total_articles = 20
134
135# Content filtering (comma-separated keywords)
136# include_keywords = "linux,opensource"
137# exclude_keywords = "windows,microsoft"
138
139# Deepseek API options
140# api_key = "your_api_key_here" # Can also be set via LLM_AGGREGATOR_API_KEY env var
141model = "deepseek-chat"
142max_tokens = 4000
143temperature = 0.7
144
145# System prompt for Deepseek API
146system_prompt = """You are an expert analyst and summariser.
147You analyse content from multiple sources and provide
148concise, insightful summaries based on user requests.
149Focus on key points, trends, and important information."""
150
151# Output options
152output = "text" # Options: text, json, markdown
153# output_file = "" # Optional output file path
154include_articles = false
155```
156
157### Environment variables
158
159All configuration options can also be set via environment variables with the `LLM_AGGREGATOR_` prefix:
160
161- `LLM_AGGREGATOR_API_KEY` – Deepseek API key
162- `LLM_AGGREGATOR_MODEL` – Model name (default: "deepseek-chat")
163- `LLM_AGGREGATOR_MAX_TOKENS` – Maximum tokens in response (default: 4000)
164- `LLM_AGGREGATOR_TEMPERATURE` – Sampling temperature (default: 0.7)
165- `LLM_AGGREGATOR_SYSTEM_PROMPT` – Custom system prompt
166- `LLM_AGGREGATOR_MAX_ARTICLES_PER_FEED` – Maximum articles per feed (default: 10)
167- `LLM_AGGREGATOR_MAX_DAYS_OLD` – Maximum article age in days (default: 7)
168- `LLM_AGGREGATOR_MAX_TOTAL_ARTICLES` – Maximum total articles (default: 20)
169- `LLM_AGGREGATOR_INCLUDE_KEYWORDS` – Comma‑separated include keywords
170- `LLM_AGGREGATOR_EXCLUDE_KEYWORDS` – Comma‑separated exclude keywords
171- `LLM_AGGREGATOR_OUTPUT` – Output format (default: "text")
172- `LLM_AGGREGATOR_OUTPUT_FILE` – Output file path
173- `LLM_AGGREGATOR_INCLUDE_ARTICLES` – Include articles in JSON output (true/false)
174
175The API key can be provided via `--api‑key`, `LLM_AGGREGATOR_API_KEY` environment variable, or in the configuration file.
176
177## Example feeds file
178
179Create a file named `feeds.txt` with your favourite RSS feeds, one per line.
180For example:
181
182 https://news.ycombinator.com/rss
183 https://lwn.net/headlines/newrss
184 https://opensource.com/feed
185 https://www.phoronix.com/rss.php
186
187Then run:
188
189 llm_aggregator --feeds-file feeds.txt --prompt "Summarise the top tech stories"
190
191## Dependencies
192
193`llm_aggregator` is written in Go and uses the following libraries:
194
195* [`gofeed`](https://github.com/mmcdole/gofeed): a robust RSS/Atom/JSON feed parser
196* [`openai‑go`](https://github.com/openai/openai-go): the official OpenAI API
197library for Go
198* [`bubbletea`](https://github.com/charmbracelet/bubbletea): a TUI framework
199 for building terminal applications
200* [`lipgloss`](https://github.com/charmbracelet/lipgloss): a library for
201 styling terminal output (colours, borders, alignment)
202* [`go‑arg`](https://github.com/alexflint/go-arg): struct‑based argument
203 parsing with automatic help and version flags
204* [`goquery`](https://github.com/PuerkitoBio/goquery): a jQuery‑like HTML
205 scraping library (used as a fallback when feed content is minimal)
206* [`viper`](https://github.com/spf13/viper): library used for reading and
207 writing to configuration files.
208
209## How do I build `llm_aggregator`?
210
211`llm_aggregator` can be built with a standard Go toolchain:
212
213 go build ./cmd/llm_aggregator.go
214
215## Licence
216
217This project is licensed under [European Union Public Licence
2181.2](https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12).