1<h1 align="center">llm_aggregator</h1>
  2
  3<p align="center">
  4    <strong>A CLI tool to aggregate RSS feeds and summarise them with LLMs</strong>
  5</p>
  6
  7![Codeberg Release](https://img.shields.io/gitea/v/release/maxwelljensen/llm_aggregator?gitea_url=https%3A%2F%2Fcodeberg.org&style=for-the-badge)
  8![Codeberg License](assets/eupl-12-badge.svg)
  9
 10---
 11
 12## What is `llm_aggregator`?
 13
 14`llm_aggregator` is a command‑line utility that fetches articles from multiple
 15RSS feeds, filters and processes the content, and sends it to an LLM through
 16OpenAI-compatible API to generate a concise summary or analysis. It’s designed
 17for keeping up with news and articles from your favourite sources without
 18having to read dozens or hundreds of individual posts.
 19
 20The tool includes a WIP terminal user interface (TUI) built with
 21`bubbletea` and `lipgloss` that shows a live progress bar, article counts, and
 22elapsed time while the aggregation runs.
 23
 24## How do I use `llm_aggregator`?
 25
 26    llm_aggregator --feeds-file FEEDS-FILE --prompt PROMPT [OPTIONS]...
 27
 28By default, `llm_aggregator` reads a list of RSS feed URLs, fetches the
 29articles, filters them by date and keywords, and sends a summary request to
 30Deepseek. The resulting output is printed to the terminal in your chosen format
 31(text, markdown, or JSON).
 32
 33### Basic Options
 34
 35    --feeds-file FILE         Path to file containing RSS feed URLs (one per line)
 36    --prompt PROMPT           User prompt for summarisation/analysis
 37    --api-key KEY             Deepseek API key (default: read from DEEPSEEK_API_KEY env var)
 38    --model MODEL             Deepseek model to use (default: deepseek-chat)
 39    --max-tokens N            Maximum tokens in response (default: 4000)
 40    --output FORMAT           Output format: text, markdown, or json (default: text)
 41    --output-file FILE        Write output to FILE instead of stdout
 42    --tui                     Enable TUI interface with progress bar
 43    --verbose, -v             Enable verbose logging
 44    --help, -h                Show this help message and exit
 45    --version                 Show version information and exit
 46
 47### Filtering & Processing
 48
 49    --max-articles-per-feed N Maximum articles to fetch from each feed (default: 10)
 50    --max-days-old N          Only include articles from the last N days (0 for all) (default: 7)
 51    --max-total-articles N    Maximum total articles to process (default: 20)
 52    --include-keywords LIST   Comma-separated list of keywords to include (case‑insensitive)
 53    --exclude-keywords LIST   Comma-separated list of keywords to exclude (case‑insensitive)
 54    --include-articles        Include original articles in JSON output
 55
 56### Deepseek Configuration
 57
 58    --temperature VALUE       Sampling temperature (0.0 to 1.0) (default: 0.7)
 59    --system-prompt TEXT      Custom system prompt for Deepseek
 60
 61### Examples
 62
 63```bash
 64# Basic usage: summarise tech news from a list of feeds
 65llm_aggregator --feeds-file feeds.txt \
 66--prompt "What are the latest AI-related trends in free software?"
 67
 68# With TUI progress bar
 69llm_aggregator --feeds-file feeds.txt --prompt "Summarise tech news" --tui
 70
 71# Output to a JSON file with included articles
 72llm_aggregator --feeds-file feeds.txt --prompt "Analyse AI developments" \
 73    --output json --output-file analysis.json --include-articles
 74
 75# Filter by keywords (only include articles about Linux or open source)
 76llm_aggregator --feeds-file feeds.txt --prompt "Linux news" \
 77    --include-keywords linux,opensource --max-days-old 3
 78
 79# Use a custom Deepseek model and higher token limit
 80llm_aggregator --feeds-file feeds.txt --prompt "Code analysis" \
 81    --model deepseek-coder --max-tokens 8000
 82
 83# Show version information
 84llm_aggregator --version
 85
 86# Show help message
 87llm_aggregator --help
 88```
 89
 90## How does `llm_aggregator` work?
 91
 92`llm_aggregator` performs the following steps for each run:
 93
 941. Parse command‑line arguments
 952. Read the feeds file: a plain text file containing one RSS/Atom feed URL per
 96   line.
 973. Fetch and parse each feed. RSS, Atom, and JSON Feed formats are supported.
 984. **Extract article content**: for each feed entry, the tool extracts the
 99   title, link, publication date, author, and description. If the feed provides
100   only a snippet, it can optionally fetch the full webpage using `goquery` to
101   extract the main content.
1025. **Filter and sort articles**: articles are filtered by age (configurable
103   with `--max-days-old`), optionally filtered by keywords (include/exclude),
104   and sorted by date, title, or source.
1056. Prepare the prompt with selected articles, formatted into a context
106   string that is sent to the LLM along with the user’s custom prompt.
1077. Call the OpenAI API via the `openai‑go` client.
1088. **Format and output the result**: the AI’s response is printed in the chosen
109   format (plain text, GitHub‑flavoured markdown, or JSON). If JSON output is
110   selected, the original articles can be included alongside the summary.
111
112When the `--tui` flag is used, the entire process is wrapped in a `bubbletea`
113TUI that shows a colourful progress bar, live article counters, and elapsed
114time (WIP).
115
116## Dependencies
117
118`llm_aggregator` is written in Go and uses the following libraries:
119
120* [`gofeed`](https://github.com/mmcdole/gofeed): a robust RSS/Atom/JSON feed parser
121* [`openai‑go`](https://github.com/openai/openai-go): the official OpenAI Go
122  SDK, configured to work with Deepseek’s compatible API
123* [`bubbletea`](https://github.com/charmbracelet/bubbletea): a TUI framework
124  for building terminal applications
125* [`lipgloss`](https://github.com/charmbracelet/lipgloss): a library for
126  styling terminal output (colours, borders, alignment)
127* [`go‑arg`](https://github.com/alexflint/go-arg): struct‑based argument
128  parsing with automatic help and version flags
129* [`goquery`](https://github.com/PuerkitoBio/goquery): a jQuery‑like HTML
130  scraping library (used as a fallback when feed content is minimal)
131
132## How do I build `llm_aggregator`?
133
134`llm_aggregator` can be built with a standard Go toolchain:
135
136    go build ./cmd/llm_aggregator.go
137
138## Configuration file
139
140A configuration file is not yet implemented; all options are passed via
141command‑line arguments or environment variables. The API key can be provided
142either with `--api-key` or by setting the `DEEPSEEK_API_KEY` environment
143variable.
144
145## Example feeds file
146
147Create a file named `feeds.txt` with your favourite RSS feeds, one per line.
148For example:
149
150    https://news.ycombinator.com/rss
151    https://lwn.net/headlines/newrss
152    https://opensource.com/feed
153    https://www.phoronix.com/rss.php
154
155Then run:
156
157    llm_aggregator --feeds-file feeds.txt --prompt "Summarise the top tech stories"
158
159## Licence
160
161This project is licensed under [European Union Public Licence
1621.2](https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12).