feat: add accurate token counting, concurrent feed fetching, and scrollable TUI summary
- Add accurate token counting with tiktoken
- New `internal/tokeniser` package using
`github.com/pkoukk/tiktoken-go`
- BPE tokenisation for precise counting vs rough character estimation
- Encoding caching and model-aware encoding selection (cl100k_base,
o200k_base, etc.)
- `CountTokens()` and `CountMessagesTokens()` functions
- Fallback to rough estimation when tiktoken initialisation fails
- Add concurrent feed fetching with errgroup
- Fetch feeds in parallel using `golang.org/x/sync/errgroup`
- Semaphore limits concurrent requests to 10 to avoid server overload
- Mutex-protected shared state for thread-safe article collection
- Partial feed failures do not stop other feeds from being processed
- Add scrollable summary view in TUI
- Integrate `bubbles/viewport` component for browsing long summaries
- Keyboard navigation: j/k or arrows (scroll), Space/B (page), g/G
(start/end)
- Mouse wheel scrolling support
- Scroll progress indicator showing position and total lines
- Streamline configuration management
- Use global Viper instance with automatic precedence handling
- Simplify configuration flow: Parse → GetViper → BindCLIArgs →
ViperToRuntime
- Remove manual precedence logic from `cmd/llm_aggregator.go`
- Replace TUI hex colour codes with ANSI 256-colour palette
- Fix viewport scroll event handling and content pre-wrapping before
display
- Rename `internal/llm/deepseek.go` to `internal/llm/llm.go` (generalise
LLM client)
- Update `CHANGELOG.md` (v0.5.0) and `README.md` to document new
features
@@ -6,6 +6,43 @@ The format is based on [Keep a
Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.5.0] - 2026-04-22 + +### Added + +- **Accurate token counting with tiktoken** + - New `internal/tokeniser` package using OpenAI's tiktoken-go library + - BPE tokenisation for accurate token counting vs rough character estimation + - Encoding caching to avoid repeated initialisation overhead + - Model-aware encoding selection (cl100k_base, o200k_base, etc.) + - `CountTokens()` and `CountMessagesTokens()` functions + - Fallback to rough estimation when tiktoken initialisation fails +- **Concurrent feed fetching**: Feeds are now fetched concurrently using + `golang.org/x/sync/errgroup` + - Semaphore limits concurrent requests to 10 to avoid server overload + - Mutex-protected shared state for thread-safe article collection + - Partial failures don't stop other feeds from being processed +- **Scrollable summary in TUI** + - Bubble Tea viewport component for browsing long summaries + - Keyboard navigation: j/k or arrows (scroll), Space/B (page), g/G + (start/end) + - Mouse wheel scrolling support + - Scroll progress indicator showing position and total lines + +### Changed + +- **Streamlined configuration management**: Viper now uses global instance with +automatic precedence handling. Simplified configuration flow: Parse → GetViper +→ BindCLIArgs → ViperToRuntime +- **TUI colour scheme**: All hex colour codes replaced with ANSI 256-colour + palette + +### Fixed + +- Viewport now properly handles scroll events via `Update()`. Summary content + pre-wrapped to viewport width before display +- Fixed string literal colour names to use actual variables + ## [0.4.0] - 2026-04-21 ### Changed
@@ -17,10 +17,6 @@ OpenAI-compatible API to generate a concise summary or analysis. It’s designed
for keeping up with news and articles from your favourite sources without having to read dozens or hundreds of individual posts. -The tool includes a WIP terminal user interface (TUI) built with -`bubbletea` and `lipgloss` that shows a live progress bar, article counts, and -elapsed time while the aggregation runs. - ## How do I use `llm_aggregator`? llm_aggregator --feeds-file FEEDS-FILE --prompt PROMPT [OPTIONS]...@@ -94,7 +90,9 @@
1. Parse command‑line arguments 2. Read the feeds file: a plain text file containing one RSS/Atom feed URL per line. -3. Fetch and parse each feed. RSS, Atom, and JSON Feed formats are supported. +3. **Fetch and parse feeds concurrently**: RSS, Atom, and JSON Feed formats are + supported. Feeds are fetched in parallel with rate limiting to maximise + throughput while avoiding server overload. 4. **Extract article content**: for each feed entry, the tool extracts the title, link, publication date, author, and description. If the feed provides only a snippet, it can optionally fetch the full webpage using `goquery` to@@ -111,7 +109,11 @@ selected, the original articles can be included alongside the summary.
When the `--tui` flag is used, the entire process is wrapped in a `bubbletea` TUI that shows a colourful progress bar, live article counters, and elapsed -time (WIP). +time. The TUI supports keyboard navigation (j/k, arrows, space, b, g/G) and +mouse wheel scrolling for browsing long summaries. + +Feeds are fetched concurrently for optimal performance, with rate limiting to +avoid overwhelming feed servers. ## Configuration@@ -194,17 +196,19 @@ `llm_aggregator` is written in Go and uses the following libraries:
* [`gofeed`](https://github.com/mmcdole/gofeed): a robust RSS/Atom/JSON feed parser * [`openai‑go`](https://github.com/openai/openai-go): the official OpenAI API -library for Go + library for Go * [`bubbletea`](https://github.com/charmbracelet/bubbletea): a TUI framework for building terminal applications * [`lipgloss`](https://github.com/charmbracelet/lipgloss): a library for styling terminal output (colours, borders, alignment) * [`go‑arg`](https://github.com/alexflint/go-arg): struct‑based argument parsing with automatic help and version flags -* [`goquery`](https://github.com/PuerkitoBio/goquery): a jQuery‑like HTML - scraping library (used as a fallback when feed content is minimal) -* [`viper`](https://github.com/spf13/viper): library used for reading and - writing to configuration files. +* [`tiktoken-go`](https://github.com/pkoukk/tiktoken-go): OpenAI's tiktoken + BPE tokeniser for accurate token counting +* [`viper`](https://github.com/spf13/viper): configuration management with + support for TOML, environment variables, and flags +* [`goquery`](https://github.com/PuerkitoBio/goquery): jQuery‑like HTML scraping + for extracting full article content when feed descriptions are minimal ## How do I build `llm_aggregator`?
@@ -22,14 +22,6 @@ func main() {
cli.BuildDate = buildDate cli.Version = version - // Load configuration from file and environment variables - cfg, err := config.Load() - if err != nil { - fmt.Fprintf(os.Stderr, "Warning: could not load configuration: %v\n", err) - // Continue with defaults - cfg = config.DefaultConfig() - } - // Parse command line arguments args, err := cli.ParseArgs() if err != nil {@@ -37,112 +29,27 @@ fmt.Fprintf(os.Stderr, "Error parsing arguments: %v\n", err)
os.Exit(1) } - // Create runtime configuration - rt := runtime.NewRuntime() - rt.FeedsFile = args.FeedsFile - rt.Prompt = args.Prompt + // Get viper instance with config file + environment variables + defaults + v := config.GetViper() - // Apply configuration with precedence: CLI args > Environment vars > Config file > Defaults - applyConfiguration(rt, cfg, args) + // Bind CLI args to viper (highest precedence) + config.BindCLIArgs(v, args.ToViperMap()) // Validate API key - if rt.APIKey == "" { + if v.GetString("api_key") == "" { fmt.Fprintln(os.Stderr, "Error: OpenAI-compatible API key is required. Set via --api-key, LLM_AGGREGATOR_API_KEY environment variable, or config file.") os.Exit(1) } + // Create runtime directly from viper configuration + rt := config.ViperToRuntime(v, args.FeedsFile, args.Prompt) + // Run with TUI if requested if args.TUI { runWithTUI(rt) } else { runWithoutTUI(rt, args.Verbose) } -} - -// applyConfiguration applies configuration with proper precedence -func applyConfiguration(rt *runtime.Runtime, cfg *config.Config, args *cli.Args) { - // Feed aggregation options - CLI args override config - if args.MaxArticlesPerFeed != 0 { - rt.MaxArticlesPerFeed = args.MaxArticlesPerFeed - } else if cfg.MaxArticlesPerFeed != 0 { - rt.MaxArticlesPerFeed = cfg.MaxArticlesPerFeed - } - - if args.MaxDaysOld != 0 { - rt.MaxDaysOld = args.MaxDaysOld - } else if cfg.MaxDaysOld != 0 { - rt.MaxDaysOld = cfg.MaxDaysOld - } - - if args.MaxTotalArticles != 0 { - rt.MaxTotalArticles = args.MaxTotalArticles - } else if cfg.MaxTotalArticles != 0 { - rt.MaxTotalArticles = cfg.MaxTotalArticles - } - - // Content filtering - CLI args override config - if args.IncludeKeywords != "" { - rt.IncludeKeywords = cli.ParseKeywords(args.IncludeKeywords) - } else if cfg.IncludeKeywords != "" { - rt.IncludeKeywords = cli.ParseKeywords(cfg.IncludeKeywords) - } - - if args.ExcludeKeywords != "" { - rt.ExcludeKeywords = cli.ParseKeywords(args.ExcludeKeywords) - } else if cfg.ExcludeKeywords != "" { - rt.ExcludeKeywords = cli.ParseKeywords(cfg.ExcludeKeywords) - } - - // LLM API options - CLI args override config - if args.APIKey != "" { - rt.APIKey = args.APIKey - } else if cfg.APIKey != "" { - rt.APIKey = cfg.APIKey - } else { - // Fall back to environment variable - rt.APIKey = os.Getenv("LLM_AGGREGATOR_API_KEY") - } - - if args.Model != "" { - rt.Model = args.Model - } else if cfg.Model != "" { - rt.Model = cfg.Model - } - - if args.MaxTokens != 0 { - rt.MaxTokens = args.MaxTokens - } else if cfg.MaxTokens != 0 { - rt.MaxTokens = cfg.MaxTokens - } - - if args.Temperature != 0 { - rt.Temperature = args.Temperature - } else if cfg.Temperature != 0 { - rt.Temperature = cfg.Temperature - } - - // System prompt - CLI args override config - if args.SystemPrompt != "" { - rt.SystemPrompt = args.SystemPrompt - } else if cfg.SystemPrompt != "" { - rt.SystemPrompt = cfg.SystemPrompt - } - - // Output options - CLI args override config - if args.Output != "" { - rt.Output = args.Output - } else if cfg.Output != "" { - rt.Output = cfg.Output - } - - if args.OutputFile != "" { - rt.OutputFile = args.OutputFile - } else if cfg.OutputFile != "" { - rt.OutputFile = cfg.OutputFile - } - - rt.IncludeArticles = args.IncludeArticles || cfg.IncludeArticles - rt.Verbose = args.Verbose } func runWithTUI(rt *runtime.Runtime) {
@@ -2,14 +2,13 @@ module llm_aggregator
go 1.25.8 +require golang.org/x/sync v0.20.0 + require ( github.com/PuerkitoBio/goquery v1.8.0 github.com/alexflint/go-arg v1.6.1 github.com/mmcdole/gofeed v1.3.0 github.com/openai/openai-go/v3 v3.32.0 -) - -require ( github.com/alexflint/go-scalar v1.2.0 // indirect github.com/andybalholm/cascadia v1.3.1 // indirect github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect@@ -25,9 +24,11 @@ github.com/clipperhouse/displaywidth v0.9.0 // indirect
github.com/clipperhouse/stringish v0.1.1 // indirect github.com/clipperhouse/uax29/v2 v2.5.0 // indirect github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect + github.com/dlclark/regexp2 v1.10.0 // indirect github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect github.com/fsnotify/fsnotify v1.9.0 // indirect github.com/go-viper/mapstructure/v2 v2.4.0 // indirect + github.com/google/uuid v1.6.0 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/lucasb-eyer/go-colorful v1.3.0 // indirect github.com/mattn/go-isatty v0.0.20 // indirect@@ -40,6 +41,7 @@ github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
github.com/muesli/cancelreader v0.2.2 // indirect github.com/muesli/termenv v0.16.0 // indirect github.com/pelletier/go-toml/v2 v2.2.4 // indirect + github.com/pkoukk/tiktoken-go v0.1.8 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect github.com/rivo/uniseg v0.4.7 // indirect github.com/sagikazarmark/locafero v0.11.0 // indirect
@@ -34,6 +34,8 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/dlclark/regexp2 v1.10.0 h1:+/GIL799phkJqYW+3YbOd8LCcbHzT0Pbo8zl70MHsq0= +github.com/dlclark/regexp2 v1.10.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8= github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6baUTXGLOoWe4PQhGxaX0KpnayAqC48p4= github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM= github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=@@ -45,6 +47,8 @@ github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=@@ -78,6 +82,8 @@ github.com/openai/openai-go/v3 v3.32.0 h1:aHp/3wkX1W6jB8zTtf9xV0aK0qPFSVDqS7AHmlJ4hXs=
github.com/openai/openai-go/v3 v3.32.0/go.mod h1:cdufnVK14cWcT9qA1rRtrXx4FTRsgbDPW7Ia7SS5cZo= github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4= github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY= +github.com/pkoukk/tiktoken-go v0.1.8 h1:85ENo+3FpWgAACBaEUVp+lctuTcYUO7BtmfhlN/QTRo= +github.com/pkoukk/tiktoken-go v0.1.8/go.mod h1:9NiV+i9mJKGj1rYOT+njbv+ZwA/zJxYdewGl6qVatpg= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=@@ -123,6 +129,8 @@ golang.org/x/exp v0.0.0-20231006140011-7918f672742d/go.mod h1:ldy0pHrwJyGW56pPQzzkH36rKxoZW1tw7ZJpeKx+hdo=
golang.org/x/net v0.0.0-20210916014120-12bc252f5db8/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU= golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY= +golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4= +golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -1,17 +1,20 @@
package aggregator import ( + "context" "fmt" "io" "net/http" "os" "strings" + "sync" "time" "llm_aggregator/internal/progress" "github.com/PuerkitoBio/goquery" "github.com/mmcdole/gofeed" + "golang.org/x/sync/errgroup" ) // FeedAggregator aggregates articles from multiple RSS feeds.@@ -44,9 +47,8 @@ }
} // ParseFeedsFromFile parses RSS feeds from a file containing one URL per line. +// Feeds are fetched concurrently for improved performance. func (fa *FeedAggregator) ParseFeedsFromFile(filePath string) ([]*Article, error) { - articles := []*Article{} - file, err := os.Open(filePath) if err != nil { return nil, fmt.Errorf("failed to open feeds file: %w", err)@@ -69,17 +71,46 @@ }
fa.progressCtx.Logf("Found %d feed URLs in %s", len(feedURLs), filePath) + // Use mutex to protect shared state + var mu sync.Mutex + var allArticles []*Article + var feedErrors []string + + // Limit concurrency to avoid overwhelming servers + const maxConcurrency = 10 + sem := make(chan struct{}, maxConcurrency) + + g, _ := errgroup.WithContext(context.Background()) for i, feedURL := range feedURLs { - fa.progressCtx.Logf("Processing feed %d/%d: %s", i+1, len(feedURLs), feedURL) - feedArticles, err := fa.ParseFeed(feedURL) - if err != nil { - fa.progressCtx.Warningf("Failed to parse feed %s: %v", feedURL, err) - continue - } - articles = append(articles, feedArticles...) + currentIndex := i + sem <- struct{}{} // Acquire semaphore + g.Go(func() error { + defer func() { <-sem }() // Release semaphore + fa.progressCtx.Logf("Processing feed %d/%d: %s", currentIndex+1, len(feedURLs), feedURL) + feedArticles, err := fa.ParseFeed(feedURL) + if err != nil { + fa.progressCtx.Warningf("Failed to parse feed %s: %v", feedURL, err) + mu.Lock() + feedErrors = append(feedErrors, fmt.Sprintf("%s: %v", feedURL, err)) + mu.Unlock() + return nil // Don't return error to continue processing other feeds + } + mu.Lock() + allArticles = append(allArticles, feedArticles...) + mu.Unlock() + return nil + }) + } + + if err := g.Wait(); err != nil { + return nil, fmt.Errorf("failed to process feeds: %w", err) + } + + if len(feedErrors) > 0 { + fa.progressCtx.Warningf("Encountered %d feed errors: %v", len(feedErrors), feedErrors) } - return articles, nil + return allArticles, nil } // ParseFeed parses a single RSS feed and extracts articles.@@ -284,4 +315,3 @@ }
return "", nil } -
@@ -115,3 +115,23 @@ }
return &args, nil } + +// ToViperMap converts Args to a map for binding to Viper. +// Only non-empty/non-zero values are included. +func (a *Args) ToViperMap() map[string]any { + return map[string]any{ + "max_articles_per_feed": a.MaxArticlesPerFeed, + "max_days_old": a.MaxDaysOld, + "max_total_articles": a.MaxTotalArticles, + "include_keywords": a.IncludeKeywords, + "exclude_keywords": a.ExcludeKeywords, + "api_key": a.APIKey, + "model": a.Model, + "max_tokens": a.MaxTokens, + "temperature": a.Temperature, + "system_prompt": a.SystemPrompt, + "output": a.Output, + "output_file": a.OutputFile, + "include_articles": a.IncludeArticles, + } +}
@@ -6,6 +6,10 @@ "os"
"path/filepath" "github.com/spf13/viper" + + "llm_aggregator/internal/cli" + "llm_aggregator/internal/progress" + "llm_aggregator/internal/runtime" ) // Config holds the application configuration loaded from TOML file, environment variables, and CLI arguments.@@ -30,9 +34,6 @@ // Output options
Output string `mapstructure:"output"` OutputFile string `mapstructure:"output_file"` IncludeArticles bool `mapstructure:"include_articles"` - - // Internal state - loaded bool } // DefaultConfig returns a Config with sensible defaults.@@ -56,46 +57,162 @@
// Load loads configuration from TOML file, environment variables, and sets defaults. // The precedence order is: CLI args > Environment variables > Config file > Defaults. func Load() (*Config, error) { - config := DefaultConfig() + // Get the global viper instance which handles precedence automatically + v := GetViper() + + // Return the current configuration as a Config struct + return ViperToConfig(v), nil +} + +// GetViper returns the global viper instance with all configuration sources set up. +// The precedence order is: CLI flags > Environment variables > Config file > Defaults. +func GetViper() *viper.Viper { + // Set defaults first (lowest priority) + setDefaults() + + // Initialise or get existing Viper instance + v := viper.GetViper() - // Initialise Viper - v := viper.New() - v.SetConfigName("config") - v.SetConfigType("toml") + // Ensure defaults are set (for fresh instances) + setDefaultsOn(v) - // Set environment variable prefix first + // Set environment variable prefix v.SetEnvPrefix("LLM_AGGREGATOR") - v.AutomaticEnv() // Read from environment variables + v.AutomaticEnv() // Bind environment variables to config fields bindEnvVars(v) // Get config path from XDG configPath, err := GetConfigPath() - if err != nil { - return nil, fmt.Errorf("failed to get config path: %w", err) + if err == nil { + // Set config file path + v.SetConfigFile(configPath) + + // Try to read config file + if err := v.ReadInConfig(); err != nil { + // If config file doesn't exist, that's OK - we'll use defaults + env vars + if _, ok := err.(viper.ConfigFileNotFoundError); !ok { + fmt.Fprintf(os.Stderr, "Warning: error reading config file: %v\n", err) + } + } } - // Set config file path - v.AddConfigPath(filepath.Dir(configPath)) + return v +} + +// setDefaults sets default values on the global viper instance. +func setDefaults() { + v := viper.GetViper() + setDefaultsOn(v) +} + +// setDefaultsOn sets default values on the given viper instance. +func setDefaultsOn(v *viper.Viper) { + // Feed aggregation defaults + v.SetDefault("max_articles_per_feed", 10) + v.SetDefault("max_days_old", 7) + v.SetDefault("max_total_articles", 20) + + // LLM API defaults + v.SetDefault("model", "deepseek-chat") + v.SetDefault("max_tokens", 4000) + v.SetDefault("temperature", 0.7) + + // System prompt default + v.SetDefault("system_prompt", `You are an expert analyst and summariser. +You analyse content from multiple sources and provide +concise, insightful summaries based on user requests. +Focus on key points, trends, and important information.`) + + // Output defaults + v.SetDefault("output", "text") + v.SetDefault("include_articles", false) +} + +// ViperToConfig converts viper configuration to a Config struct. +func ViperToConfig(v *viper.Viper) *Config { + return &Config{ + MaxArticlesPerFeed: v.GetInt("max_articles_per_feed"), + MaxDaysOld: v.GetInt("max_days_old"), + MaxTotalArticles: v.GetInt("max_total_articles"), + IncludeKeywords: v.GetString("include_keywords"), + ExcludeKeywords: v.GetString("exclude_keywords"), + APIKey: v.GetString("api_key"), + Model: v.GetString("model"), + MaxTokens: v.GetInt("max_tokens"), + Temperature: v.GetFloat64("temperature"), + SystemPrompt: v.GetString("system_prompt"), + Output: v.GetString("output"), + OutputFile: v.GetString("output_file"), + IncludeArticles: v.GetBool("include_articles"), + } +} - // Try to read config file - if err := v.ReadInConfig(); err != nil { - // If config file doesn't exist, that's OK - we'll use defaults + env vars - if _, ok := err.(viper.ConfigFileNotFoundError); !ok { - return nil, fmt.Errorf("error reading config file: %w", err) +// BindCLIArgs binds CLI argument values to viper with highest precedence. +// This should be called after parsing CLI arguments to override config file/env defaults. +func BindCLIArgs(v *viper.Viper, args map[string]any) { + for key, value := range args { + if value != nil && !isZero(value) { + v.Set(key, value) } - } else { - // Config file was found - config.loaded = true } +} - // Unmarshal config into struct - if err := v.Unmarshal(config); err != nil { - return nil, fmt.Errorf("error unmarshalling config: %w", err) +// isZero checks if a value is the zero value for its type. +func isZero(v any) bool { + switch val := v.(type) { + case string: + return val == "" + case int, int8, int16, int32, int64: + return val == 0 + case float32, float64: + return val == 0 + case bool: + return !val + default: + return false } +} - return config, nil +// ViperToRuntime converts viper configuration directly to runtime.Runtime. +func ViperToRuntime(v *viper.Viper, feedsFile, prompt string) *runtime.Runtime { + rt := &runtime.Runtime{ + FeedsFile: feedsFile, + Prompt: prompt, + Progress: &progress.NoopLogger{}, + } + + // Feed aggregation options + rt.MaxArticlesPerFeed = v.GetInt("max_articles_per_feed") + rt.MaxDaysOld = v.GetInt("max_days_old") + rt.MaxTotalArticles = v.GetInt("max_total_articles") + + // Keywords + includeKeywords := v.GetString("include_keywords") + excludeKeywords := v.GetString("exclude_keywords") + if includeKeywords != "" { + rt.IncludeKeywords = cli.ParseKeywords(includeKeywords) + } + if excludeKeywords != "" { + rt.ExcludeKeywords = cli.ParseKeywords(excludeKeywords) + } + + // LLM API options + rt.APIKey = v.GetString("api_key") + rt.Model = v.GetString("model") + rt.MaxTokens = v.GetInt("max_tokens") + rt.Temperature = v.GetFloat64("temperature") + + // System prompt + rt.SystemPrompt = v.GetString("system_prompt") + + // Output options + rt.Output = v.GetString("output") + rt.OutputFile = v.GetString("output_file") + rt.IncludeArticles = v.GetBool("include_articles") + + return rt } // bindEnvVars binds environment variables to viper keys.@@ -189,5 +306,11 @@ }
// IsLoaded returns true if the configuration was loaded from a file. func (c *Config) IsLoaded() bool { - return c.loaded + // Check if config file exists - if it does, values would have been loaded from it + configPath, err := GetConfigPath() + if err != nil { + return false + } + _, err = os.Stat(configPath) + return err == nil }
@@ -8,6 +8,7 @@ "time"
"llm_aggregator/internal/aggregator" "llm_aggregator/internal/progress" + "llm_aggregator/internal/tokeniser" ) // ContentProcessor processes and prepares aggregated content for LLM analysis.@@ -224,34 +225,167 @@
return processed } -// EstimateTokenCount estimates token count for articles (rough approximation). -// Note: This is a rough estimate (1 token ≈ 4 characters for English). -// Actual tokenisation may vary. -func (cp *ContentProcessor) EstimateTokenCount(articles []map[string]any) int { - totalChars := 0 +// EstimateTokenCount estimates token count for articles using tiktoken. +// This is the accurate method using OpenAI's tokenisation. +func (cp *ContentProcessor) EstimateTokenCount(articles []map[string]any, model string) int { + totalTokens := 0 for _, article := range articles { for _, field := range []string{"title", "content", "author", "source_feed"} { if val, ok := article[field]; ok && val != nil { - if str, ok := val.(string); ok { - totalChars += len(str) + if str, ok := val.(string); ok && str != "" { + tokens, err := tokeniser.CountTokens(str, model) + if err != nil { + // Fallback to rough estimate on error + if cp.logger != nil { + cp.logger.Logf("Warning: token count error for %s: %v", field, err) + } + totalTokens += len(str) / 4 + continue + } + totalTokens += tokens } } } } - // Rough estimate: 1 token ≈ 4 characters for English - estimatedTokens := totalChars / 4 - if cp.logger != nil { - cp.logger.Logf("Estimated tokens: %d (~%d chars)", estimatedTokens, totalChars) + cp.logger.Logf("Estimated tokens: %d", totalTokens) } - return estimatedTokens + return totalTokens } // CreateConciseContext creates a concise context from articles, respecting token limits. -func (cp *ContentProcessor) CreateConciseContext(articles []map[string]any, maxTotalTokens int) string { +func (cp *ContentProcessor) CreateConciseContext(articles []map[string]any, maxTotalTokens int, model string) string { + if len(articles) == 0 { + return "No articles available." + } + + // Get the appropriate encoding for accurate token counting + encodingName, err := tokeniser.EncodingForModel(model) + if err != nil { + // Fallback to rough estimate if model not recognised + if cp.logger != nil { + cp.logger.Logf("Warning: unknown model %s, using rough token estimate", model) + } + return cp.createConciseContextRough(articles, maxTotalTokens) + } + + enc, err := tokeniser.GetEncoding(encodingName) + if err != nil { + if cp.logger != nil { + cp.logger.Logf("Warning: failed to get encoding: %v", err) + } + return cp.createConciseContextRough(articles, maxTotalTokens) + } + + contextParts := []string{} + currentTokens := 0 + + for i, article := range articles { + // Create article summary + articleText := fmt.Sprintf("Article %d: %s\n", i+1, article["title"]) + + // Add source if available + if source, ok := article["source_feed"].(string); ok && source != "" { + articleText += fmt.Sprintf("Source: %s\n", source) + } + + // Add publication date if available + if published, ok := article["published"]; ok { + switch pub := published.(type) { + case time.Time: + if !pub.IsZero() { + articleText += fmt.Sprintf("Published: %s\n", pub.Format("2006-01-02")) + } + case string: + articleText += fmt.Sprintf("Published: %s\n", pub) + } + } + + // Add content (we'll truncate based on actual tokens) + content := "" + if c, ok := article["content"].(string); ok { + content = c + } + + // First, check if we can add the header tokens + headerTokens := len(enc.Encode(articleText, nil, nil)) + + // Calculate remaining tokens for this article + remainingTokens := maxTotalTokens - currentTokens - headerTokens - 3 // -3 for separator + + if remainingTokens <= 0 { + if cp.logger != nil { + cp.logger.Logf( + "Reached token limit (%d). Included %d of %d articles.", + maxTotalTokens, i, len(articles), + ) + } + break + } + + // Convert token budget to approximate character limit (rough guide) + // Characters are typically 3-4x tokens for English + maxContentChars := remainingTokens * 4 + + if len(content) > maxContentChars { + // Need to encode and truncate more precisely + contentTokens := len(enc.Encode(content, nil, nil)) + if contentTokens > remainingTokens { + // Binary search for truncation point would be more accurate, + // but for performance, use the rough estimate + content = content[:maxContentChars] + "... [truncated]" + } + } + + articleText += fmt.Sprintf("Content: %s\n", content) + + // Count actual tokens for this article + articleTokens := len(enc.Encode(articleText, nil, nil)) + + // Double-check we haven't exceeded limit + if currentTokens+articleTokens > maxTotalTokens { + // Truncate content more aggressively + maxArticleChars := (maxTotalTokens - currentTokens - headerTokens - 20) * 4 + if maxArticleChars > 0 { + content = content[:maxArticleChars] + "... [truncated]" + articleText = fmt.Sprintf("Article %d: %s\n", i+1, article["title"]) + if source, ok := article["source_feed"].(string); ok && source != "" { + articleText += fmt.Sprintf("Source: %s\n", source) + } + articleText += fmt.Sprintf("Content: %s\n", content) + articleTokens = len(enc.Encode(articleText, nil, nil)) + } + + if currentTokens+articleTokens > maxTotalTokens { + if cp.logger != nil { + cp.logger.Logf( + "Reached token limit (%d). Included %d of %d articles.", + maxTotalTokens, i, len(articles), + ) + } + break + } + } + + contextParts = append(contextParts, articleText) + currentTokens += articleTokens + } + + context := strings.Join(contextParts, "\n---\n") + + if cp.logger != nil { + cp.logger.Logf("Created context with %d articles, ~%d tokens", len(contextParts), currentTokens) + } + + return context +} + +// createConciseContextRough creates a concise context using rough character-based estimation. +// Used as fallback when tiktoken is unavailable. +func (cp *ContentProcessor) createConciseContextRough(articles []map[string]any, maxTotalTokens int) string { if len(articles) == 0 { return "No articles available." }@@ -286,7 +420,8 @@ if c, ok := article["content"].(string); ok {
content = c } - maxContentTokens := maxTotalTokens / len(articles) // Rough allocation + // Rough allocation per article + maxContentTokens := maxTotalTokens / len(articles) maxContentChars := maxContentTokens * 4 if len(content) > maxContentChars {@@ -295,7 +430,7 @@ }
articleText += fmt.Sprintf("Content: %s\n", content) - // Estimate tokens for this article + // Rough estimate: 1 token ≈ 4 characters articleTokens := len(articleText) / 4 // Check if adding this article would exceed limit@@ -316,9 +451,8 @@
context := strings.Join(contextParts, "\n---\n") if cp.logger != nil { - cp.logger.Logf("Created context with %d articles, ~%d tokens", len(contextParts), currentTokens) + cp.logger.Logf("Created context with %d articles, ~%d tokens (rough estimate)", len(contextParts), currentTokens) } return context } -
@@ -108,7 +108,7 @@ // Step 3: Initialise LLM client
r.Progress.SetStage("Connecting to LLM") r.Progress.SetSubStage(fmt.Sprintf("Using model: %s", r.Model)) - deepseek, err := llm.NewLLMClient( + llm, err := llm.NewLLMClient( r.APIKey, "", // default base URL r.Model,@@ -118,13 +118,13 @@ )
if err != nil { return fmt.Errorf("error initialising LLM client: %w", err) } - deepseek.SetLogger(progCtx) // MODIFIED: Pass the new progress context + llm.SetLogger(progCtx) // Pass the new progress context // Step 4: Get summary from LLM r.Progress.SetStage("Getting summary") r.Progress.SetSubStage(fmt.Sprintf("Sending %d articles to LLM", len(processedArticles))) - summary, err := deepseek.SummariseArticles( + summary, err := llm.SummariseArticles( processedArticles, r.Prompt, r.SystemPrompt,
@@ -0,0 +1,161 @@
+package tokeniser + +import ( + "fmt" + "strings" + "sync" + + "github.com/pkoukk/tiktoken-go" +) + +// Encoding names for different model families +const ( + EncodingCl100kBase = "cl100k_base" // GPT-4, GPT-3.5-turbo, embeddings + EncodingO200kBase = "o200k_base" // GPT-4o, GPT-4.1, GPT-4.5 + EncodingP50kBase = "p50k_base" // Codex models, davinci-002, davinci-003 + EncodingR50kBase = "r50k_base" // GPT-3 models +) + +// tokenizerCache caches encodings to avoid repeated initialization overhead +type tokenizerCache struct { + mu sync.RWMutex + encodings map[string]*tiktoken.Tiktoken +} + +var ( + cache = &tokenizerCache{encodings: make(map[string]*tiktoken.Tiktoken)} + modelLock sync.RWMutex +) + +// GetEncoding returns a Tiktoken encoding for the given encoding name. +func GetEncoding(encodingName string) (*tiktoken.Tiktoken, error) { + // Check cache first + cache.mu.RLock() + if enc, ok := cache.encodings[encodingName]; ok { + cache.mu.RUnlock() + return enc, nil + } + cache.mu.RUnlock() + + // Initialise encoding + enc, err := tiktoken.GetEncoding(encodingName) + if err != nil { + return nil, fmt.Errorf("failed to get encoding %s: %w", encodingName, err) + } + + // Cache it + cache.mu.Lock() + cache.encodings[encodingName] = enc + cache.mu.Unlock() + + return enc, nil +} + +// EncodingForModel returns the appropriate encoding name for a given model. +func EncodingForModel(model string) (string, error) { + model = strings.ToLower(model) + + // GPT-4o and newer use o200k_base + if strings.Contains(model, "gpt-4o") || + strings.Contains(model, "gpt-4.1") || + strings.Contains(model, "gpt-4.5") || + strings.HasPrefix(model, "gpt-4o") || + strings.HasPrefix(model, "gpt-4.1") || + strings.HasPrefix(model, "gpt-4.5") { + return EncodingO200kBase, nil + } + + // GPT-4 family uses cl100k_base + if strings.HasPrefix(model, "gpt-4") { + return EncodingCl100kBase, nil + } + + // GPT-3.5-turbo uses cl100k_base + if strings.Contains(model, "gpt-3.5-turbo") { + return EncodingCl100kBase, nil + } + + // Embedding models use cl100k_base + if strings.Contains(model, "text-embedding") { + return EncodingCl100kBase, nil + } + + // Codex and davinci models use p50k_base + if strings.Contains(model, "code-") || + strings.Contains(model, "davinci") { + return EncodingP50kBase, nil + } + + // Default to cl100k_base (most common for modern models) + return EncodingCl100kBase, nil +} + +// CountTokens counts the tokens in a text string using the appropriate encoding. +func CountTokens(text string, model string) (int, error) { + encodingName, err := EncodingForModel(model) + if err != nil { + return 0, err + } + + enc, err := GetEncoding(encodingName) + if err != nil { + return 0, fmt.Errorf("failed to get encoding: %w", err) + } + + tokens := enc.Encode(text, nil, nil) + return len(tokens), nil +} + +// CountMessagesTokens counts tokens for chat API messages. +// Based on OpenAI's official token counting method. +func CountMessagesTokens(messages []Message, model string) (int, error) { + encodingName, err := EncodingForModel(model) + if err != nil { + return 0, err + } + + enc, err := GetEncoding(encodingName) + if err != nil { + return 0, fmt.Errorf("failed to get encoding: %w", err) + } + + var tokensPerMessage, tokensPerName int + switch { + case strings.Contains(model, "gpt-3.5-turbo-0613"), + strings.Contains(model, "gpt-3.5-turbo-16k-0613"), + strings.Contains(model, "gpt-4-0314"), + strings.Contains(model, "gpt-4-32k-0314"), + strings.Contains(model, "gpt-4-0613"), + strings.Contains(model, "gpt-4-32k-0613"): + tokensPerMessage = 3 + tokensPerName = 1 + case strings.Contains(model, "gpt-3.5-turbo-0301"): + tokensPerMessage = 4 + tokensPerName = -1 + default: + // Use default values for other models + tokensPerMessage = 3 + tokensPerName = 1 + } + + numTokens := 0 + for _, msg := range messages { + numTokens += tokensPerMessage + numTokens += len(enc.Encode(msg.Content, nil, nil)) + numTokens += len(enc.Encode(msg.Role, nil, nil)) + numTokens += len(enc.Encode(msg.Name, nil, nil)) + if msg.Name != "" { + numTokens += tokensPerName + } + } + numTokens += 3 // Every reply is primed with <|start|>assistant<|message|> + + return numTokens, nil +} + +// Message represents a chat message for token counting. +type Message struct { + Role string + Content string + Name string +}
@@ -8,6 +8,7 @@ "time"
"github.com/charmbracelet/bubbles/progress" "github.com/charmbracelet/bubbles/spinner" + "github.com/charmbracelet/bubbles/viewport" tea "github.com/charmbracelet/bubbletea" "github.com/charmbracelet/lipgloss" "llm_aggregator/internal/runtime"@@ -19,27 +20,30 @@ maxWidth = 80
) var ( - subtle = lipgloss.AdaptiveColor{Light: "#D9DCCF", Dark: "#383838"} - highlight = lipgloss.AdaptiveColor{Light: "#874BFD", Dark: "#7D56F4"} - special = lipgloss.AdaptiveColor{Light: "#43BF6D", Dark: "#73F59F"} + colorSubtle = lipgloss.Color("7") // Gray (dim text) + colorHighlight = lipgloss.Color("13") // Magenta (status) + colorSuccess = lipgloss.Color("2") // Green (completion) + colorError = lipgloss.Color("1") // Red (errors) + colorGradientStart = lipgloss.Color("205") // Pink + colorGradientEnd = lipgloss.Color("226") // Yellow titleStyle = lipgloss.NewStyle(). MarginLeft(2). MarginRight(2). Padding(0, 1). Italic(true). - Foreground(lipgloss.Color("#FFF7DB")) + Foreground(lipgloss.Color("15")) infoStyle = lipgloss.NewStyle(). - Foreground(subtle). + Foreground(colorSubtle). Italic(true) statusStyle = lipgloss.NewStyle(). - Foreground(highlight). + Foreground(colorHighlight). Bold(true) progressBarStyle = lipgloss.NewStyle(). - Foreground(special) + Foreground(colorSuccess) stepNames = []string{ "Initialising",@@ -67,6 +71,7 @@
type Model struct { progress progress.Model spinner spinner.Model + viewport viewport.Model runtime *runtime.Runtime currentStep int totalSteps int@@ -80,18 +85,19 @@ done bool
errorMsg string articlesCount int processedCount int + showSummary bool } func New(rt *runtime.Runtime) *Model { prog := progress.New( - progress.WithScaledGradient("#FF7CCB", "#FDFF8C"), + progress.WithSolidFill("colorSuccess"), progress.WithWidth(40), progress.WithoutPercentage(), ) sp := spinner.New() sp.Spinner = spinner.Dot - sp.Style = lipgloss.NewStyle().Foreground(lipgloss.Color("#FF7CCB")) + sp.Style = lipgloss.NewStyle().Foreground(colorHighlight) return &Model{ progress: prog,@@ -102,14 +108,19 @@ totalSteps: len(stepNames) - 1, // Last step is "Complete"
status: "Initialising...", subStatus: "Loading configuration", startTime: time.Now(), + showSummary: false, } } func (m *Model) Init() tea.Cmd { + // Initialise the viewport with default size (will be resized on WindowSizeMsg) + m.viewport = viewport.New(80, 20) + m.viewport.MouseWheelEnabled = true + return tea.Batch(m.spinner.Tick, m.startProcessing) } -// MODIFIED: This is now the standard bubbletea command pattern. +// This is now the standard bubbletea command pattern. // The bubbletea runtime will execute this function in a goroutine. // We no longer need to manage the goroutine or use program.Send(). func (m *Model) startProcessing() tea.Msg {@@ -120,6 +131,19 @@
func (m *Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) { switch msg := msg.(type) { case tea.KeyMsg: + // Let the viewport handle navigation keys if showing summary + if m.showSummary { + var cmd tea.Cmd + m.viewport, cmd = m.viewport.Update(msg) + if cmd != nil { + return m, cmd + } + // If viewport handled a key, we're done + switch msg.String() { + case "j", "down", "k", "up", "g", "G", " ", "b": + return m, nil + } + } switch msg.String() { case "q", "ctrl+c": return m, tea.Quit@@ -128,6 +152,11 @@ case tea.WindowSizeMsg:
m.width = msg.Width m.height = msg.Height m.progress.Width = min(msg.Width-padding*2-4, maxWidth) + // Resize viewport for summary display + if m.showSummary { + m.viewport.Width = msg.Width - padding*2 + m.viewport.Height = msg.Height - 10 // Leave space for header/footer + } case progressMsg: // This message type seems unused, but leaving it in case. progress := float64(msg)@@ -172,8 +201,19 @@ m.done = true
m.currentStep = m.totalSteps // Go to "Complete" step m.status = stepNames[m.totalSteps] m.subStatus = "Summary generated successfully!" + // Enable summary viewport if there's content to display + if m.runtime.Summary != "" { + m.showSummary = true + // Resize viewport to fit available space + m.viewport.Width = m.width - padding*2 + m.viewport.Height = m.height - 15 // Leave space for header/footer + // Wrap content to viewport width before setting + contentWidth := m.viewport.Width + summaryContent := "📝 Summary\n\n" + wrapText(m.runtime.Summary, contentWidth) + m.viewport.SetContent(summaryContent) + } } - // MODIFIED: Capture the final elapsed time. + // Capture the final elapsed time. m.elapsed = time.Since(m.startTime) // We are done, so we return a command to quit the program automatically. // The user can see the final summary. If we want the user to quit manually,@@ -193,13 +233,18 @@ if m.width == 0 {
return "Initialising..." } + // If we're showing the summary with scrolling, use a different layout + if m.showSummary { + return m.viewSummary() + } + var sb strings.Builder // Title sb.WriteString(titleStyle.Render("🤖 LLM Aggregator")) sb.WriteString("\n\n") - // MODIFIED: If done, show 100% progress. + // If done, show 100% progress. var progressPercent float64 if m.done && m.errorMsg == "" { progressPercent = 1.0@@ -216,11 +261,11 @@
// Status with spinner sb.WriteString(statusStyle.Render("Status: ")) - // MODIFIED: Replace spinner with a checkmark when done. + // Replace spinner with a checkmark when done. if m.done && m.errorMsg == "" { - sb.WriteString(lipgloss.NewStyle().Foreground(special).Render("✓")) + sb.WriteString(lipgloss.NewStyle().Foreground(colorSuccess).Render("✓")) } else if m.done && m.errorMsg != "" { - sb.WriteString(lipgloss.NewStyle().Foreground(lipgloss.Color("#FF0000")).Render("✗")) + sb.WriteString(lipgloss.NewStyle().Foreground(colorError).Render("✗")) } else { sb.WriteString(m.spinner.View()) }@@ -258,14 +303,14 @@
// Error message (if any) if m.errorMsg != "" { errorStyle := lipgloss.NewStyle(). - Foreground(lipgloss.Color("#FF0000")). + Foreground(colorError). Bold(true) sb.WriteString(errorStyle.Render("Error: ")) sb.WriteString(m.errorMsg) sb.WriteString("\n\n") } - // MODIFIED: Simplified help text logic + // Simplified help text logic if m.done { // The summary is not part of the TUI, it's printed to stdout *after* the TUI exits. // The TUI's job is just to show the progress.@@ -275,33 +320,6 @@ } else {
sb.WriteString(infoStyle.Render("Press 'q' or Ctrl+C to quit")) } - // If the runtime has a summary and we are done, we can show it here. - // However, the typical flow is for the TUI to finish, and then the main - // function prints the final output. If we want to show it *in* the TUI, - // we would add this block: - if m.done && m.runtime.Summary != "" { - summaryTitleStyle := lipgloss.NewStyle().Bold(true).Underline(true) - sb.WriteString(summaryTitleStyle.Render("Summary")) - sb.WriteString("\n") - - // Calculate a safe width for wrapping (terminal width minus some margin) - wrapWidth := max(m.width-4, - // Sane minimum fallback - 20) - - // Create a style that enforces word-wrapping - summaryStyle := lipgloss.NewStyle(). - Width(wrapWidth) - - // Render the summary through the style to apply the wrapping - sb.WriteString(summaryStyle.Render(m.runtime.Summary)) - sb.WriteString("\n\n") - - sb.WriteString(infoStyle.Render("Press 'q' to exit.")) - } else if m.done { - sb.WriteString(infoStyle.Render("Press 'q' to exit.")) - } - return sb.String() }@@ -339,3 +357,47 @@ return func() tea.Msg {
return ErrorMsg{Error: err} } } + +// viewSummary renders the summary view with scrolling support. +func (m *Model) viewSummary() string { + var sb strings.Builder + + // Header + sb.WriteString(titleStyle.Render("🤖 LLM Aggregator")) + sb.WriteString("\n\n") + + // Show completion status + sb.WriteString(statusStyle.Render("✓ Complete")) + sb.WriteString(" ") + sb.WriteString(m.subStatus) + sb.WriteString("\n\n") + + // Elapsed time and article stats + sb.WriteString(infoStyle.Render(fmt.Sprintf("Articles: %d | Processed: %d | Elapsed: %v", + m.articlesCount, m.processedCount, m.elapsed.Round(time.Second)))) + sb.WriteString("\n\n") + + // Scroll progress indicator + scrollPercent := int(m.viewport.ScrollPercent() * 100) + if m.viewport.TotalLineCount() > m.viewport.VisibleLineCount() { + sb.WriteString(infoStyle.Render(fmt.Sprintf("Scroll: %d%% (%d/%d lines)", + scrollPercent, m.viewport.TotalLineCount()-m.viewport.VisibleLineCount()-m.viewport.YOffset, + m.viewport.TotalLineCount()))) + sb.WriteString("\n\n") + } + + // Viewport (scrollable summary) + sb.WriteString(m.viewport.View()) + sb.WriteString("\n") + + // Navigation help + sb.WriteString(infoStyle.Render("Navigate: ↑/↓ or j/k (scroll) | Space/B (page) | g/G (start/end) | q (quit)")) + + return sb.String() +} + +// wrapText wraps text to a given width using lipgloss. +func wrapText(text string, width int) string { + style := lipgloss.NewStyle().Width(width) + return style.Render(text) +}