Add feed aggregation and embedding generation features

README.md

@@ -1,32 +1,71 @@
 # Augor
 
-> Drilling beneath surface sentiment to extract trading signals from financial news.
+> Using embeddings to find meaningful patterns in financial news
 
 ## What is this?
 
-Augor uses Large Language Models to interpret business events and generate trading signals. Instead of just counting positive/negative words, it understands what events *mean* for markets.
+Augor is a Flutter desktop application that aggregates financial news from multiple RSS sources and uses OpenAI embeddings to cluster related articles about the same business events.  Instead of manually tracking dozens of news sources, Augor automatically groups stories so you can see what's actually happening in markets.
 
-Traditional sentiment analysis: "This news mentions 'partnership' → positive"  
-Augor: "This partnership expands market access but dilutes margins → mixed signal"
+## Current Features
 
-## Why?
+**RSS Feed Aggregation**
+- Supports RSS and Atom feeds
+- Pre-configured with 20+ major business news sources (Reuters, Bloomberg, WSJ, etc)
+- Add custom feeds through the settings interface
+- Enable/disable feeds individually
 
-Traditional NLP methods are limited:
-- Can't understand context or nuance
-- Miss complex implications of business events
-- Struggle with negation and sarcasm
+**AI-Powered Processing**
+- Generates embeddings using OpenAI's text-embedding-3-small model
+- Filters articles for business relevance using keyword similarity
+- Groups related articles about the same event using cosine similarity
+- Exports results as JSON for further analisis
 
-LLMs can reason about *why* news matters, not just classify it as positive/negative.
+**Settings Management**
+- Configure OpenAI API key
+- Manage RSS feed sources
+- Set custom storage location for output files
 
-## What it does
+## Technical Stack
 
-1. Aggregates news from multiple sources
-2. Interprets business events (mergers, partnerships, product launches)
-3. Generates trading signals based on reasoned analysis
-4. Backtests performance against traditional approaches
+- **Flutter** - Cross-platform desktop app (macOS, Windows, Linux)
+- **shadcn_flutter** - UI component library
+- **OpenAI API** - Text embeddings for semantic similarity
+- **Provider** - State managment
+- **go_router** - Navigation
+
+## Setup
+
+1. Clone the repository
+2. Install dependencies:
+   ```bash
+   flutter pub get
+   ```
+3. Create a `.env` file with your OpenAI API key (or configure it in the app settings)
+4. Run the app:
+   ```bash
+   flutter run
+   ```
+
+## How it works
+
+1. **Aggregate** - Fetches articles from all enabled RSS feeds
+2. **Embed** - Generates vector embeddings for article titles and descriptions
+3. **Filter** - Removes articles not relevant to business/finance using keyword matching
+4. **Cluster** - Groups similar articles (cosine similarity >= 0.7) to identify events
+5. **Export** - Saves results to JSON files in your configured storage location
+
+## Output Files
+
+The app generates several JSON files in your storage directory:
+
+- `aggregated_feed.json` - All fetched articles
+- `enriched_aggregated_feed.json` - Articles with embeddings
+- `relevant_aggregated_feed.json` - Filtered relevant articles
+- `grouped_relevant_aggregated_feed.json` - Articles clustered by event
+- `readable_*.json` - Human-readable versions without embeddings
 
 ## Author
 
-Benjamin Watt  
-Supervisor: Panagiotis Kanellopoulos  
+Benjamin Watt
+Supervisor: Panagiotis Kanellopoulos
 University of Essex