README
¶
rss-feeds-go
A go utility package for handling RSS/Atom feeds.
Features
- Fetch feeds from URLs
- RSS feeds (0.90 to 2.0)
- Atom feeds (0.3, 1.0))
- JSON feeds (1.0, 1.1)
- Manually
- Periodically
- Cache fetched feeds locally
- In memory
- In SQLite3 file
- Summarize contents of fetched feed items with Google Gemini API
- Save summarized contents locally
- In memory
- In SQLite3 file
- Transfer summarized contents to somewhere else
- Save summarized contents locally
- Convert cached feeds as RSS XML
Usages / Samples
Usage:
package main
import (
"log"
rf "github.com/meinside/rss-feeds-go"
)
func main() {
if client, err := rf.NewClientWithDB(
[]string{"abcdefghijklmnopqrstuvwxyz0123456789"}, // google ai api keys (for rotation)
[]string{ // feeds' urls
"https://hnrss.org/newest?points=50",
"https://www.hackster.io/news.atom",
},
"./database.sqlite3", // sqlite3 db filepath
); err == nil {
// TODO: do something with `client`
}
}
Other sample applications are in ./samples/ directory.
Documentation
¶
Overview ¶
Package rf for handling RSS feeds
Index ¶
- Constants
- func Prettify(v any) string
- func StandardizeJSON(b []byte) ([]byte, error)
- type CachedItem
- type Client
- func (c *Client) DeleteOldCachedItems()
- func (c *Client) FetchFeeds(ctx context.Context, ignoreAlreadyCached bool, ...) (feeds []gofeed.Feed, err error)
- func (c *Client) ListCachedItems(includeItemsMarkedAsRead bool) []CachedItem
- func (c *Client) MarkCachedItemsAsRead(items []CachedItem)
- func (c *Client) PublishXML(title, link, description, author, email string, items []CachedItem) (bytes []byte, err error)
- func (c *Client) SetDesiredLanguage(lang string)
- func (c *Client) SetGoogleAIModels(models []string)
- func (c *Client) SetSummarizeIntervalSeconds(seconds int)
- func (c *Client) SetVerbose(v bool)
- func (c *Client) SummarizeAndCacheFeeds(feeds []gofeed.Feed, urlScrapper ...*ssg.Scrapper) (err error)
- type FeedsItemsCache
Constants ¶
const ( ErrorPrefixSummaryFailedWithError = `Summary failed with error` PublishContentType = `application/rss+xml` )
Variables ¶
This section is empty.
Functions ¶
func StandardizeJSON ¶
StandardizeJSON standardizes given JSON (JWCC) bytes.
Types ¶
type CachedItem ¶
type CachedItem struct {
gorm.Model
Title string
Link string // url to the original article
Comments string // url to the community comments
GUID string `gorm:"uniqueIndex"`
Author string
PublishDate string
Description string
Summary string
MarkedAsRead bool `gorm:"index"`
}
CachedItem is a struct for a cached item
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client struct
func NewClientWithDB ¶
func NewClientWithDB( googleAIAPIKeys []string, feedsURLs []string, dbFilepath string, ) (client *Client, err error)
NewClientWithDB returns a new client with SQLite DB cache.
func (*Client) DeleteOldCachedItems ¶
func (c *Client) DeleteOldCachedItems()
DeleteOldCachedItems deletes old cached items.
func (*Client) FetchFeeds ¶
func (c *Client) FetchFeeds( ctx context.Context, ignoreAlreadyCached bool, ignoreItemsPublishedBeforeDays uint, ) (feeds []gofeed.Feed, err error)
FetchFeeds fetches feeds.
func (*Client) ListCachedItems ¶
func (c *Client) ListCachedItems(includeItemsMarkedAsRead bool) []CachedItem
ListCachedItems lists cached items.
func (*Client) MarkCachedItemsAsRead ¶
func (c *Client) MarkCachedItemsAsRead(items []CachedItem)
MarkCachedItemsAsRead marks given cached items as read.
func (*Client) PublishXML ¶
func (c *Client) PublishXML( title, link, description, author, email string, items []CachedItem, ) (bytes []byte, err error)
PublishXML returns XML bytes (application/rss+xml) of given cached items.
func (*Client) SetDesiredLanguage ¶
SetDesiredLanguage sets the client's desired language for summaries.
func (*Client) SetGoogleAIModels ¶ added in v0.3.2
SetGoogleAIModels sets the client's Google AI models.
func (*Client) SetSummarizeIntervalSeconds ¶
SetSummarizeIntervalSeconds sets the client's summarize interval seconds.
func (*Client) SetVerbose ¶
SetVerbose sets the client's verbose mode.
func (*Client) SummarizeAndCacheFeeds ¶
func (c *Client) SummarizeAndCacheFeeds( feeds []gofeed.Feed, urlScrapper ...*ssg.Scrapper, ) (err error)
SummarizeAndCacheFeeds summarizes given feeds items and caches them.
Each feed item will be summarized with a timeout of `summarizeTimeoutSeconds` seconds.
If summary fails, the original content prepended with the error message will be cached.
If there was a retriable error(eg. model overloads), it will return immediately. (remaining feed items will be retried later)
type FeedsItemsCache ¶
type FeedsItemsCache interface {
Exists(guid string) bool
Save(item gofeed.Item, title, summary string)
Fetch(guid string) *CachedItem
MarkAsRead(guid string)
List(includeItemsMarkedAsRead bool) []CachedItem
DeleteOlderThan1Month()
SetVerbose(v bool)
}
FeedsItemsCache is an interface of feeds items' cache
Source Files
Directories