finds.readers.alfred

Class to access ALFRED/FRED apis, and FRED-MD/FRED-QD

FRED, ALFRED: St Louis Fed api’s, with revision vintages
FRED-MD, FRED-QD: See McCracken website at St Louis Fed
- https://research.stlouisfed.org/econ/mccracken/fred-databases/

Data are cached in memory. Vintage releases retrieved by sequence or date window.

MIT License

class finds.readers.alfred.Alfred(api_key: str, savefile: str = '', convert_date: bool = True, verbose=1)[source]

Bases: object

Base class for Alfred/Fred access, and manipulating retrieved data series

Parameters:

api_key – API key string registered with FRED
savefile – Name of local file to auto-save retrieved series
convert_date – Whether to convert date to int
verbose – whether to display messages

Variables:

tcode – Reference dict of transformation codes
_fred_url() – Formatter to construct FRED api query from key-value string
_alfred_url() – Formatter to construct vintage FRED api query
_category_url() – Formatter to construct FRED category api query

__call__(series_id: str, start: int = 0, end: int = 0, release: int | DateOffset = 0, vintage: int = 99991231, label: str = '', realtime: bool = False, freq: str = '', **kwargs) → Series | None[source]

Retrieve from cache, else call FRED api, and apply transforms

Parameters:

series_id – Label of series to retrieve
start – Start and end period dates (inclusive) to keep
end – Start and end period dates (inclusive) to keep
label – New label to rename returned series
release – release number (1 first, 0 latest), or latest up to date offset
vintage – Latest realtime_start date of observations to keep
realtime – Whether to return realtime_start and realtime_end
freq – Resample and replace date index with at periodic frequency; in {‘M’, ‘A’. ‘Q’, ‘D’, ‘Y’}, else empty ‘’ to auto select
kwargs – transformations key-value pairs diff: Number of difference operations to apply log: Number of log operations to apply pct_change: Number of pct_change to apply

Returns:

transformed values; name is set to label if provided else series_id

__getitem__(series_id: str) → Dict[source]: Get observations and metadata from cache for {series_id}

categories(series_id: str | List[str])[source]: Returns categories parent_id’s of series

clear()[source]: Clear internal memory cache of previously loaded series

static construct_series(observations: DataFrame, vintage: int = 99991231, release: int | DateOffset = 0, start: int = 0, end: int = 99991231, freq: str = '', convert_date: bool = True) → Series[source]

Helper to construct series from given full observations dataframe

Parameters:

observations – DataFrame from FRED ‘series/observations’ api call
release – Sequence num (0 for latest), or latest to max date offset
vintage – Latest realtime_start date (inclusive) allowed
start – Start and end period dates (inclusive) to keep
end – Start and end period dates (inclusive) to keep
freq – in {‘M’, ‘A’. ‘Q’, ‘D’, ‘Y’}, else empty ‘’ to auto select
convert_date – Whether to convert date format to int

Returns:

value as of each period date, optionally indexed by realtime_start

date_spans(series_id: str = 'USREC', threshold: int = 0) → List[Tuple[Timestamp, Timestamp]][source]

Return date spans as tuples of Timestamp

Parameters:

series_id – Name of series
threshold – Values of series strictly above are included in date span

dump(savefile: str = '') → int[source]: Save all memory-cached series to pickle file, return number saved

get_category(category_id: str, api_key: str = '')[source]

Retrieve category information by calling related API

Parameters:

category_id – id of category to retrieve
api_key – credentials to FRED
verbose – verbose flag

Notes:

Uses request_category method to call these FRED API on given category_id

‘category’ API gets meta information
‘category/series’ API gets series_ids
‘category/children’ API gets child categories

get_series(series_id: str | List[str], api_key: str = '', start: int = 0, end: int = 0) → int[source]

Retrieve metadata and full observations of a series with FRED api

Parameters:: series_id – list of ids of series to retrieve
Returns:: length of observations dataframe

header(series_id: str | List[str], column: str = 'title')[source]: Returns title (or other column) from last meta record of a series

keys()[source]: Return id names of all loaded series data

load(savefile: str = '')[source]: Load all series to memory from pickle file, return number loaded

static multpl(page: str) → DataFrame[source]

Helper method to retrieve shiller series by scraping multpl.com

Parameters:: page – Web page name in {‘s-p-500-dividend-yield’, ‘shiller-pe’}
Returns:: Dataframe of monthly series (for updating FRED-MD)

observations(series_id: str, date: int, freq: str = '') → DataFrame[source]

Return all release of observations for a series and date

Parameters:

series_id – series name
date – date of series
freq – in {‘M’, ‘A’. ‘Q’, ‘D’, ‘Y’, ‘’}

pop(series_id: str) → Dict[str, DataFrame][source]: Pop and return desired series, then clear from memory cache

static popular(page: int = 1)[source]: Static method to web scrape popular series names, by page number

request_category(category_id: str, api: str = 'category', api_key: str = '', verbose: int = -1, **kwargs) → Dict[source]: Request ‘category’ related API for category data

request_series(series_id: str, api_key: str = '', start: int = 0, end: int = 0, verbose: int = -1) → DataFrame[source]: Requests ‘series’ API for series metadata

request_series_categories(series_id: str, api_key: str = '', verbose: int = -1) → DataFrame[source]: Request series/categories API for series category ids

request_series_observations(series_id: str, api_key: str = '', start: int = 0, end: int = 0, archive: bool = False, verbose: int = -1) → DataFrame[source]: Request series/observations API for full observations data

splice(series_id: str, start: int = 19590101, freq: str = 'M') → Series[source]

Retrieve raw series to update a FRED-MD series

e.g. Shiller series:

http://www.econ.yale.edu/~shiller/data/ie_data.xls
multpl.com

static transform(data: DataFrame, tcode: int | str = 1, freq: str = '', **kwargs) → DataFrame[source]

Static method to apply time series transformations

Parameters:

data – DataFrame of input data
tcode – int transformation code in {1, …, 7} or str. describing ho to apply sequence of operators to make series stationary
freq – str periodicity of dates in {‘M’, ‘Q’, ‘A’}
kwargs – transformation operators and number of times

Transformation operators:

log (int): Number of times to take log (default 0)

diff (int): Number of times to take difference (default 0)

pct_change (bool): Whether to apply pct_change operator

periods (int): lags for pct_change or diff operator (default 1)

annualize (int): annualization multiplier (default 1)

values(columns: List[str] = ['id', 'observation_start', 'observation_end', 'frequency_short', 'title', 'popularity', 'seasonal_adjustment_short', 'units_short']) → DataFrame[source]

Return headers (last metadata row) of all loaded series

Parameters:: columns – subset of header columns to return
Returns:: DataFrame of latest headers of all series loaded

splice_: Dict = {'AMDMNO': 'DGORDER', 'CLAIMS': 'ICNSA', 'COMPAPFF': 'CPFF', 'CONSPI': [<function Series.truediv>, 'NONREVSL', 'PI'], 'CP3M': 'CPF3M', 'CPF3MTB3M': [<function Series.sub>, 'CPF3M', 'DTB3'], 'HWI': 'JTSJOL', 'HWIURATIO': [<function Series.truediv>, 'JTSJOL', 'UNEMPLOY'], 'OILPRICE': 'MCOILWTICO', 'RETAIL': 'RSAFS', 'S&P 500': 'SP500', 'S&P PE ratio': 'shiller-pe', 'S&P div yield': 's-p-500-dividend-yield'}

tcode = {1: {'diff': 0, 'log': 0}, 2: {'diff': 1, 'log': 0}, 3: {'diff': 2, 'log': 0}, 4: {'diff': 0, 'log': 1}, 5: {'diff': 1, 'log': 1}, 6: {'diff': 2, 'log': 1}, 7: {'diff': 1, 'log': 0, 'pct_change': True}, 'lin': {'diff': 0, 'log': 0}, 'chg': {'diff': 1, 'log': 0}, 'ch1': {'diff': 0, 'log': 0, 'pct_change': True, 'periods': 12}, 'pch': {'diff': 0, 'log': 0, 'pct_change': True}, 'pc1': {'diff': 0, 'log': 0, 'pct_change': True, 'periods': 12}, 'pca': {'diff': 1, 'log': 1, 'annualize': 12}, 'cch': {'diff': 1, 'log': 1}, 'cca': {'diff': 1, 'log': 1, 'annualize': 12}, 'log': {'diff': 0, 'log': 1}}

finds.readers.alfred.csv_from_zip(zip_path: str, filename: str) → DataFrame | None[source]

finds.readers.alfred.fred_md(vintage: int | str = 0, url: str = 'https://www.stlouisfed.org/-/media/project/frbstl/stlouisfed/research/fred-md/', verbose: int = 1) → DataFrame[source]

Retrieve and parse current or vintage csv from McCracken FRED-MD site

Parameters:

vintage – file name relative to base url or zipfile, or int date YYYYMM
url – base name of url, local file path or zipfile archive

Returns:

DataFrame indexed by end-of-month date

Notes:

if vintage is int: derive vintage csv file name from input date YYYYMM
if url is ‘’: derive subfolder or zip archive name, from vintage

Examples:

>>> md_df, mt = fred_md()  # current in monthly/current.csv
>>> md_df, mt = fred_md('Historical FRED-MD Vintages Final/2013-12.csv',
                        url=fred_md_url+'Historical_FRED-MD.zip') # pre-2015
>>> md_df, mt = fred_md('monthly/2015-05.csv',
                        url=fred_md_url+'FRED_MD.zip')      # post-2015

finds.readers.alfred.fred_qd(vintage: int | str = 0, url: str = 'https://www.stlouisfed.org/-/media/project/frbstl/stlouisfed/research/fred-md/', verbose: int = 1)[source]

Retrieve and parse current or vintage csv from McCracken FRED-MD site

Parameters:

vintage – file name relative to base url or zipfile, or int date YYYYMM
url – base name of url, local file path or zipfile archive

Returns:

DataFrame indexed by end-of-month date

Notes:

if vintage is int: derive vintage csv file name from input date YYYYMM
if url is ‘’: derive subfolder or zip archive name, from vintage