{
"cells": [
{
"cell_type": "markdown",
"id": "93c38816",
"metadata": {},
"source": [
"# Economic Indicators\n",
"\n",
"_What we learn from history is that people don't learn from history_ - Warren Buffett \n",
"\n",
"Economic data is fundamental to financial analysis, policymaking, and investment strategies. However, many economic indicators are subject to revisions, meaning initial estimates may change over time as more accurate data becomes available. Understanding these revisions is crucial for interpreting past economic conditions, refining forecasting models, and making informed decisions. We explore retrieving data from online sources such as the Federal Reserve Economic Data (FRED), its archival counterpart (ALFRED), and key derived datasets such as FRED-MD and FRED-QD. Additionally, we examine the impact of data revisions on critical economic indicators like Total Nonfarm Payrolls (PAYEMS), and methods for detecting outliers in historical data."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ca23887b",
"metadata": {},
"outputs": [],
"source": [
"# By: Terence Lim, 2020-2025 (terence-lim.github.io)\n",
"import numpy as np\n",
"import pandas as pd\n",
"from pandas import DataFrame, Series\n",
"import matplotlib.pyplot as plt\n",
"import textwrap\n",
"from finds.readers import Alfred, fred_md, fred_qd\n",
"from finds.utils import plot_date, plot_groupbar\n",
"from finds.recipes import is_outlier\n",
"from datetime import datetime\n",
"from pprint import pprint\n",
"from secret import credentials\n",
"VERBOSE = 0\n",
"# %matplotlib qt"
]
},
{
"cell_type": "markdown",
"id": "3f420d36",
"metadata": {},
"source": [
"## FRED\n",
"\n",
"Federal Reserve Economic Data (FRED) is a widely used online database maintained by the Federal Reserve Bank of St. Louis, providing access to hundreds of thousands of economic data series from national and international sources. Users can retrieve data via the website, an Excel add-in, or API calls. \n",
"\n",
"## Retrieving data from websites\n",
"\n",
"Economic data can be retrieved from the web through several methods: \n",
"\n",
"1. **Downloading structured files** – Many websites provide data in formats like CSV, Excel, or JSON, making it easy to import into analytical tools. \n",
"2. **Web scraping** – Extracting information directly from web pages by identifying specific HTML tags or text patterns. \n",
"3. **Using APIs** – Some platforms, including FRED, offer APIs that allow developers to automate data retrieval via structured queries. \n"
]
},
{
"cell_type": "markdown",
"id": "021b38a7",
"metadata": {},
"source": [
"### Download structured files \n",
"\n",
"Many economic data providers allow users to download pre-structured files containing historical and current data. These files often include metadata, timestamps, and adjustment information. \n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "9f332383",
"metadata": {},
"outputs": [],
"source": [
"# This URL is the location of the FRED-MD csv file from the St Louis FRED\n",
"url = 'https://www.stlouisfed.org/-/media/project/frbstl/stlouisfed/research/fred-md/monthly/current.csv'"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "30ef7573",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" sasdate | \n",
" RPI | \n",
" W875RX1 | \n",
" DPCERA3M086SBEA | \n",
" CMRMTSPLx | \n",
" RETAILx | \n",
" INDPRO | \n",
" IPFPNSS | \n",
" IPFINAL | \n",
" IPCONGD | \n",
" ... | \n",
" DNDGRG3M086SBEA | \n",
" DSERRG3M086SBEA | \n",
" CES0600000008 | \n",
" CES2000000008 | \n",
" CES3000000008 | \n",
" UMCSENTx | \n",
" DTCOLNVHFNM | \n",
" DTCTHFNM | \n",
" INVEST | \n",
" VIXCLSx | \n",
"
\n",
" \n",
" \n",
" \n",
" 0 | \n",
" Transform: | \n",
" 5.000 | \n",
" 5.0 | \n",
" 5.000 | \n",
" 5.000000e+00 | \n",
" 5.00000 | \n",
" 5.0000 | \n",
" 5.0000 | \n",
" 5.0000 | \n",
" 5.0000 | \n",
" ... | \n",
" 6.000 | \n",
" 6.000 | \n",
" 6.00 | \n",
" 6.00 | \n",
" 6.00 | \n",
" 2.0 | \n",
" 6.00 | \n",
" 6.00 | \n",
" 6.0000 | \n",
" 1.0000 | \n",
"
\n",
" \n",
" 1 | \n",
" 1/1/1959 | \n",
" 2583.560 | \n",
" 2426.0 | \n",
" 15.188 | \n",
" 2.766768e+05 | \n",
" 18235.77392 | \n",
" 21.9616 | \n",
" 23.3868 | \n",
" 22.2620 | \n",
" 31.6664 | \n",
" ... | \n",
" 18.294 | \n",
" 10.152 | \n",
" 2.13 | \n",
" 2.45 | \n",
" 2.04 | \n",
" NaN | \n",
" 6476.00 | \n",
" 12298.00 | \n",
" 84.2043 | \n",
" NaN | \n",
"
\n",
" \n",
" 2 | \n",
" 2/1/1959 | \n",
" 2593.596 | \n",
" 2434.8 | \n",
" 15.346 | \n",
" 2.787140e+05 | \n",
" 18369.56308 | \n",
" 22.3917 | \n",
" 23.7024 | \n",
" 22.4549 | \n",
" 31.8987 | \n",
" ... | \n",
" 18.302 | \n",
" 10.167 | \n",
" 2.14 | \n",
" 2.46 | \n",
" 2.05 | \n",
" NaN | \n",
" 6476.00 | \n",
" 12298.00 | \n",
" 83.5280 | \n",
" NaN | \n",
"
\n",
" \n",
" 3 | \n",
" 3/1/1959 | \n",
" 2610.396 | \n",
" 2452.7 | \n",
" 15.491 | \n",
" 2.777753e+05 | \n",
" 18523.05762 | \n",
" 22.7142 | \n",
" 23.8459 | \n",
" 22.5651 | \n",
" 31.8987 | \n",
" ... | \n",
" 18.289 | \n",
" 10.185 | \n",
" 2.15 | \n",
" 2.45 | \n",
" 2.07 | \n",
" NaN | \n",
" 6508.00 | \n",
" 12349.00 | \n",
" 81.6405 | \n",
" NaN | \n",
"
\n",
" \n",
" 4 | \n",
" 4/1/1959 | \n",
" 2627.446 | \n",
" 2470.0 | \n",
" 15.435 | \n",
" 2.833627e+05 | \n",
" 18534.46600 | \n",
" 23.1981 | \n",
" 24.1903 | \n",
" 22.8957 | \n",
" 32.4019 | \n",
" ... | \n",
" 18.300 | \n",
" 10.221 | \n",
" 2.16 | \n",
" 2.47 | \n",
" 2.08 | \n",
" NaN | \n",
" 6620.00 | \n",
" 12484.00 | \n",
" 81.8099 | \n",
" NaN | \n",
"
\n",
" \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
"
\n",
" \n",
" 788 | \n",
" 8/1/2024 | \n",
" 20007.209 | \n",
" 16322.1 | \n",
" 121.052 | \n",
" 1.530317e+06 | \n",
" 710038.00000 | \n",
" 103.0135 | \n",
" 100.9825 | \n",
" 100.9803 | \n",
" 102.2118 | \n",
" ... | \n",
" 119.653 | \n",
" 128.291 | \n",
" 31.26 | \n",
" 35.81 | \n",
" 27.97 | \n",
" 67.9 | \n",
" 551667.22 | \n",
" 933066.90 | \n",
" 5327.6461 | \n",
" 19.6750 | \n",
"
\n",
" \n",
" 789 | \n",
" 9/1/2024 | \n",
" 20044.142 | \n",
" 16333.7 | \n",
" 121.690 | \n",
" 1.541305e+06 | \n",
" 716388.00000 | \n",
" 102.5969 | \n",
" 100.3826 | \n",
" 100.0630 | \n",
" 101.9696 | \n",
" ... | \n",
" 119.220 | \n",
" 128.682 | \n",
" 31.44 | \n",
" 36.00 | \n",
" 28.11 | \n",
" 70.1 | \n",
" 553347.06 | \n",
" 934283.59 | \n",
" 5368.5818 | \n",
" 17.6597 | \n",
"
\n",
" \n",
" 790 | \n",
" 10/1/2024 | \n",
" 20128.752 | \n",
" 16397.9 | \n",
" 121.948 | \n",
" 1.539382e+06 | \n",
" 720393.00000 | \n",
" 102.0854 | \n",
" 99.5434 | \n",
" 98.9267 | \n",
" 101.3127 | \n",
" ... | \n",
" 119.064 | \n",
" 129.169 | \n",
" 31.55 | \n",
" 36.22 | \n",
" 28.14 | \n",
" 70.5 | \n",
" 554377.25 | \n",
" 937299.96 | \n",
" 5407.3304 | \n",
" 19.9478 | \n",
"
\n",
" \n",
" 791 | \n",
" 11/1/2024 | \n",
" 20161.687 | \n",
" 16432.8 | \n",
" 122.519 | \n",
" 1.544190e+06 | \n",
" 725925.00000 | \n",
" 102.2549 | \n",
" 99.8216 | \n",
" 99.4970 | \n",
" 101.7893 | \n",
" ... | \n",
" 119.112 | \n",
" 129.375 | \n",
" 31.61 | \n",
" 36.21 | \n",
" 28.29 | \n",
" 71.8 | \n",
" 555000.61 | \n",
" 938899.31 | \n",
" 5382.4019 | \n",
" 15.9822 | \n",
"
\n",
" \n",
" 792 | \n",
" 12/1/2024 | \n",
" 20184.060 | \n",
" 16457.8 | \n",
" 123.013 | \n",
" NaN | \n",
" 729191.00000 | \n",
" 103.1942 | \n",
" 100.5351 | \n",
" 100.1302 | \n",
" 102.2582 | \n",
" ... | \n",
" 119.689 | \n",
" 129.760 | \n",
" 31.73 | \n",
" 36.44 | \n",
" 28.34 | \n",
" 74.0 | \n",
" NaN | \n",
" NaN | \n",
" 5370.6184 | \n",
" 15.6997 | \n",
"
\n",
" \n",
"
\n",
"
793 rows × 127 columns
\n",
"
"
],
"text/plain": [
" sasdate RPI W875RX1 DPCERA3M086SBEA CMRMTSPLx \\\n",
"0 Transform: 5.000 5.0 5.000 5.000000e+00 \n",
"1 1/1/1959 2583.560 2426.0 15.188 2.766768e+05 \n",
"2 2/1/1959 2593.596 2434.8 15.346 2.787140e+05 \n",
"3 3/1/1959 2610.396 2452.7 15.491 2.777753e+05 \n",
"4 4/1/1959 2627.446 2470.0 15.435 2.833627e+05 \n",
".. ... ... ... ... ... \n",
"788 8/1/2024 20007.209 16322.1 121.052 1.530317e+06 \n",
"789 9/1/2024 20044.142 16333.7 121.690 1.541305e+06 \n",
"790 10/1/2024 20128.752 16397.9 121.948 1.539382e+06 \n",
"791 11/1/2024 20161.687 16432.8 122.519 1.544190e+06 \n",
"792 12/1/2024 20184.060 16457.8 123.013 NaN \n",
"\n",
" RETAILx INDPRO IPFPNSS IPFINAL IPCONGD ... \\\n",
"0 5.00000 5.0000 5.0000 5.0000 5.0000 ... \n",
"1 18235.77392 21.9616 23.3868 22.2620 31.6664 ... \n",
"2 18369.56308 22.3917 23.7024 22.4549 31.8987 ... \n",
"3 18523.05762 22.7142 23.8459 22.5651 31.8987 ... \n",
"4 18534.46600 23.1981 24.1903 22.8957 32.4019 ... \n",
".. ... ... ... ... ... ... \n",
"788 710038.00000 103.0135 100.9825 100.9803 102.2118 ... \n",
"789 716388.00000 102.5969 100.3826 100.0630 101.9696 ... \n",
"790 720393.00000 102.0854 99.5434 98.9267 101.3127 ... \n",
"791 725925.00000 102.2549 99.8216 99.4970 101.7893 ... \n",
"792 729191.00000 103.1942 100.5351 100.1302 102.2582 ... \n",
"\n",
" DNDGRG3M086SBEA DSERRG3M086SBEA CES0600000008 CES2000000008 \\\n",
"0 6.000 6.000 6.00 6.00 \n",
"1 18.294 10.152 2.13 2.45 \n",
"2 18.302 10.167 2.14 2.46 \n",
"3 18.289 10.185 2.15 2.45 \n",
"4 18.300 10.221 2.16 2.47 \n",
".. ... ... ... ... \n",
"788 119.653 128.291 31.26 35.81 \n",
"789 119.220 128.682 31.44 36.00 \n",
"790 119.064 129.169 31.55 36.22 \n",
"791 119.112 129.375 31.61 36.21 \n",
"792 119.689 129.760 31.73 36.44 \n",
"\n",
" CES3000000008 UMCSENTx DTCOLNVHFNM DTCTHFNM INVEST VIXCLSx \n",
"0 6.00 2.0 6.00 6.00 6.0000 1.0000 \n",
"1 2.04 NaN 6476.00 12298.00 84.2043 NaN \n",
"2 2.05 NaN 6476.00 12298.00 83.5280 NaN \n",
"3 2.07 NaN 6508.00 12349.00 81.6405 NaN \n",
"4 2.08 NaN 6620.00 12484.00 81.8099 NaN \n",
".. ... ... ... ... ... ... \n",
"788 27.97 67.9 551667.22 933066.90 5327.6461 19.6750 \n",
"789 28.11 70.1 553347.06 934283.59 5368.5818 17.6597 \n",
"790 28.14 70.5 554377.25 937299.96 5407.3304 19.9478 \n",
"791 28.29 71.8 555000.61 938899.31 5382.4019 15.9822 \n",
"792 28.34 74.0 NaN NaN 5370.6184 15.6997 \n",
"\n",
"[793 rows x 127 columns]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Pandas has several built-in readers for csv, xml, json, excel and even html files \n",
"df = pd.read_csv(url, header=0)\n",
"df"
]
},
{
"cell_type": "markdown",
"id": "db65a278",
"metadata": {},
"source": [
"\n",
"### Web scraping\n",
"\n",
"Web scraping involves extracting data from unstructured web pages by identifying patterns in the HTML structure. This method is useful when structured data files are unavailable, but it requires compliance with website policies. \n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "336c92a0",
"metadata": {},
"outputs": [],
"source": [
"# URL that displays the most popular series in the FRED economic data web site\n",
"url = f\"https://fred.stlouisfed.org/tags/series?ob=pv&pageID=1\""
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "be575083",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# use requests package to retrieve the web page\n",
"import requests\n",
"data = requests.get(url)\n",
"data # a response code of 200 indicates the request has succeeded"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "ac8b4ba5",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"b'\\n\\n\\n \\n \\n Economic Data Series by Tag | FRED | St. Louis Fed'"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# the content is just a byte-string that you can parse with Python string (or other) methods\n",
"data.content[:200]"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "899f3ce3",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"s=\"series-title pager-series-title-gtm\" href=\"/series/T10Y2Y\" id=\"titleLink\" style=\"font-size:1.2em; padding-bottom: 2px\">10-Year Treasury Constant Maturity Minus 2-Year Treasury Constant Maturity\n",
"\n",
"\n",
"\n",
" \n",
"
\n",
"\n",
"\n",
"