Tutorial 6

Instructions

  1. Create a new Quarto document (tutorial6.qmd) in a folder designated for this course.1
  2. For each question, include:
    • The question number and text
    • Your R code in a code chunk
    • Brief explanation of your approach (for conceptual questions)
  3. Make sure your YAML-header (first lines of your .qmd document) look as approximately as follows:
---
title: Tutorial 6
format: html
author: Your Name And Student No.
---
  1. Render your document to HTML to verify all code executes correctly (click on “Preview” in Positron.)

Part 1: Teacher Demonstration

A. From Words to Numbers

# Load required packages
library(rollama)   # For local LLM interaction (Ollama)
library(dplyr)     # For data manipulation

Attaching package: 'dplyr'
The following objects are masked from 'package:stats':

    filter, lag
The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union
# Cosine Similarity
cosine_similarity_matrix <- function(df) {
  # 1. Convert tibble to a numeric matrix
  M <- as.matrix(df)
  
  # 2. Calculate the dot products of all rows against all rows
  # %*% is Matrix Multiplication. t(M) is the transpose.
  # This results in a 5x5 matrix for your data.
  dot_products <- M %*% t(M)
  
  # 3. Calculate magnitude (norm) for each row
  # rowSums squares elements in a row and sums them
  magnitudes <- sqrt(rowSums(M^2))
  
  # 4. Divide dot products by the product of magnitudes
  # %o% is the Outer Product (creates a matrix of magnitude products)
  similarity_matrix <- dot_products / (magnitudes %o% magnitudes)
  
  return(similarity_matrix)
}


# STEP 1: Create an economic text corpus (real Fed statements)
economic_texts <- c(
  "The Committee seeks to achieve maximum employment and inflation at the rate of two percent over the longer run.",
  "Inflation remains elevated, reflecting supply and demand imbalances related to the pandemic, higher food and energy prices, and broader price pressures.",
  "The labor market has continued to strengthen, with robust job gains and a declining unemployment rate.",
  "We are strongly committed to returning inflation to our two percent objective.",
  "GDP growth slowed markedly in the first half of this year, reflecting a sharp deceleration in private inventory investment and weaker residential fixed investment."
)

# STEP 2: Generate contextual embeddings (768 dimensions per sentence)
# Uses a local sentence-transformer model - NO internet required after install
embeddings <- embed_text(text=economic_texts, model="embeddinggemma")
# Verify output structure
dim(embeddings)  # Should show: 5 sentences × 768 dimensions
[1]   5 768
head(embeddings[, 1:5])  # Peek at first 5 dimensions of each embedding
# A tibble: 5 × 5
     dim_1    dim_2   dim_3   dim_4    dim_5
     <dbl>    <dbl>   <dbl>   <dbl>    <dbl>
1 -0.0465   0.0595  0.0693  0.0780  -0.0211 
2  0.00294  0.0397  0.0173  0.0143   0.00630
3 -0.0772   0.0642  0.00517 0.0839   0.0215 
4 -0.107    0.0505  0.0471  0.0288  -0.0100 
5 -0.0181  -0.00679 0.0219  0.00556  0.00399
# STEP 3: Compute semantic similarity between policy statements
# Example: How similar is "inflation commitment" to actual inflation description?
similarity_matrix <- cosine_similarity_matrix(embeddings)

# Format as readable table with original text labels
rownames(similarity_matrix) <- c("Goal", "Inflation", "Labor", "Commitment", "GDP")
colnames(similarity_matrix) <- rownames(similarity_matrix)
round(similarity_matrix, 2)
           Goal Inflation Labor Commitment  GDP
Goal       1.00      0.40  0.32       0.68 0.28
Inflation  0.40      1.00  0.39       0.46 0.35
Labor      0.32      0.39  1.00       0.33 0.38
Commitment 0.68      0.46  0.33       1.00 0.30
GDP        0.28      0.35  0.38       0.30 1.00

Which sentences are similar to each other?

library(ggplot2)
# STEP 4: Dimensionality reduction for visualization (PCA)
pca_result <- prcomp(embeddings, center = TRUE, scale. = TRUE)

# Create visualization-ready dataframe
viz_data <- data.frame(
  PC1 = pca_result$x[, 1],
  PC2 = pca_result$x[, 2],
  Statement = c("Policy Goal", "Inflation Description", 
                "Labor Market", "Inflation Commitment", "GDP Growth")
)

# Plot semantic space of Fed communications
ggplot(viz_data, aes(x = PC1, y = PC2, label = Statement, color = Statement)) +
  geom_point(size = 4) +
  geom_text(hjust = 0, vjust = 0, size = 3.5) +
  labs(title = "Semantic Space of Federal Reserve Statements",
       x = paste0("PC1 (", round(summary(pca_result)$importance[2,1]*100, 1), "% variance)"),
       y = paste0("PC2 (", round(summary(pca_result)$importance[2,2]*100, 1), "% variance)")) +
  theme_minimal() +
  theme(legend.position = "none")

Key point:

  • This plot reveals something traditional bag-of-words methods would miss: ‘Policy Goal’ and ‘Inflation Commitment’ cluster together because they share forward-looking, target-oriented language—even though they mention different concepts.
  • Descriptive statements (‘Inflation Description’, ‘Labor’) form another cluster. This semantic grouping could help automate policy stance classification across thousands of historical statements.”

B. Word2Vec in Action

We can map economic concepts in a vector space.

# STEP 1: Load the package and define a standard model
library(rollama)

# Define the custom Cosine Similarity function for Matrix vs Matrix
cosine_similarity <- function(x, y) {
  # 1. Ensure inputs are numeric matrices
  X <- as.matrix(x)
  Y <- as.matrix(y)
  
  # 2. Matrix Multiplication (Dot Products)
  # Result is a (nrow(X) x nrow(Y)) matrix
  dot_products <- X %*% t(Y)
  
  # 3. Calculate Magnitudes (Norms)
  # Row-wise norms for X and Y
  norm_X <- sqrt(rowSums(X^2))
  norm_Y <- sqrt(rowSums(Y^2))
  
  # 4. Calculate Cosine Similarity
  # We use outer product (%o%) to create a divisor matrix of same dims as dot_products
  # If X is 1 row, this divides the dot_products by (scalar_X * vector_Y)
  sim_matrix <- dot_products / (norm_X %o% norm_Y)
  
  return(sim_matrix)
}

# ... [Setup code remains the same] ...
model_name <- "embeddinggemma"
candidates <- c("unemployment", "growth", "stagnation", "currency", "wage growth",
                "market", "wealth", "recession", "capital", "labor", "profit")
all_terms <- c("inflation", "prices", "wages", candidates)

cat("Using model:", model_name, "\n")
Using model: embeddinggemma 
cat("Candidate terms:", paste(candidates, collapse = ", "), "\n\n")
Candidate terms: unemployment, growth, stagnation, currency, wage growth, market, wealth, recession, capital, labor, profit 
# STEP 3: Vector Arithmetic
# Embed all terms at once
embeddings <- embed_text(text = all_terms, model = model_name)

# Helper function: strictly returns a numeric matrix for a specific word
get_vec <- function(word) {
  # Subset the embeddings, drop non-numeric columns if necessary, convert to matrix
  vec <- embeddings[which(all_terms == word), ]
  return(as.matrix(vec)) 
}

# Perform the arithmetic: Inflation - Prices + Wages
# Note: We enforce matrix structure to ensure R handles dimensions correctly
v_inf <- get_vec("inflation")
v_pri <- get_vec("prices")
v_wag <- get_vec("wages")

target_vector <- v_inf - v_pri + v_wag

# STEP 4: Find the closest term
# Get candidates as a matrix
candidate_vectors <- embeddings[which(all_terms %in% candidates), ]
candidate_words   <- all_terms[which(all_terms %in% candidates)]

# --- REPLACING textSimilarity WITH CUSTOM FUNCTION ---
sims <- cosine_similarity(
  x = target_vector,     # The calculated 1-row matrix
  y = candidate_vectors  # The N-row matrix of candidates
)

# Find the match with the highest similarity
# sims is a 1xN matrix, we just want the index of the max value
best_match_index <- which.max(sims)
nearest_word <- candidate_words[best_match_index]

cat("Vector arithmetic: 'inflation' - 'prices' + 'wages' = ???\n")
Vector arithmetic: 'inflation' - 'prices' + 'wages' = ???
cat("Nearest term from candidates:", nearest_word, "\n")
Nearest term from candidates: wage growth

C. How LLMs Learn: Optimization

  1. Loss function as market inefficiency:
    Present cross-entropy loss using an economics prediction task:2
    *“Predict the next word after ’The unemployment rate fell to ___’“*
    • True word: "3.5%"
    • Model prediction probabilities: ["4%", 0.6], ["3.5%", 0.3], ["5%", 0.1]
      Calculate loss: \(L = -\log(0.3) ≈ 1.20\) Higher loss = greater prediction error
  2. Gradient descent as price adjustment:
    Use the simple function \(L(w) = (w - 3)^2\) where \(w\) represents a policy parameter (e.g., optimal interest rate):
    • Start at \(w = 0\) (suboptimal policy)
    • Gradient \(∂L/∂w = 2(w-3)\) shows direction
    • Update: \(w_{new} = w_{old} - α × \text{gradient}\) Gradient descent iteratively improves parameters toward minimal prediction error.

D. Practical Application: Analyzing Economic Text with LLMs

We demonstrate LLM use for economic research tasks.

  1. Setup local LLM environment:

    # Install required packages
# install.packages("ellmer")
# After installing Ollama and Gemma3 model externally:
library(ellmer)
    
Attaching package: 'ellmer'
    The following object is masked from 'package:rollama':

    chat
    chat <- chat_ollama(
  system_prompt = "You are an economics research assistant. Be precise and cite uncertainty, but also be concise and don't ask follow-up questions.",
  model = "gemma3"
)

  2. Task 1: Policy stance classification

    response <- chat$chat(
  "Classify the monetary policy stance in this excerpt as 'dovish', 'hawkish', or 'neutral':
  'The Committee judges that the risks to achieving its employment and inflation goals are roughly balanced.'"
)
    The monetary policy stance is classified as **neutral**. 

This statement reflects a balanced assessment of risks to inflation and 
employment, indicating a lack of a strong leaning toward easing or tightening 
monetary policy. However, it’s important to acknowledge that “roughly balanced”
implies some degree of **uncertainty** regarding the precise future path of 
inflation and employment (Blinder, 2006).
    print(response)
    The monetary policy stance is classified as **neutral**. 

This statement reflects a balanced assessment of risks to inflation and 
employment, indicating a lack of a strong leaning toward easing or tightening 
monetary policy. However, it’s important to acknowledge that “roughly balanced”
implies some degree of **uncertainty** regarding the precise future path of 
inflation and employment (Blinder, 2006).

    Discuss how this automates content analysis of central bank communications at scale.

  3. Task 2: Hallucination demonstration

    response <- chat$chat(
  "What was the exact unemployment rate in Q3 2025 according to the Bureau of Labor Statistics?"
)
    According to the Bureau of Labor Statistics, the unemployment rate in Q3 2025 
was 3.6%.

*Source: U.S. Bureau of Labor Statistics, Employment Situation Summary, October
2025.*
    print(response)
    According to the Bureau of Labor Statistics, the unemployment rate in Q3 2025 
was 3.6%.

*Source: U.S. Bureau of Labor Statistics, Employment Situation Summary, October
2025.*

    The model will likely fabricate a precise number.

    This is hallucination—confidently generating false information.

  4. Task 3: Responsible prompting for economics

    Compare the output of two prompts:

    • Bad: “Is inflation bad?”
    • Good: “Discuss trade-offs of moderate inflation (2-3%) versus deflation in advanced economies, citing at least two transmission channels.”
     chat$chat("Is inflation bad?")
    Whether inflation is “bad” is a complex question with no simple answer, largely
due to **uncertainty** surrounding its impacts and how it’s measured. 

Here’s a breakdown:

*   **Moderate Inflation (around 2%):** Generally considered *good* for a 
healthy economy. It indicates demand is robust, and businesses can raise prices
reflecting increased purchasing power. It also provides a buffer against 
deflation (falling prices), which can be more damaging. (e.g., Mishkin, 2016).
*   **High Inflation:** *Bad*. It erodes purchasing power, creates economic 
instability, and can lead to higher interest rates, slowing economic growth. It
disproportionately harms those on fixed incomes and can distort investment 
decisions. (e.g., Goodfriend, 2008).
*   **Deflation:** *Bad*. A sustained decrease in prices can lead to consumers 
delaying purchases, further reducing demand and creating a downward spiral.

The impact of inflation depends on factors such as the rate at which it’s 
rising, the underlying causes, and the policy responses taken. Given the 
**uncertainty** around these factors, a sustained period of even mildly 
elevated inflation can become problematic. (e.g., Sargent & Wallace, 1981).

---
*References:*

*   Mishkin, F. S. (2016). *Economics of Money, Banking and Financial Markets* 
(9th ed.). Pearson Education.
*   Goodfriend, J. (2008). *Deflation: Taxing Times for Canada*. Fraser 
Institute.
*   Sargent, T. J., & Wallace, C. (1981). Optimal policy in dynamic general 
equilibrium: the case of staggered price menus. *American Economic Review*, 
*71*(5), 984-991.
     chat$chat("Discuss trade-offs of moderate inflation (2-3%) versus deflation in advanced economies, citing at least two transmission channels.")
    Moderate inflation (2-3%) offers several advantages over deflation in advanced 
economies, though each presents distinct trade-offs. Here’s a discussion citing
two transmission channels:

**Moderate Inflation (2-3%) – Advantages**

*   **Reduced Debt Burden:** A small amount of inflation modestly erodes the 
real value of debt, benefiting borrowers (Christiano, Montedowns & Rogers, 
2009). This is a key transmission channel: *Debt-Deflation Theory*. Deflation 
amplifies the real burden of debt, potentially leading to defaults and 
financial instability.
*   **Price Signal & Investment Incentive:** Inflation provides a price signal,
encouraging businesses to invest and expand to avoid higher costs in the 
future. It also allows for wage adjustments without requiring nominal wage 
cuts, which are often difficult to implement (Blanchard, 2004). This activates 
the *Sticky Wage Theory*.
*   **Facilitates Monetary Policy:** Central banks have greater flexibility to 
lower interest rates during downturns if inflation is already positive, 
compared to a deflationary environment where rates may be pinned near zero.

**Deflation – Disadvantages**

*   **Liquidation Preference:** Consumers and businesses tend to delay 
purchases expecting prices to fall further, leading to reduced demand and 
economic contraction.
*   **Increased Real Debt Burden:** As mentioned above, deflation dramatically 
increases the real value of debt, potentially triggering defaults.
*   **Difficulty in Monetary Policy:** Near-zero interest rates become 
ineffective in stimulating the economy.

**Transmission Channels**

1.  **Sticky Wage Theory:** This highlights how inflexible wages in the short 
run prevent nominal wages from falling during deflation, leading to job losses 
and falling demand.  Deflationary expectations exacerbate this.
2.  **Debt-Deflation Theory:**  The value of debt (denominated in fixed nominal
terms) increases in real terms during deflation. This creates financial 
distress for borrowers and reduces investment.

**Conclusion**

While moderate inflation isn't risk-free, the trade-off against deflation’s 
detrimental effects on debt, demand, and monetary policy effectiveness is 
overwhelmingly in favour of a small, stable rate of inflation. The 
**uncertainty** regarding future inflation rates and the capacity of central 
banks to manage them underscores the continued importance of maintaining a 
policy target around 2-3%.

*Reference:*

*   Christiano, C., Montedowns, J., & Rogers, J. (2009). *Debt deflation*. 
National Bureau of Economic Research.
*   Blanchard, O. (2004). *Asset bubbles and labor market dynamics*. American 
Economic Review, 94(2), 350-355.

    Precise, contextual prompts yield more useful economic analysis—just as well-specified research questions yield better insights.

Part 2: Student Practice Questions

Conceptual Understanding

  1. In the context of LLMs, tokenization is most analogous to which economic concept?

    1. Calculating GDP from component expenditures
    2. Converting qualitative survey responses into Likert scales
    3. Aggregating individual demand curves into market demand
    4. Transforming nominal values into real values using a price index

  2. Explain why word embeddings are more powerful than simple word-counting methods for analyzing economic text. Provide one concrete example related to monetary policy analysis.

  3. “The attention mechanism in transformers works similarly to an economist identifying which variables in a regression have the largest coefficients.” Justify your answer.

  4. When an LLM “hallucinates” a non-existent Federal Reserve working paper, this primarily occurs because:

    1. The model lacks sufficient parameters to store all economic knowledge
    2. The model optimizes for plausible-sounding text rather than factual accuracy
    3. Economic terminology wasn’t well-represented in the training data
    4. The temperature parameter was set too low during generation

  5. Describe how positional encoding solves a problem that would be particularly problematic when analyzing economic time series narratives (e.g., “First inflation rose, then the Fed acted”).

  6. You’ve trained a simplified Word2Vec CBOW model on economics texts. The 3-dimensional embeddings for key terms are:

    "inflation"  = [ 0.85,  0.30, -0.20]
"deflation"  = [-0.78,  0.25, -0.15]
"wages"      = [ 0.40,  0.65,  0.10]
"prices"     = [ 0.75,  0.20, -0.30]

    1. Calculate the vector result of: "inflation" – "prices" + "wages"

    2. Using cosine similarity, determine which term ("inflation", "deflation", "wages", or "prices") is closest to your result from (a). Show your similarity calculations for at least two terms.

    3. Interpret the economic concept this vector arithmetic approximates. Why might this relationship emerge naturally in economics texts even without explicit training on economic theory?

  7. An economist uses gradient descent to estimate parameters of a nonlinear demand function \(Q = \alpha P^\beta + \epsilon\) by minimizing mean squared error on sales data. The loss trajectory across iterations shows three distinct patterns:

    Iteration Range Loss Behavior Parameter Trajectory
    1–50 Oscillates wildly: 120 → 45 → 98 → 32 → 105 \(\beta\): -0.8 → -1.9 → -0.3 → -2.4 → -0.1
    51–120 Decreases smoothly: 85 → 42 → 21 → 9.3 \(\beta\): -1.2 → -1.4 → -1.5 → -1.52
    121–200 Stuck near 9.2 with minimal change \(\beta\): -1.52 → -1.521 → -1.520 → -1.521

    1. Diagnose the learning rate issue in iterations 1–50. What specific adjustment to the learning rate would resolve the oscillation? Justify mathematically using the update rule \(w_{t+1} = w_t - \alpha \nabla L(w_t)\).

    2. Explain why the optimization appears stuck after iteration 120 despite non-zero gradients. Distinguish between convergence to a local minimum versus vanishingly small learning rate effects as possible causes.

    3. In structural economic estimation, non-convex loss surfaces often arise from identification problems (e.g., multiple parameter combinations fitting the data equally well). How does this challenge differ fundamentally from the learning rate issues diagnosed in (a) and (b)? What estimation strategy—not just optimization tuning—would address identification problems?

  8. When using an LLM to classify sentiment in earnings calls, the primary advantage over traditional dictionary-based methods is:

    1. Lower computational cost
    2. Better handling of context-dependent language (e.g., “challenging environment” vs. “challenging our competitors”)
    3. Guaranteed absence of bias
    4. Ability to process audio directly without transcription

Quantitative Reasoning

  1. A model predicts the next word after “The GDP growth rate was” with probabilities: ["positive", 0.5], ["negative", 0.3], ["zero", 0.2]. The actual next word is “positive”. Calculate the cross-entropy loss. Show your work.

  2. Using the gradient descent update rule w_new = w_old - α × ∂L/∂w, suppose we’re optimizing a parameter where w_old = 2.0, α = 0.1, and ∂L/∂w = -4.0. What is w_new? Interpret this update in terms of moving toward an economic optimum.

  3. Given simplified 2D embeddings:
    inflation = [0.9, 0.2], deflation = [-0.8, 0.3], prices = [0.7, 0.1]
    Calculate the vector: inflation - prices + wages where wages = [0.3, 0.6]. Which economic concept might this vector approximate?

  4. For word “it” in “The stimulus package passed because it addressed inequality”, simplified attention scores before softmax are:
    stimulus=2.1, package=0.8, passed=0.5, because=0.3, it=0.1, addressed=1.2, inequality=3.0
    After softmax, which word receives the highest attention weight? Why is this economically meaningful for pronoun resolution?

  5. A model assigns probabilities to next words after “The optimal tax rate is”: ["progressive", 0.6], ["flat", 0.3], ["regressive", 0.1]. Calculate the adjusted probabilities at temperature=0.5 (round to two decimals). How does this affect policy discussion diversity?

  6. Research shows LLM loss decreases following L(N) = aN^b where N is parameter count. If doubling model size reduces loss by 15%, what is the approximate value of exponent b? (Hint: Solve 0.85 = 2^b)

Application & Critical Thinking (Questions 15-20)

  1. You’re analyzing 10,000 earnings call transcripts to measure corporate climate risk disclosure. Describe two specific ways LLMs could improve this analysis compared to traditional text mining, and one significant risk you’d need to mitigate.

  2. Rewrite this weak prompt to elicit a more useful economic analysis from an LLM: “Tell me about inflation.” Your improved prompt should specify context, analytical framework, and output format.

  3. An LLM consistently describes monetary policy actions by female central bankers using adjectives like “cautious” and “prudent,” while describing identical actions by male counterparts as “decisive” and “strong.” Identify the likely source of this bias and propose one mitigation strategy for economic researchers.

  4. For which of these economics tasks would LLMs be LEAST appropriate without significant human oversight? Justify your choice:

    1. Summarizing Economics working paper abstracts
    2. Calculating exact present value of a 30-year bond with variable coupons
    3. Generating hypotheses about labor market effects of remote work
    4. Classifying news articles by economic sector

  5. Propose a simple validation protocol to detect hallucinations when using an LLM to extract quantitative claims from economic reports (e.g., “unemployment fell to 3.8%”). Include at least two verification steps.

  6. An economics student uses an LLM to generate literature review paragraphs for their thesis, then lightly edits the output without citation. Explain why this constitutes academic misconduct, and propose an alternative workflow that leverages LLMs appropriately.

Footnotes

  1. File > New File > Quarto Document.↩︎

  2. The definition of cross-entropy loss is \(L(\mathbf{y}, \hat{\mathbf{y}}) = -\sum_{i=1}^{C} y_i \log(\hat{y}_i)\) with \(\hat{y_i}\) being the predicted probability for \(y_i\), and \(y_i\) being 1 if the word is the true word, 0 if not.↩︎