- 2024-07-31
- News
Can AI Track the Fed?
Recently, there has been an in-depth exploration of the potential and limitations of generative artificial intelligence in analyzing speeches by Federal Reserve officials and predicting the direction of monetary policy.
HSBC has developed an AI tool based on GPT-4, specifically designed to automatically assess the relevance, content summary, policy inclination (hawkish/dovish/neutral) of Federal Reserve speeches and the reasons behind them.
By analyzing 54 Federal Reserve speeches, the AI tool's performance in analyzing speeches highly relevant to monetary policy is close to that of a "hardworking junior analyst," and its conclusions are mostly in line with those of human experts.
However, when dealing with speeches unrelated or less relevant to monetary policy, AI may generate inaccurate or even fabricated content, reducing user trust.
Therefore, the conclusion is that while AI tools have the potential to increase productivity, they are not yet sufficient to completely replace human analysts, especially in tasks requiring high precision.
AI Analysis of Federal Reserve Speeches
The AI tool conducts four aspects of analysis on Federal Reserve officials' speeches through a "sentence-by-sentence screening + overall analysis" approach: relevance screening, content summary, policy inclination judgment (hawkish, dovish, or neutral), and explanation of decision-making basis.
Advertisement
The research team hopes to answer a core question with this tool: whether AI can reliably and accurately analyze Federal Reserve policy signals to provide practical references for investors.
The report points out that the AI tool's performance in analyzing highly relevant speeches is impressive, with output quality comparable to that of a "hardworking junior analyst." Especially in judging policy inclinations, AI results are largely consistent with those of human experts.
For example, among the 54 speeches studied, AI was completely consistent with human experts in judging them as "neutral," and there were few significant deviations when assessing "hawkish" or "dovish" speeches.
"We can see that there is no disagreement on dovish or neutral speeches. For the 10 speeches that the model views as hawkish, our human experts agreed with this view for 8 speeches, and other experts believed that only 2 speeches were neutral."
AI's Shortcomings: Randomness and "Creativity" Challenges
Although the AI tool has shown some capabilities, the report also points out several shortcomings of AI.
Firstly, the "creativity" of generative AI can sometimes lead to problems, especially when dealing with speeches unrelated to monetary policy, AI may fabricate information to "satisfy" user needs. This phenomenon, known as "hallucination," can seriously damage user trust in the tool.
"Artificial intelligence models have been trained to show a strong desire to help, and they tend to fabricate information to give you an answer rather than reminding you that your question is confusing or unanswerable."
In addition, the inconsistency of AI tools in repetitive tasks is also a key challenge. The report found that even under the same conditions, AI's analysis results for the same speech may vary.
The research team emphasized that while human analysts also have a certain degree of subjectivity to some extent, a single analyst's interpretation of the same content at different times is usually consistent, which AI cannot guarantee.
Can AI Replace Analysts?
The study shows that although AI tools have some performance in analysis tasks, they are still unable to completely replace the role of human analysts at present.
The report points out that AI performs well in some more basic and repetitive tasks, such as screening relevant information and generating summaries. However, in work involving complex reasoning and in-depth interpretation, the gap between AI and humans remains significant.
The performance of AI tools is similar to the level of junior analysts, but they cannot continuously improve like humans through experience and feedback. This point is particularly important for companies because training human analysts is not only for current output but also for future development potential.
The report also emphasizes that AI's "randomness" and "creativity" traits, while bringing flexibility to the technology, also become barriers to widespread application. In high-precision demand fields such as the financial market, users have extremely high requirements for the stability and credibility of results. If the output of AI tools requires repeated verification by human experts, the so-called efficiency improvement will be greatly reduced.
"But when you ask an artificial intelligence tool for its opinion, its opinion will change from moment to moment. Unless there is a fundamental change in human nature, artificial intelligence tools are likely to be unable to serve as trustworthy advisors. This seems unlikely to happen in the short term."
- 165 Comments
- 122
- 128