HOW BIGDATA TRENDS WORKS
How news becomes
a signal you can explore.
Bigdata Trends is the product. Rare Earths is the theme. Each theme contains narratives, organized into mechanisms with beneficial and adverse statements. We use this framework to find and assess company news, then combine the evidence into signals such as the Rare Earth Index. Here’s how it works—and what the charts do not tell you.
Define the theme and its company basket
Start with an economic question, then find the businesses exposed to it.
We start by defining the economic activity we want to understand. For rare earths, that means the companies exposed to mining, processing, magnets and their end markets. Bigdata’s search and relationship APIs help us discover candidates; we then check whether each company has a real business connection to the theme. A company appearing alongside “rare earths” in the news is a starting point, not enough to qualify it for the basket.
Find reporting that establishes business exposure.
Surface associated topics and recurring connections.
Discover connected companies, products and concepts.
Documented as “Connected Entities.”Screening for publicly traded securities
We filter candidate companies before including them in the basket. Alongside verified thematic exposure, we require public trading status and apply the price, size and liquidity thresholds below to exclude penny stocks, very small companies and insufficiently liquid securities.
Inside the method
Search and relationship discovery identify candidates; co-occurrence is not proof of economic exposure. We review source-backed business activity, resolve aliases to company IDs and apply public-company and liquidity checks.
These are run-time screening thresholds, not historical eligibility guarantees. The website’s curated basket is not a survivorship-bias-free investment universe. The reusable service and frozen website export have separate configuration provenance.
Break the theme into testable statements
A taxonomy of economic mechanisms—not a collection of bullish keywords.
A theme is too broad to test against an individual news story. We divide it into narratives, such as Government Support, and then into mechanisms: the specific ways those forces affect companies. Each mechanism has a beneficial and an adverse statement. For example, defense demand can encourage investment, while uncertainty about that demand can delay it. Both sides belong in the research framework.
Defense Requirements Drive Investment
Defense Demand Uncertainty Delays Rare-Earth Investment
We pair beneficial and adverse statements so our searches actively seek both bullish and bearish developments. The aim is balanced search coverage, not a bullish-only news feed. We do not force the retrieved evidence into a 50/50 split: the actual mix should reflect the reporting.
The complete Rare Earths theme
Inside the method
Policy, Supply, Markets and Demand organize distinct narrative channels. Each mechanism has two testable statements describing opposing developments. Government Support has five pairs; each other narrative has two.
Statement direction is not company impact: a supply constraint may harm one business and benefit a competitor. Statements cover different projects and companies while remaining specific enough to test against an atomic event.
Retrieve news and extract triplets
Queries find candidate passages. Extraction identifies what actually happened to whom.
For each statement, six queries look for different ways the same mechanism could appear in reporting. We use the Bigdata Search API to retrieve candidate passages within the chosen company and date scope. Extraction then turns a passage into a triplet: the company, whether it benefits or is harmed, and the specific event causing that effect. The original document remains attached as evidence.
Inside the method
Balanced specifications use six retrieval queries per statement to cover different expressions of the mechanism. Shared passage identities prevent overlapping searches from becoming unrelated extraction inputs. Retrieval finds candidates; it does not establish statement membership.
The extractor requires a named company and a causal effect. Resolution links it to the basket. Pure macro commentary without an attributable company effect does not enter the impact dataset. A document can contain several distinct events.
Check whether the event is relevant
Relevance uses the source, the narrative and the statement—not keywords alone.
Finding an article is not the same as finding evidence for a statement. We assess the extracted event against the statement and its narrative context. Consider a government deal with a producer: a purchase commitment belongs in a commercial-commitments narrative, but an equity investment alone does not. This check is about the mechanism described in the event, not whether the article mentions the right company or topic.
MP Materials → Benefits from → A government commitment to purchase its magnet output for defense use
The same event is tested against each statement independently.
- ✓ RelevantCustomer Commitments Secure Rare-Earth SalesThe purchase commitment secures demand for output.
- ✓ RelevantDefense Requirements Drive InvestmentDefense procurement provides a demand commitment for the business.
- × Not relevantRecovery Projects Expand Secondary Rare-Earth SupplyThe event says nothing about recycling or secondary supply.
Green and red indicate relevance to the statement, not positive or negative company impact.
Inside the method
Every atomic claim is assessed against the narrative’s statements, not only the query that retrieved it. A deal can express several mechanisms, but each assignment needs source evidence. Rejected candidates do not contribute to impact. The historical data also underwent relevance review and cleanup; automated classifications can still be wrong.
Separate fresh developments from background
Publication time alone cannot distinguish a new development from repeated background.
An article published today may describe a deal announced months ago. The freshness check asks whether the event itself adds new information at publication time. An announcement or a material change can contribute to the signal. A passage simply recalling an old agreement receives zero impact weight. This prevents background coverage from being treated as a fresh catalyst just because it appears in a new article.
Funding announced
Eligible for impactFunding approved
Eligible for impactOld funding recalled
Zero impact weightInside the method
Freshness compares the encoded event with the source’s publication timestamp; there is no single age cutoff. Changed approvals, milestones or commercial terms may be new. Historical background receives zero weight and is excluded from the eligible registry. Uncertain freshness retains weight 1 in the service—it does not mean verified new.
Freshness asks whether information is new. Hashing asks whether the exact triplet is already known.
Score the company effect
A signed score combines company effect, impact strength, attention and source authority.
Once evidence passes these checks, we assign a signed weight. Benefits are positive and harms are negative; the size also reflects the assessed impact, the attention the event receives and source authority. Freshness determines whether historical background contributes at all. The equation below combines those factors. Its result is an evidence score, not a forecast of a stock’s return.
Benefits +1 · harms −1
Economic significance
Coverage weight
Actor and publisher role
New information vs background
What the weights mean
Economic impact: the model assesses the event’s importance to the named company, not how dramatic the headline sounds. Limited or incremental effects receive 0.75; meaningful but bounded effects receive 1; events that plausibly change company-level economics receive 1.25. These are relative weights, not percentage returns.
Authority: an explicit action, decision or official statement by a government agency, regulator, central bank or policymaker receives 1.5. Otherwise, reporting from a major wire, financial publication or influential industry institution receives 1.2. Other or unknown sources receive 1. The official actor must be part of the atomic event; merely mentioning government does not qualify.
Freshness: new developments and material updates retain their weight (1); historical background contributes nothing (0). The service also retains uncertain cases at 1, so this is a classification rule, not a guarantee that every retained event is new.
More coverage matters.
Not one-for-one.
Logarithmic frequency tempers repeated reporting. Relative density is measured within the statement’s corpus. Publisher breadth is also considered in the narrative-attention calculation: reporting spread across different publishers receives a bounded diversity boost, rather than treating concentrated coverage from one publisher as equally broad. This is separate from the event-attention formula shown here.
The scoring calculation
The reusable service multiplies these five factors. Historical background has F = 0; new, materially updated and uncertain records have F = 1. Direction comes from the company-effect predicate, not the statement’s label.
Attention retains one triplet per statement, company and document using source order. Local clusters normalize text to measure frequency. Density uses all processed statement documents, including those without retained triplets; percentile ties receive an average rank. This normalization is not the final event hash.
These are evidence weights, not expected returns or probabilities. The website uses frozen, precomputed historical scores and series; current service settings are not applied retroactively. Comparisons require pinned scoring provenance.
Deduplicate repeated triplets
A stable hash connects repeated evidence without inventing another impact contribution.
The same development can appear in several publications and on different days. We create a stable identity from the canonical company, predicate and exact atomic-event text. When all three match, we keep one triplet and attach the additional sources to it. The earliest eligible publication day determines its impact date, so later copies do not create another contribution. Different wording remains separate, even if it describes the same deal.
Publisher, document, date and narrative membership.
Exact wording, case, punctuation and whitespace.
Inside the method
Impact is attributed to the earliest eligible publication day. First-day retained observation scores are averaged; later matching evidence does not revise that contribution. Evidence keeps its actual dates and source links.
The theme index counts shared exact events once across narratives while preserving memberships. Different wording can describe the same real-world deal and remain separate: this is not semantic deduplication.
Build the dataset and the chart
Every chart starts with structured events and their contributing evidence.
The output is a traceable dataset, not just a line on a chart. Each event retains its company, classification, score and contributing sources. To build the company-impact series, we combine eligible event scores for each day, temper the effect of event volume, cap extremes and average over a rolling 60-day window. This lets you move from a change in the chart back to the developments behind it.
- Identity
- Event hash · canonical company ID
- Development
- Company → benefit / harm → atomic event
- Classification
- Narrative + statement memberships
- Assessment
- Relevance · freshness · score provenance
- Evidence
- First eligible day · dated sources · headlines · links
Scale event volume; cap extremes.
A quiet day does not erase yesterday’s evidence.
Inspect the developments behind a move.
Inside the method
The daily formula applies when there are eligible events; no-event days contribute zero. Robust calibration caps extremes. Company evidence is averaged across 60 calendar days, then combined across the basket. Early values remain blank during warm-up. A decline may reflect old positive events leaving the window rather than a new negative event.
Impact measures weighted company developments. Attention measures reporting activity relative to its baseline. Prices remain a separate series and axis. Website source previews are capped at 256 characters; pipeline exports retain event, evidence, observation and configuration records.
Add new evidence each day
The daily service reuses the same pinned research and processing stages.
After the historical dataset is built, the daily service retrieves one publication day at a time using the pinned basket, taxonomy and processing configuration. It applies the same extraction and assessment stages, checks new triplets against the accumulated hash registry, and appends the results. The website is updated from a validated publication snapshot; its as-of date tells you how far that snapshot extends.
Inside the method
An external scheduler invokes date-only updates without a three-day overlap or daily basket rediscovery. Failures resume checkpoints. Updates progress through calendar days, including weekends, with validated volume snapshots. Partial multi-narrative days cannot publish as complete.
Daily and historical attention have different retrieval-window corpora; weights are not guaranteed to be partition-invariant. The website is a published snapshot with an as-of date, not a promise of a continuously running feed.
INSIDE THE RARE EARTHS THEME
One market. Competing stories.
Select a mechanism to compare its beneficial and adverse statements.
