From news
to evidence.

Rare earths offered a way to examine upstream industrial activity, from mining and processing to magnet manufacturing. Government policy and demand changes could affect each part of that chain differently. An export restriction, for example, could help one producer’s competitive position while hurting a manufacturer that relied on the restricted material.

The first requirement was a defensible company basket. Early versions admitted companies with weak connections to the theme and missed major producers. Reviewing those omissions led us back to search coverage, ranking cutoffs and listing mappings. Duplicate identifiers also needed to be resolved before we could attribute evidence consistently.

We used Bigdata’s Search API, Connected Entities and Co-mentions API to discover candidates, then checked their business exposure, listings and market eligibility. Co-mentions was particularly useful: exploring companies reported together around a topic gave us better leads than broad searches alone. I discovered it about six weeks in, which was excellent timing for my three-day project. It remains my contender for best thing since sliced bread.

Co-mentions supplied leads, not proof. Appearing alongside rare earths does not establish business exposure, and appearing alongside another company does not establish a supplier or customer relationship. Each candidate still needed evidence of why it belonged.

One of the most useful improvements was a change in language. My early queries described economic mechanisms in technical terms. The reporting we needed described a company receiving funding, a delayed plant or a signed customer agreement. We were asking search to bridge an unnecessary gap between our terminology and the language of the news.

We made the queries “newsy”: concrete, reportable events expressed in ordinary article language. This surfaced evidence that more elaborate phrasing had missed. Precise definitions still governed what qualified for the dataset. Broader retrieval also brought noise, so we reviewed the results to check whether each revision improved useful coverage.

We organized the theme into narratives such as government support, export policy, mining supply and magnet demand. Each narrative contained mechanisms and statements describing how they could develop. Funding being announced, becoming available and supporting construction are distinct stages; mentioning a government and a mine does not establish which occurred.

Each mechanism had beneficial and adverse statements, with queries covering both. Searching only for expansion and investment would give us an unusually enthusiastic account of the sector. Delays, constraints and reversals needed deliberate coverage too. The Rare Earths taxonomy reached 11 narratives, 25 mechanisms and 50 statements, each with targeted queries.

We then extracted company-event triplets: a company, a predicate such as “benefits from” or “is harmed by,” and one specific event. An illustrative example is “Producer A benefits from a customer signing a long-term purchase agreement.” Each part can be checked against the source, including whether the claimed effect on that company is supported.

Consistency proved harder than defining the format. Models could attribute an event to the wrong company, combine separate developments or infer an unsupported impact. Some extracted the share-price reaction instead of the underlying business event. We tested semantic-event extraction and more explicit company-event pairing. Higher triplet counts were only useful when the additional records survived inspection.

Quality control accounted for much of the development. We reviewed company attribution, statement classification, duplicate handling and scoring together, because errors in one stage could distort the next.

An event can legitimately belong to multiple statements. Earlier versions handled this poorly; allowing multiple memberships then required us to prevent repeated contributions from the same event. We also tightened classification around the extracted claim. An article discussing financing and mining does not make every financing event evidence of additional mining supply.

Relevance and company impact needed separate treatment. An event can be relevant to an adverse statement while contradicting it, or benefit a company within a negative-sounding narrative. We tested a separate stance layer for support and contradiction, then disabled that version after reviewing its results. It was an experiment, not part of the active scoring described here.

AI accelerated implementation, while manual inspection exposed errors that successful runs and valid JSON could not. I repeatedly reviewed charts and triplets, revised the process and checked the output again. Aakarsh’s reviews often found cases I had missed. “Nearly finished” became a phrase I used more carefully.

Repeated coverage needed particular care. One announcement could appear in several articles, match multiple queries and qualify for several statements. Without a stable event identity, those observations could look like independent developments.

We used exact hashing around the canonical company, predicate and atomic-event text. Date was excluded from that identity, so an identical triplet reported later did not become a new event. We retained later observations and source links separately from the event’s first contribution.

Exact hashing made duplicate handling reproducible, but differently worded accounts of the same announcement could still produce separate identities. Resolving those cases requires semantic event matching; exact hashing alone does not solve it.

Freshness was classified separately as a new development, material update, historical background or uncertain evidence. A newly published article can contain an old event. We kept historical background in the research record without treating it as fresh impact, and preserved uncertainty rather than assigning false confidence.

Scoring separated the direction of company impact from its assessed economic strength, attention, source authority and freshness. This let us distinguish a material development from a recap, while using repeated coverage as information about attention without treating every report as a new event.

Attention considered document frequency, relative evidence density and publisher breadth. We also retained eligible documents that produced no triplets. Leaving those out would remove part of the denominator and distort the calculation. Reconstructing attention later made the importance of preserving that underlying document data especially clear.

The charts revealed another interpretation problem. In one audit, 53 of the 55 largest downward narrative moves were dominated by evidence leaving the rolling window. On 47 of those days, no new evidence arrived. A falling line could therefore reflect older supportive evidence expiring, even without a new adverse development.

We explained the rolling window, marked the warm-up period and let readers inspect the evidence contributing to a move. Those details belong in the interface: a user should not need to read the implementation to interpret a chart.

MP Materials’ Department of Defense deal illustrated the kind of exploration I wanted to enable. Readers could examine a substantial business development alongside market activity and open the reporting behind it. That connection helps investigate what happened; it does not establish that the index predicted the price move.

The website connects basket and individual-company charts to an evidence table. Readers can click a day or drag across a period to inspect its developments, then explore the theme’s narratives and mechanisms through an expandable taxonomy.

Several visual treatments preceded the full-width table. Simplifying that view made room for the evidence and reduced the work required to reach it. The goal was a clear path from a company or period of interest to the underlying reporting.

We consolidated processing into a Python service with a pinned basket, taxonomy and configuration. The daily updater retrieves the requested date and uses checkpoints and exact deduplication to extend the dataset while preserving accepted historical scores.

For one catch-up, three days ran in parallel in isolated databases. A single process then merged them chronologically into the canonical dataset. We checked historical scores and event identities after merging, keeping parallel processing separate from updates to shared history.

Security, compliance and legal review also needed time in the plan. In financial services, “the code works” is an encouraging update, not a release checklist. We addressed authentication, workload limits, private versus public exports and source-preview restrictions. Docker packaging made the service easier for others to run and review. None of this was optional, regardless of my original estimate.

The verified October 14, 2026 snapshot contained more than 10,000 unique events across 16 companies and 11 narratives. The packaged service included a repeatable generation process and a daily updater, so another operator could continue the dataset without reconstructing my sequence of scripts.

I’m proud of taking an open-ended idea to a reproducible proof of concept, doing much of the build myself at 23. RavenPack’s team helped make it better by questioning the evidence and challenging the output. That scrutiny improved both the system and how I evaluated it.

This project taught me how much depends on choosing the right companies, retrieving the right reporting and checking what a score represents. Those decisions shape the research experience as much as the chart itself.

The ambition is to apply the process to more themes, potentially hundreds of trend experiences. Repeatable infrastructure helps, but each theme still needs a reviewed basket, a well-defined taxonomy and careful evidence inspection. Bigdata Trends gives investors a way to move between a broad theme and the specific developments affecting its companies.

Early-catalyst and forward-earnings research still interests me. For the next phase, my focus is on expanding theme coverage while preserving the quality controls established here. I will also be giving future time estimates a little more room.

Cheers to increasing shareholder value.