Why this matters
Cannabis law does not sit still. In the past two years New York has changed how medical patients register, opened its medical programme to visitors from other states, cut one cannabis tax and created a new kind of event licence. California has changed its excise tax twice and rewritten its rules on hemp products. If you ask a question about any of this, the answer you get depends entirely on where it comes from.
That is the case for a curated knowledge base: every fact checked against an official source, dated, and updated when the law moves. But a case is not evidence. So we tested it.
How we tested it
We took 24 everyday questions about cannabis law in California and New York, the kind people actually ask: How much can I carry? What is the tax rate? Can I use my out-of-state medical card? Is delta-8 legal? Twelve were about each state, and they were worded the way a member of the public would put them, not the way a lawyer would.
Each question was answered three ways:
- A general AI model on its own. Anthropic's Claude, answering from what it learned in training, with no access to search or documents.
- The same model with general web search. One or two searches per question and the pages they returned — roughly what a careful person does with a search engine.
- The same model with the Cannabis Knowledge Base. It was given only the five passages our search returned for that question and told to answer from those alone.
Every answer was capped at 120 words. Then a separate reviewer, which could not see which method produced which answer, scored each one against a reference set of facts, each tied to an official source — a statute, a regulator's page, a tax authority notice — checked on 11 October 2026.
The scoring was simple: 2 for an answer that was correct and current, 1 for one that got the main point right but missed or garbled a key detail, and 0 for one that was wrong, out of date, or did not answer.
What we found
| Method | Score (out of 48) | Correct | Partly right | Wrong | Out of date |
|---|---|---|---|---|---|
| Cannabis Knowledge Base | 48 | 24 | 0 | 0 | 0 |
| General web search | 39 | 16 | 7 | 1 | 3 |
| AI model on its own | 37 | 15 | 7 | 2 | 3 |
On settled questions all three did well. The rules on driving after cannabis use in California, off-duty use and employment, the types of business licence on offer — these have not changed much recently, and every method answered them accurately.
The gap opened on anything that had changed in the last two years. A few examples from New York:
- Getting a medical card. Both the stand-alone model and the web search described the old process of registering online and receiving a registry card. New York replaced that with Chapter 544 of the Laws of 2025: from 19 February 2026, patients certified by a practitioner are validated at the dispensary instead. Both answers scored zero.
- Using an out-of-state medical card. The stand-alone model said New York does not accept them. Since February 2026 it does, under section 32-A of the Cannabis Law.
- Tax. Both outside methods gave New York's medical cannabis excise as 7%. The New York Tax Department lists 3.15% for sales on or after 1 June 2024, the same date the state replaced its THC-based tax on distributors with a 9% wholesale tax.
- Events. The web search said temporary cannabis event permits were not available. New York's Cannabis Showcase Event rules have been in force since April 2026 (consumption is not allowed at them).
California showed the same pattern on a smaller scale: the date the current 15% excise rate runs to, and the new restrictions on intoxicating hemp products, tripped up the outside methods; the knowledge base had both.
None of this is a criticism of the general model or of search engines. A model's training has a cut-off date, and the web is full of pages written before the law changed that still rank well. Without a way of knowing which source is current, a careful-sounding answer can be two years out of date.
Finding the right passage
Answering well depends on retrieving the right passage first, so we tested that separately, with 140 questions covering every section of our California and New York profiles — from possession limits to social equity licensing and consumption lounges.
When the search was limited to the state in question, as our Regulations Advisor does, the most relevant section came back first in 93% of cases and within the top five in 99%. Without the state filter, those figures were 85% and 98%. Questions worded by professionals did slightly better than everyday wording, which tells us where to keep improving.
What this does not show
We want to be clear about the limits, because a result like 48 out of 48 invites more confidence than 24 questions can carry.
- It is a small test. Twenty-four questions in two states, chosen from topics where our profiles contain specific, checkable facts. It does not tell you how the knowledge base performs on a state we have not yet researched in depth.
- The reference facts came from our own research. Each one is tied to an official source, and the reviewer was blind to which method wrote which answer, but a test marked against our own work will favour it wherever that work is wrong. To guard against that, we checked every point where an outside answer disagreed with us against the official sources directly.
- The web search was done by an AI model, not a person. A determined researcher reading statutes line by line could match the knowledge base. The point is how much work that takes, and how easily it goes wrong.
- Legal information is not legal advice. The knowledge base describes what the law says and when it changed; it does not replace a lawyer for a specific situation.
The evaluation was useful in a second way too. Checking the disagreements line by line surfaced a few places where our own entries could be more precise, and because each fact sits in one place with its source attached, we were able to correct them the same day. That is the less glamorous half of a knowledge base: not just being right on the day you publish, but being quick to fix it when you are not.
Where this goes next
California and New York are the first two states built to our new 34-section template, which covers everything from possession limits and licensing to delivery, housing and family law. Florida, Illinois and Michigan are next, and we will run the same tests on each before it goes live — new questions, the same blind marking, and the results published here.