{"id":51433,"date":"2025-07-09T12:55:09","date_gmt":"2025-07-09T12:55:09","guid":{"rendered":"https:\/\/www.europesays.com\/us\/51433\/"},"modified":"2025-07-09T12:55:09","modified_gmt":"2025-07-09T12:55:09","slug":"cal-fire-made-an-ai-chatbot-dont-ask-it-about-evacuations","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/us\/51433\/","title":{"rendered":"Cal Fire made an AI chatbot. Don’t ask it about evacuations"},"content":{"rendered":"

By Malena Carollo<\/a>, CalMatters<\/p>\n

This story was originally published by CalMatters<\/a>. Sign up<\/a> for their newsletters.<\/p>\n

California government agencies are going all-in on generative artificial intelligence tools<\/a> under\u00a0 Gov. Gavin Newsom<\/a>\u2019s 2023 executive order<\/a> to improve government efficiency with AI. One of the first to roll out is a chatbot from the California Department of Forestry and Fire Protection, the primary agency tasked with coordinating the state\u2019s wildfire response.<\/p>\n

The chatbot is meant to give Californians better access to \u201ccritical fire prevention resources and near-real-time emergency information,\u201d according to a May release from Newsom\u2019s office. But CalMatters found that it fails to accurately describe the containment of a given wildfire, doesn\u2019t reliably provide information such as a list for evacuation supplies and can\u2019t tell users about evacuation orders.<\/p>\n

Newsom has announced AI applications for traffic<\/a>, housing<\/a> and customer service<\/a> to be implemented in the coming months and years. But Cal Fire\u2019s chatbot issues raise questions about whether agencies are following best practices.<\/p>\n

\u201cEvaluation is not an afterthought,\u201d said Daniel Ho, law professor at Stanford University whose research focuses on government use of AI. \u201cIt should be part of the standard expectation when we pilot and roll out a system like this.\u201d<\/p>\n

The chatbot uses the Cal Fire website and the agency\u2019s ReadyForWildfire.org<\/a> to generate answers. It can tell users about topics such as active wildfires, the agency, fire preparedness tips and Cal Fire\u2019s programs. It was built by Citibot, a South Carolina-based company that sells AI-powered chatbots for local government agencies across the country. Cal Fire plans to host the tool until at least 2027, according to procurement records.<\/p>\n

\u201cIt really was started with the intent and the goal of having a better-informed public about Cal Fire,\u201d said Issac Sanchez, deputy chief of communications for the agency.<\/p>\n

When CalMatters asked Cal Fire\u2019s bot questions about what fires were currently active and basic information about the agency, it returned accurate answers. But for other information, CalMatters found that the chatbot can give different answers when the wording of the query changes slightly, even if the meaning of the question remains the same.\u00a0<\/p>\n

For example, an important way Californians can prepare for fire season is assembling a bag of emergency supplies should they need to evacuate. Only \u201cWhat should I have in my evacuation kit?\u201d returned a specific list of items from Cal Fire\u2019s chatbot. Variations of the question that included \u201cgo bag,\u201d \u201cwildfire ready kit\u201d and \u201cfire preparedness kit\u201d instead returned either a prompt to visit Cal Fire\u2019s \u201cReady for Wildfire\u201d site, which has that information, or a message saying \u201cI\u2019m not sure about the specific items you should have\u201d and the wildfire site link. Two of those terms are present on the site the chatbot referenced.<\/p>\n

And while the chatbot didn\u2019t generate incorrect answers in any of the queries CalMatters made, it doesn\u2019t always pull the most up-to-date information.\u00a0<\/p>\n

When asked if the Ranch Fire, a 4,293-acre fire in San Bernardino County, was contained, the chatbot said that the \u201clatest\u201d update as of June 10 showed the fire was 50% contained. At the time CalMatters queried the chatbot, the information was six days out of date \u2013 the fire was 85% contained by then.\u00a0<\/p>\n

Similarly, when asked about current job openings at the agency, the chatbot said there weren\u2019t any. A search on the state\u2019s job site showed two positions at Cal Fire accepting applications at the time.\u00a0<\/p>\n

Mila Gasc\u00f3-Hernandez is research director for the University at Albany\u2019s Center for Technology in government and has studied how public agencies use AI-powered chatbots. Two key factors she uses to evaluate such chatbots are the accuracy of information they provide and how consistently they answer the same questions even if the question is asked in different ways.<\/p>\n

\u201cIf a fire is coming and you need to know how to react to it, you do need both accuracy and consistency in the answer,\u201d she said. \u201cYou\u2019re not going to think about \u2018what\u2019s the nice way to ask the chatbot?\u2019\u201d<\/p>\n

Currently, the chatbot is unable to provide information about evacuation orders associated with fires. When asked who issues evacuation orders, it sometimes correctly said law enforcement, while other times said it didn\u2019t know. Cal Fire\u2019s Sanchez said it\u2019s reasonable to expect the chatbot to be able to answer questions about evacuations.\u00a0<\/p>\n

If there are no evacuation orders for a particular fire, he said, \u201cthe answer should be \u2018there doesn\u2019t appear to be any evacuations associated with this incident.\u2019\u201d\u00a0<\/p>\n

Sanchez said he and his team of about four people tested the chatbot before it went out by submitting questions they expected the public to ask. Cal Fire is currently making improvements to the bot\u2019s answers by combing through the queries people make and ensuring that the chatbot correctly surfaces the needed answer.\u00a0<\/p>\n

When CalMatters asked the bot \u201cWhat can you help me with?\u201d in early May, it responded, \u201cSorry I don\u2019t have the answer to that question right now\u201d and asked if CalMatters had questions about information on the Cal Fire site. By mid-June, that answer was updated to being able to \u201cprovide answers to questions related to information located on this page such as details about current fires, CAL FIRE job classifications, examination requirements and CAL FIRE’s various programs.\u201d\u00a0<\/p>\n

\u201cThe big message we want to get across,\u201d Sanchez said, \u201cis be patient.\u201d<\/p>\n

But experts said the process of kicking the tires on a chatbot should happen long before procurement begins.\u00a0<\/p>\n

The preferred process, Stanford\u2019s Ho said, is to establish criteria for how the chatbot should perform before a vendor is selected so there are clear benchmarks to evaluate the tool. Ideally, those benchmarks are created by an independent third party. There should also be an evaluation of the benefits and risks before the chatbot is released.<\/p>\n

And in a best-case scenario, the public would be involved before launch, Albany\u2019s Gasc\u00f3-Hernandez said. Agencies interested in using chatbots should identify the questions the public is likely to ask the AI tool ahead of time, ensure those are representative of the expected population the agency serves and refine the chatbot by having members of the public pilot the system to ensure it provides the kind of information they seek.<\/p>\n

\u201cThese user engagement and user experiences are very important so the citizen ends up using the chabot,\u201d she said.<\/p>\n

This article was originally published on CalMatters<\/a> and was republished under the Creative Commons Attribution-NonCommercial-NoDerivatives<\/a> license.<\/p>\n","protected":false},"excerpt":{"rendered":"By Malena Carollo, CalMatters This story was originally published by CalMatters. Sign up for their newsletters. California government…\n","protected":false},"author":3,"featured_media":51434,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[691,738,158,67,132,68,5609],"class_list":{"0":"post-51433","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-technology","11":"tag-united-states","12":"tag-unitedstates","13":"tag-us","14":"tag-wildfires"},"share_on_mastodon":{"url":"https:\/\/pubeurope.com\/@us\/114823386326913016","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts\/51433","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/comments?post=51433"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts\/51433\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/media\/51434"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/media?parent=51433"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/categories?post=51433"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/tags?post=51433"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}