Interview transcript:

Terry Gerton A couple of months ago, we covered your first report in this disaster assistance high-risk series where you looked at the federal response workforce. You’re back with report number two, looking at state and local response capabilities. Talk to us about the headlines.

Chris Currie The headline for this report is that the capabilities of state and local governments across the country vary drastically for a disaster or other type of event. You know, what we did is we actually look at data that the states prepare and provide to FEMA as part of their justification for federal preparedness grants. It’s meant to be a very, very honest self-assessment of capabilities. And for that reason, we actually don’t provide states individually, we sort of roll it up and wrap it up anonymously because some of that information, as you imagine, could be sensitive. We looked at states that have been involved in major disasters over the last two to three years, and some of these states are very experienced, large states, and even they vary in terms of their capabilities. There’s actually 32 capabilities that FEMA sets in the National Preparedness System that you want to achieve to be prepared to respond for a disaster or a large event. And states vary. Some of the areas, they were less than 10% prepared — met less than 10% of those capabilities — and others, they were much more. So the reason that’s important right now is to understand that if you were to change the support that FEMA and the federal government provide to states quickly, then they’re going to have capability gaps that are going to have to get filled.

Terry Gerton Let’s talk about some of the support that FEMA does provide. One of the ways that they support the states is through preparedness grants, and those help build local capacity. What did you find as you dug into the preparedness grants?

Chris Currie Those preparedness grants started after 9/11, and since 9/11, there’s been over $60 billion provided to states. It’s the main way that the federal government transfers funds to state and local governments to get them ready to handle something bad that could happen, not just a natural disaster, but it could be a terrorist attack. And those grants have built capabilities tremendously over the years. But those capabilities change over time, and we identify through real-world events and exercises the gaps that still need to be addressed. So I’ll give you a great example. After Hurricane Helene and after other disasters, housing for disaster survivors is always a perennial challenge. Housing is a capability area that is assessed and we want to build up through these preparedness grants. It’s an area that states, even very experienced disaster states, still fall short of in terms of their capabilities. And the federal government kind of comes in after a disaster and provides a lot of that support because states don’t. So if the federal governments not going to provide it, then someone else is going to have to provide it. And that’s going to be someone at the state or local level.

Terry Gerton Talk to me about the flexibility and the allocation framework for these grants. Is it meeting requirements? Does it seem to be focused on the places that have the greatest need?

Chris Currie There’s a couple different ways they’re given out. There’s a portion of the grants that are supposed to go towards certain national priorities, and FEMA sets those targets. So think about things like election security or other national priorities. But then a large part of the grant, they’re discretionary, and the states can use them and they’re supposed to use them in the areas where they assess they have gaps. And that’s the data I was talking about earlier that we provided. For example, certain states may have gaps in their ability to handle a mass casualty situation or may struggle to house disaster survivors because they don’t have a lot of housing stock or rental. So those are things they’re supposed to identify and then target those grants towards those specific areas, which makes sense. You want to close your gaps so you’re ready to go when something happens.

Terry Gerton FEMA also provides a great deal of training and technical assistance. How effective has that been in helping states be ready?

Chris Currie This is, I think, one of the biggest success stories since Hurricane Katrina. If you remember Hurricane Katrina, the issue was the role of various levels of government was not clear, and thus, nobody stepped up and was proactive in responding to that event. And people lost their lives. Since that time, the National Preparedness System and FEMA leading that has been extremely effective through exercises, through training, through just regional relationships in taking care of a lot of those problems. So today we are way more proactive and responsive to disasters than we were 20 years ago in Hurricane Katrina. So that’s a huge success story. Having said that, a disaster is a disaster. There’s always going to be things that happen that you don’t expect. And there’s areas where states still have major gaps and require resources and people to address those. And the federal government comes in fills a lot of those gaps. Here’s a great example. Hurricane Helene happened and devastated a very remote part of our country in places like rural Tennessee and North Carolina and Virginia. States and localities don’t have the search and rescue assets for such a large swath of that kind of terrain. Federal government provided a lot of that. They provided a lot of the air support, the land support, the temporary bridges — Army Corps of Engineers. You know, the federal government really kicks in when something’s too big for a state or locality to handle.

Terry Gerton I’m speaking with Chris Currie. He’s director, Homeland Security and Justice at GAO. So Chris, all of this begs the question. This administration has been very clear that it wants states and localities to pick up more of the disaster response mission and that it wants a much smaller FEMA. Given what you found in your first study about the federal response workforce and the impacts of downsizing there, and now the variability in state and local readiness, what are the implications for national disaster response?

Chris Currie I want to make one thing really clear, because all I know is what we know now and the data that we’ve looked at. And I want it to be clear that nothing has changed in terms of FEMA’s responsibilities today. There’s been a lot of talk about it. There’s the president’s council that studied it. But there has been no change so far. So FEMA is still responsible for what it was responsible for two years ago. They have lost some staff. We looked at that in our first report, as you mentioned. They have lost about 1,000 staff, and maybe a little bit more than that, at this point, but they haven’t been cut drastically or cut in half as has been discussed. So they still have the same responsibilities and they’re still performing the same functions on disasters throughout the country, even though last year we didn’t have a huge land-falling hurricane. So what’s important about that is that everybody’s waiting to hear what the next steps are going to be and what’s going to happen to FEMA. One of the things we wanted to do in this report is we wanted to provide a comprehensive picture of preparedness to show what’s going to be necessary if that FEMA support is pulled back or FEMA is made smaller. And the bottom line is that states and localities are going to have to do more. However, it’s going to be critical that they have the time to prepare for that. For example, a lot of the assistance that’s provided to individual survivors, like cash payments and housing, that comes from the federal government. It does not come from the state or local government. So if FEMA is not going to be providing that, the state of the locality is going to have to fill that need. And that requires a lot of money and a lot preparation and planning that you can’t just turn on in a heartbeat. You don’t want to start figuring out programs to help people after a disaster happens.

Terry Gerton You bring up a good point on that time to prepare. As you did the survey, you talked to lots of state and local response officials. What did they tell you, beyond time to prepare, that they were going to need to be effective?

Chris Currie Very simple: Just tell us what we need to do. Tell us what were going to expect from you, the federal government. Nobody knows right now. The FEMA Council has not finished its work. There has been reform legislation introduced in the House and in the Senate, but nothing has passed yet. So the key message is, tell us what the roles and responsibilities are going to be so we know what to prepare for, so we don’t get caught flat-footed in the case of something really bad happening. One of my fears is that last year, like I said, we didn’t have a large land-falling hurricane. It was the first year in a long time we did not. We did not have a catastrophic disaster, other than Los Angeles fires early in the year. So my fear is that folks are going to look at last year and say, hey, things have gone pretty well. We don’t need to be thinking about it. And that is an absolute mistake. Because we’ve seen in years like 2017, 2018, 2024 — my fear is we’re going to have another situation this year or next with multiple concurrent disasters, and we’re just not going to the resources to deal with them.

Terry Gerton So what will you be watching for in the next few months to see if Congress and the federal government and the states have taken your recommendations on board?

Chris Currie Well, when the FEMA Council report comes out, I would like to see, in whatever the execution is for FEMA reform or the changes in how the system works now, an understanding of how this needs to be rolled out so states and localities can prepare and have as clear roles and responsibilities as possible. We’d also like to see them address many of the problems that we’ve pointed out. And to be clear, we’ve pointed out a number of issues with FEMA, particularly in the frustrating recovery phase. I want to see that they’re making sure that we don’t break what’s not broken and we fix the issues that are broken. And there are a number those things.

The post Readiness gaps may leave communities vulnerable when the next disaster strikes first appeared on Federal News Network.

Interview transcript:

Terry Gerton I suspect that most people don’t know that Medicare funds graduate medical education. This is a topic you’ve looked into quite a bit at GAO. Give us an overview of this situation first.

Leslie Gordon Medicare is the biggest funder of graduate medical education. There are a number of federal programs that fund graduate medical education, but Medicare is the primary funder, funding about $22 billion in 2023. We’ve been looking at it over a course of years, because that’s a chunk of change to look at and understand if it’s being effective. Over the course of our work in 2017, 2018, we’ve noticed that there’s a misalignment; there’s an unevenness in how medical residency is distributed across the country and where those graduate medical education dollars go.

Terry Gerton How’s it supposed to work?

Leslie Gordon Well, the impetus behind funding for graduate medical education is to ensure we have a well-trained workforce. And indeed, it should be distributed across the country so that people have access to services. And Medicare cares about having trained providers to care for all the Medicare beneficiaries across the country.

Terry Gerton So the Consolidated Appropriations Act of 2021, if we can take our minds back that far, put in some provisions maybe to try to address this misallocation of graduate medical education, and you’ve just published a report on that. Tell us about what you found in terms of the first three years of this program.

Leslie Gordon Our report is an interim report. The Consolidated Appropriations Act of ’21 funded 1,000 new residency positions. That’s in a framework of over 163,000 medical residents. So it’s not a lot of positions, but it’s designed to alleviate situations where hospitals would like to expand, or smaller hospitals would like to open new medical residency programs, and to direct some attention to underserved communities.

Terry Gerton And as you looked at the distribution of those thousand positions, what did you find?

Leslie Gordon We found that awarded hospitals were very similar to those that applied who were not awarded. So we did a comparison to look at, were there any biases in how these positions were being distributed when CMS was distributing them? Or, did they follow the rules and the categories set out in the CAA? And generally, the awardees and those that were not awarded were similar in nature. There was an effort and an impetus to focus on underserved areas, particularly rural areas, as we noted earlier. And while there wasn’t a redistribution of a majority of positions going to rural areas, there was an emphasis and a success in funding the rural hospitals that applied. Ten rural hospitals applied, nine were awarded. That’s different from about 50% of urban hospitals that applied that were awarded.

Terry Gerton Sounds like this really didn’t get to the crux of putting more residents in urban hospitals.

Leslie Gordon It didn’t put more residents necessarily in urban hospitals. It put more residents everywhere, let me say that. It emphasized and allowed for more residency positions and programs in rural and underserved areas. But with a small portion of residencies that were being awarded, it’s a big pile to redistribute and only have a thousand pieces with which to do that.

Terry Gerton I’m speaking with Leslie Gordon. She’s director for Medicare at GAO. Are there particular issues that the smaller or rural hospitals faced in terms of applying for this program or really making the best use of the resources that could potentially be available?

Leslie Gordon In the course of our work, we talked to representatives from hospitals in all kinds of areas, including rural hospitals. We talked to some hospitals and we talked to hospital associations. We heard that CMS’s use of the health professional shortage area as the primary criterion for distributing and allocating prioritization of who would be awarded got in the way for some smaller communities and those that might be training in rural areas or serving rural residents that traveled out of their local area to seek care. The way in which it didn’t quite land for rural areas is that the HPSA score is based on a population-to-provider ratio. And if you add one new doctor to population of a thousand people, that can really change the score a lot, as opposed to adding one new doctor to a population of 200,000 people. In that way, it wasn’t quite aligned with the goal of focusing on rural areas.

Terry Gerton And so, are there changes that CMS could make to this distribution model in the last couple of years of this program to help address those shortcomings?

Leslie Gordon We provided this feedback and other feedback to CMS as a part of the course of our work. They have two more rounds to distribute. And one of the things we also learned about is that hospitals needed other funding to make good use of these additional spots, these additional residency spots. I think being more aware of upfront costs, the need to maintain accreditation, and some of the challenges that we highlight in our report will help CMS, perhaps, and those who apply.

Terry Gerton If CMS does adjust the criteria or support, are there metrics that they should track to make sure that the changes are working?

Leslie Gordon CMS is tracking the metrics that were set out in the Consolidated Appropriations Act, and we will be looking at and reporting again in 2027.

Terry Gerton So when you think about this over the next couple of years, are there things that Congress, or educators, or other folks should watch to gauge whether or not these new slots are meeting workforce goals? Are they helping advance the accreditation or the certification of young medical students?

Leslie Gordon I think the experience of awarding these positions helps highlight that it’s not just funding that will solve the problem in terms of distribution of medical residency. There’s a support infrastructure that needs to be there in terms staffing, in terms equipment, in terms considering the types of experiences that medical residents need to have to be fully trained. So we cover all these things in our report and I think that this experience with the allocation of the thousand positions helps highlight all the infrastructure that’s needed to support medical residency training.

Terry Gerton Are there companion programs designed to address those infrastructure shortcomings?

Leslie Gordon The federal government actually has 72 programs. Yes, there’s a lot of programs and there are 72 health care workforce programs. And we have open recommendations from our prior work that HHS needs to examine the gaps in the workforce and take action to address those gaps and needs to communicate around them. We have other open recommendations that they don’t have the information necessary to identify and evaluate the cost effectiveness of those 72 programs. So we do have open work, not directly related to this report, but we have open recommendations that focus in on the need to have better information and truly evaluate the effectiveness of all of the Work First programs in a comprehensive way.

Terry Gerton And do you have a sense that CMS and HHS are taking on that task to sort of harmonize all of these programs so that they make sense and they are optimized for best outcomes?

Leslie Gordon They are making progress on our recommendations and we will continue to follow up to see how they progress.

The post Medicare-funded medical residencies falling short of goals: Report first appeared on Federal News Network.

Interview transcript:

Stephen Bacon We’re seeing a lot more protests, particularly at the Government Accountability Office, in the last several months that have been filed using AI — a party that’s not represented by counsel using AI to generate the protest and then file it. But we’re seeing some problems with that in some of the decisions that are coming out of the GAO.

Terry Gerton So tell me more about how companies are using AI. You mentioned that they’re doing this without the help of legal counsel as well.

Stephen Bacon That’s right, at least the ones that we’ve seen so far in public decisions at GAO. It’s not entirely clear how the protesters are using it, but we can imagine that maybe they’re taking the debriefing information that they’re getting from the agency, they’re uploading that into an LLM like ChatGPT or Claude, and using it to develop a protest argument that they can file with the GAO. And what we’re seeing in the decisions is that many of the protests that have been filed using AI contain hallucinations. Case citations that don’t exist to actual cases that have decided by GAO. So the legal precedent that the protesters are relying on, in fact, don’t exist. And that’s one of the inherent limitations of LLMs is that they hallucinate. They come up with decisions, citations that don’t exist. To be clear, we’re not just seeing this by protesters that are not represented by counsel. This is happening in courts all across the country where attorneys are using AI to help generate legal filings and then getting in trouble with the courts when those citations don’t actually exist. Because when you file a protest or any kind of legal filing that has a citation in it, the court is relying on you to make an accurate representation that the legal authority that you’re relying on is in fact correct and is in fact a decision that has been issued in the past. And so both courts and the GAO now are saying that you can get in trouble, you can be sanctioned as a protester if you submit a protest that has some kind of fake citation that’s inaccurate.

Terry Gerton What does that mean to be sanctioned as GAO reviews the case?

Stephen Bacon At GAO, they have inherent authority to sanction protesters, and really the main sanction that they have is to dismiss a protest. If you happen to file a protest that contains fake citations, they reserve the right to dismiss your protest. Even if you have legally valid grounds to protest — maybe you have identified an error in the agency selection process — if GAO determines that you relied on fake citations in your protest, they could dismiss the protest, even if the actual merits of it may have some validity to it.

Terry Gerton So there’s some interesting intersections of situation going on here, I think. There’s a lot of uncertainty on the contractor side about the new FAR regulations and how those are going to be enforced, certainly across different agencies. We’ve had a reduction in the contractor workforce, so there are fewer contractors managing more acquisitions. And now we have AI coming in to sort of simplify, but potentially also make much more complex, the whole protest market. So do you expect all of this to be leading to an increase in protests? And what does that mean for GAO as they’re trying to sort out the validity of all the claims?

Stephen Bacon I think it certainly has the potential to, if what we’re seeing in the decisions is a trend towards more pro se protesters — pro se being parties that are not represented by counsel — using AI. To the extent that that trend continues, I think that there’ll be a lower barrier for protesters to file at GAO if they think that they can use an LLM to generate a protest without having to spend legal fees on outside counsel. Which is understandable, particularly for small businesses who may have resource constraints. If they feel like they can use an LLM to help them challenge an award decision, we may see more of that at GAO. I think what GAO is saying in these opinions that have come out…at first, they’re warning protesters that using LLMs that create fake citations is sanctionable. They didn’t actually take the step of issuing a sanction. But finally, in the last several months, we saw that they did, in fact, take that step of dismissing a protest, actually several protests that were filed by the same company, that contained fake citations. They actually took that step and dismissed those protests on the grounds that they misrepresented legal authority in their filings with the GAO.

Terry Gerton I’m speaking with Stephen Bacon. He’s a partner in the government contracts practice group at Rogers Joseph O’Donnell. You mentioned small businesses and their capacity constraints in terms of they may not have in-house counsel, they may have a lot of folks who can review all of this. But does this have the effect of sort of adding some equality into the protest market where they can use AI to submit? And do you think then that that’s going to change the protest space? Is this just the tip of the iceberg in terms of transformation?

Stephen Bacon It certainly lowers the barrier for companies. The GAO was set up to be a relatively informal forum to allow for the quick and efficient resolution of protests. I don’t think what GAO is saying necessarily is that AI cannot be used. But what they are saying is that we have a process to resolve bid protests and we want to maintain the integrity of that. And if you’re going to use AI, you need to be sure that you verify that what you’re filing is accurate. For anybody that is thinking about using AI to generate a protest, there needs to be some level of quality-checking of what is in the draft that’s generated by an LLM to be sure that you’re making accurate representations to the GAO in your protest. So that means checking the legal citations to make sure that the cases actually exist. That basic level of quality-checking needs to happen. Otherwise, GAO could just be flooded with protests that have no merit and that have lots of inaccuracies in them. And that’s not going to help them resolve protests in a way that’s efficient and achieves their ultimate goal.

Terry Gerton So where do you think we go from here, and what’s your guidance to the companies who are considering using AI to file their protests?

Stephen Bacon For any company that’s contemplating using an AI to generate protests, the basic point: If you’re going to do it, you have to verify that the citations are accurate. You have verify that what an LLM is generating is citing to a decision that has been published by GAO in the past. And that’s relatively easy to do. GAO has all of their decisions on their website, and you can go and check those and verify not only that the citations are accurate, but the legal proposition that you’re asserting is supported by the case that’s being cited. That’s important, too. That’s kind of table stakes. But the other thing I would say is that what we’re seeing in a lot of these decisions, where it’s obvious from the decision that AI has been used and that GAO is pointing out that there are these fake citations, is that oftentimes those protests are being dismissed for procedural defects as well. So things like timeliness and bid protest standing. Those kinds of procedural issues are being missed by the protesters who are using LLMs to generate the filings. And that’s because of another inherent limitation of an LLM; it often will tell you what you want it to say in a lot of ways. So if you tell the LLM, generate me a protest on this issue or that issue, it will do that and it might produce something that looks, on its face, credible and compelling. But if you don’t have the domain knowledge of the timeliness rules and the standing rules, you’re often going to overlook those things and the LLM is not going to catch it for you. And so you may be in a situation where you file something that looks on its face credible, but is in fact an untimely protest.

The post AI-crafted bid protests are on the rise, but what’s the legal fallout? first appeared on Federal News Network.