The International Olympic Committee (IOC) has insisted that the Olympics, which begin in two weeks, will be “safe” from COVID-19. But a fresh surge of virus cases in Japan is casting doubt over just how safe the event will be—particularly for the Japanese people.

On Thursday, the Japanese government imposed a COVID-19 state of emergency on Tokyo just three weeks after lifting restrictions—as daily infections hit a two-month high. The order ensures the entire 2020 Tokyo Olympics will take place with the host city under emergency measures. Authorities also canceled an earlier plan to allow up to 10,000 Japanese spectators at events; athletes will compete in Tokyo without crowds.

Despite the latest COVID-19 countermeasures, and the safety precautions in place for the 60,000 athletes, coaches, journalists and others who will travel to Japan for the Games, health experts—who have been warning with increasing fervor for months about the risks—say that they’re concerned that the Olympics could be the catalyst for a major outbreak across the country.

“Medically speaking, I’m not very worried about the Games per se, particularly for athletes,” says Kentaro Iwata, an infectious disease expert at Japan’s Kobe University. “But it might create the atmosphere to spread the disease all over Japan.”