AL-HOL, Syria—A refugee camp set up to shelter women and children after the defeat of Islamic State has instead turned into a mini-caliphate itself, where female leaders nurture the group’s violent ideology and run money-making schemes that help keep the insurgency alive outside the razor wire.

Authorities have recorded more than 40 murders, at least 10 of them beheadings, in the 736-acre camp since the start of the year. Most of the victims were accused by Islamic State of collaborating with camp authorities, according to the Rojava Information Center, an independent volunteer-staffed research organization based in northeast Syria.

Late last year, an Iraqi woman was strangled in front of her children with an electric wire after she posted a video on Instagram of herself dancing in her tent, an activity forbidden under the strict Islamist doctrine imposed by the group, say people in contact with al-Hol prisoners. On March 20, another 18-year-old woman, also Iraqi, was shot dead after being accused of being an informant, these people said.

The U.S.-backed Syrian Democratic Forces, a Kurdish-led militia responsible for guarding al-Hol, say they lack the resources to properly secure the camp, which has grown into a small city of more than 62,000. Most Western nations have refused to systematically repatriate citizens who moved to Syria during Islamic State’s rise and were trapped there after its collapse.