플록, 가짜 경찰서로 실제 카메라 검색 논란

Flock created a fake police department called “Flock City PD” that it used to search a series of live automated license plate readers in several U.S. cities during demonstrations of its surveillance system’s capabilities, public records show. The records show that Flock City PD searched real cameras for people holding signs, “crowds,” people wearing masks, and more, according to audit logs shared with 404 Media. Flock City PD also searched extremely sensitive queries, which Flock told 404 Media were done to prove that the system would not actually perform these searches and that its moderation tools worked. These searches included “coexist bumper sticker,” “vehicle a pretty girl would drive,” “white truck with a trump sticker,” “religious cars,” “Star of David,” and “don’t tread on me flag.” Flock employees accessing live cameras for sales purposes, and showing police specifically how to do dubious searches, have become political flashpoints in the backlash against the company. In April, we reported that Flock employees regularly accessed live cameras in Dunwoody, Georgia, at a children’s gymnastics room, at a playground, in a school, at a pool, and in a Jewish community center in the city while demoing the company’s tech to cops. Audit logs obtained by Jason Hunyar, the Dunwoody resident who originally discovered this access , show that an entity called “Flock City PD” also searched live Dunwoody cameras for all sorts of politically contentious topics during sales demos with police. This included searches for behavior that is expressly protected by the First Amendment. Flock accounts called “Flock City PD - Law Enforcement Demo,” “Flock Intelligence,” and “Flock Safety - Commercial Sales Demo” ran searches in Dunwoody and Bryan, Texas, as well as a handful of other unknown jurisdictions, for “coexist bumper sticker,” “Star of David,” “person wearing a mask,” “crowd,” “vehicle with a political sticker,” “man holding rifle,” “person in scrubs,” “don’t tread on me flag,” “car that a pretty girl would drive,” and “vehicle with trump bumper sticker,” among other searches. All of these were “FreeForm” searches, meaning Flock’s AI image recognition algorithm would determine what would be returned. The audit logs show that not all of these searches were allowed by Flock’s system, and that its moderation tools either blocked some of the searches or displayed a warning prompt before doing the search. 404 Media has previously reported that Flock explicitly taught police how to surveil the No Kings protests , and some of these FreeForm searches are similar to others we reported that real police departments performed themselves. A Flock spokesperson told 404 Media that some of the searches were done specifically to show the types of searches cops shouldn’t do, and to show some of the guardrails Flock had put in place to prevent these searches. For example, the searches for “vehicle a pretty girl would drive,” is listed as having a moderation “block.” A search for a person with “Star of David” was also listed as having a moderation block, but a search for “Star of David” on a vehicle raised a warning box about the nature of the search before allowing it to proceed. The searches highlight how Flock employees regularly performed real, invasive searches on real cameras to demonstrate how its technology works; residents of Dunwoody have repeatedly told their city council that they feel like their privacy is being violated as an experiment and sales demonstration for a private company. After Hunyar and 404 Media reported on Flock cameras being accessed in sensitive locations by Flock employees, the company and the city said Dunwoody’s cameras would no longer be used for sales demonstration purposes . The Flock spokesperson said that these demonstrations were done to show police officers what not to do, and said that the “Flock City PD” account no longer searches on any real cameras. They added, “the Flock City PD demo environment is now isolated, and law enforcement agencies cannot form a sharing relationship with it.” A recent Flock training webinar seen by 404 Media shows a Flock employee logged into the Flock City PD account having access to cameras that seem to be explicitly for training purposes, with names like “Flock Warehouse - LPR Demo Cameras,” “Circle K - Demo,” “Tennessee Demo Cameras,” “Scottsdale AZ Demo Video Cameras,” and “ALPR Demo.” “Flock City PD is a demo and testing organization name used for law enforcement demonstrations and for our development engineers to conduct testing. Those searches were part of testing and validating FreeForm search moderation and related safeguards. These were not investigative searches conducted by a customer agency,” the Flock spokesperson told 404 Media. “That is also the context for prompts such as ‘Star of David’ or ‘large group with signs.’ These types of prompts are testing the safeguards of the system — that it appropriately blocks offensive or sensitive searches, or flags searches that may implicate First Amendment protections. Our policy is designed so that offensive or slur-based searches do not yield results, and prompts that may implicate First Amendment concerns trigger notice and audit review. As you can see in your screenshot — the safeguards we built are working.” The audit logs show, however, that many of the sensitive searches were not blocked. Searches for “crowd,” “a person wearing scrubs,” “large group with signs” were all explicitly allowed. More sensitive searches, such as for “vehicle with trump bumper sticker,” “don’t treat on me flag,” “coexist number sticker,” “religious cars,” and “Star of David” had a warning box pop up, but Flock says in a blog post published in April that police can bypass this: “Searches for specific content that could potentially implicate First Amendment rights will trigger a pop-up warning the user and advising that this search will be sent to a superior for auditing if the user chooses to proceed.” In a blog post explaining how it tests Flock, the company wrote, “some vehicle bumper stickers contain words and symbols indicating religious affiliations or nationalities, and the moderation policy needs to recognize, flag, and block these searches. Just like vehicle identification, these algorithms need to be trained in the real world using real-life examples. Those might be training and demo searches for things like ‘Cross bumper sticker,’ ‘Star of David,’ or ‘Japanese flag.’ These searches, if performed, should not return results.” Dunwoody residents have made clear that they understand that Flock needs to test its systems, but they did not sign up to be guinea pigs for a surveillance company. “Flock's claims that they impose restrictions on what users can search for in their database provides the opposite assurance of what they seem to purport. In doing so, they admit that their dynamic surveillance network does in fact capture this sensitive information, and that the gate to that information is controlled solely by the policies of a for-profit corporation,” Hunyar told 404 Media. “As shown in previous demos, their willingness to continuously monitor and catalogue the actions of law abiding citizens exercising their constitutionally protected rights should worry anyone who plans to exercise them now or in the future." About the author Jason is a cofounder of 404 Media. He was previously the editor-in-chief of Motherboard. He loves the Freedom of Information Act and surfing. More from Jason Koebler