메뉴
BL
404 Media • 37일 전

‘서틀페이크’: 살짝 변조된 AI 비동의 이미지가 X를 장악하다

IMP
7/10
핵심 요약

딥페이크 이후 등장한 새로운 형태의 합성 매체인 '서틀페이크(subtlefake)'가 X(구 트위터)에서 확산하고 있습니다. 유명인의 실제 사진을 AI로 미묘하게 수정해 더 선정적으로 보이게 만든 이 이미지들은 기존 딥페이크보다 구별이 어렵고 더욱 그럴듯해 보입니다. 문제는 기술만 발전하는 것이 아니라 이를 악용하는 사람들의 수법도 정교해지고 있다는 점이며, X는 참여 유치형 계정에 노출 기반 보상을 지급하며 이러한 문제를 사실상 방치하고 있습니다.

번역된 본문

내가 처음 딥페이크를 본 순간은 생생하게 기억한다. 2017년, Vice 사무실로 가는 길에 기차를 기다리던 중 레딧에 올라온 것을 본 사람에게서 트위터 다이렉트 메시지를 받았다. 그것은 포르노 영상 도입부의 짧은 .gif 파일로, 침대에 누워 카메라를 향해 이야기하는 여성이 나왔는데, 그 여성의 얼굴이 갤 가돗(Gal Gadot)처럼 보이도록 편집되어 있었다. 그 딥페이크는 당시 기준으로 놀라울 정도로 사실적이었고, 그날 이후 샘의 보도가 보여주듯 딥페이크는 주로 여성을 괴롭히고 모욕하는 데 사용되어 왔다. 우리는 당시 그 기술이 얼마나 앞서 있는지 놀랐고, 인터넷과 법이 이를 다룰 준비가 되어 있지 않다는 것을 알았다. 그 이후 기술은 기하급수적으로 발전했지만, 인터넷과 법은 여전히 따라잡지 못하고 있다.

하지만 딥페이크가 처음 보았을 때부터 분명 우려스러웠던 만큼, 문제는 갤 가돗이 실제로 성인 영상을 찍었다고 우리가 믿었다는 것이 아니었다. 딥페이크는 그럴듯해 보일 수 있지만, 우리가 보고 있는 것이 진짜라고 속인 적은 한 번도 없었다. 그런데 지난 몇 달간 나는 X에서 많은 사람들을 속일 것이라고 생각하는 새롭고 기묘한 유형의 AI 생성 비동의(非同意) 이미지를 보아왔다. 내가 '서틀페이크(subtlefake)'라고 부르기 시작한 이 이미지들은 그렇게 명백히 공격적이지는 않지만, 훨씬 더 그럴듯하다. 우려스러운 시사점은 이제 기술만 발전하는 것이 아니라, 이를 악의적으로 사용하는 사람들 역시 더 정교해지고 있다는 것이다.

2019년 데이터 앤 소사이어티(Data & Society)가 AI나 다른 첨단 기술 없이 진실을 조작하는 편집된 미디어로 정의한 '치프 페이크(cheap fake)'와 마찬가지로, 서틀페이크는 2017년 딥페이크의 등장 이후 인터넷을 장악한 또 다른 형태의 합성 미디어다. 이 더 그럴듯한 이미지들은 유명인의 AI 생성 누드나 포르노 영상에 얼굴을 합성한 것이 아니라, 유명인의 이미지를 더 노출이 심하거나 선정적으로 보이도록 미묘하게 편집한 AI 생성 이미지다.

우리는 유명인과 일반인의 AI 생성 비동의 미디어에 대해 셀 수 없이 많은 기사를 써왔다. 이는 사람들에게 심리적·물질적 피해를 입히는 괴롭힘의 한 형태이며, 그러려면 그럴듯해 보일 필요조차 없다. 이미지가 사진처럼 사실적으로 보이더라도, 세계 최고의 배우가 갑자기 하드코어 포르노를 찍어 아무런 익명 계정이 X에 게시할 가능성은 거의 없다는 단순한 이유로 대부분 명백히 가짜임을 알 수 있다.

최근 내가 보고 있는 이미지들은 대체로 아주 살짝만 변조되어 있다. 예를 들어 한 유형은 유명 배우가 레드커피트를 걷는 실제 사진을 편집해 드레스가 더 노출되게 하거나 엉덩이나 가슴을 더 크게 만든다. 다른 유형은 운동 직후 체육관에서 찍은 셀카처럼 보이는 완전히 AI 생성된 이미지다.

내가 완전히 안심하고 공유할 수 있는 유일한 사례는 이런 가짜의 대상이었던 배우 소칠틀 고메스(Xochitl Gomez)가 지적하며 직접 공유한 것이다. 그녀가 인스타그램에 공유한 나란히 놓은 비교 사진에서 보듯, 누군가 주차장에서 어깨 너머로 카메라를 보며 웃고 있는 그녀의 실제 사진을 가져와 AI로 한 이미지에서는 몸을 굽히는 것처럼, 다른 이미지에서는 손을 엉덩이에 올리는 것처럼 보이게 만들었다. 마찬가지로 누군가 그녀의 레드커퍼트 사진을 가져와 카메라에서 등을 돌리고 혀를 내미는 것처럼 보이게 만들었다.

이 이미지들은 단순히 사진처럼 사실적일 뿐만 아니라, 우리가 비동의 이미지에서 일반적으로 기대해 온 통상적인 맥락적 단서도 없다. 나는 AI 생성 이미지를 식별하는 일이 우리가 여기서 하는 일의 큰 부분이기 때문에 매우 잘 알아본다고 생각하지만, 그냥 X를 스크롤하다가는 이것들이 AI 생성이라고 말할 수 있을지 모르겠다.

이 이미지들은 거의 항상 X의 인증된 참여 유치(engagement farming) 계정에 의해 게시되는데, 이 계정 운영자들은 게시물이 충분한 노출을 얻으면 보상을 받는다. X는 사람들이 이런 일과 이만큼 나쁘거나 더 나쁜 다른 일들을 하는지 전혀 개의치 않는다는 것이 분명하다. 나는 최소한 이런 이미지 몇 개를 공유한 계정을 하나 목격했다.

원문 보기
원문 보기 (영어)
I can clearly remember the first time I saw a deepfake. I was waiting for the train on my way to the Vice office in 2017 when I got a Twitter DM from someone who saw it posted to Reddit. It was a short .gif from the start of a porn video, showing a woman lying on a bed and talking to the camera, only the woman’s face had been edited to look like Gal Gadot. The deepfake was shockingly realistic for the time, and as Sam’s reporting on it from that day on has shown, deepfakes have primarily been used to harass and degrade women. We were surprised by how advanced the technology seemed at the time and knew that the internet and the law wasn’t prepared to deal with it. The technology has advanced exponentially since then, and the internet and the law has still not caught up to it. But as concerning as deepfakes clearly were the moment we laid eyes on them, the problem was never that we thought Gal Gadot actually made an adult video. Deepfakes can look convincing, but they never fooled us that what we were seeing was real. But for the past few months I’ve been looking at a new and strange type of AI generated nonconsensual image on X that I think will fool a lot of people. These images, which I’ve started calling subtlefakes, are not as clearly offensive, but they are a lot more convincing. The concerning implication is that it’s not just the technology that is advancing now, but that the people who wield it maliciously are getting more sophisticated. Much like “cheap fakes,” which in 2019 Data & Society described as an edited piece of media that manipulates the truth without requiring AI or any other advanced technology, subtlefakes are another form of synthetic media that has taken over the internet since the emergence of deepfakes in 2017. These more convincing images are not AI generated nudes of celebrities or face-swapped identities onto porn videos, but AI generated images of celebrities that are subtly edited to make them more revealing or provocative. We’ve written countless stories about AI generated nonconsensual media of celebrities and non celebrities. It’s a form of harassment that causes psychological and material damage to people, and they don’t have to be believable to do that. Even if the image looks photorealistic, it is often clearly fake for the simple reason that it’s unlikely that the biggest actor in the world would suddenly produce hardcore pornography that’s posted by a random account to X. The images I’ve been seeing lately are often only slightly altered. One type of image, for example, will edit a real photograph of a famous actor walking a red carpet, but edit it to make her dress more revealing or to make her butt or boobs bigger. Another type is entirely AI-generated post-workout selfies at the gym. The only example I feel totally comfortable sharing is one that was called out and shared by the target of one of these fakes, actor Xochitl Gomez. As a side-by-side she shared on Instagram shows , someone had taken a real photograph of her at a parking lot, looking over her shoulder and smiling at the camera, and used AI to make it look like she was bending over in one image and putting her hand on her butt in the other. Similarly, someone had taken an image of her on a red carpet and used AI to make it seem like she’s turning her back to the camera and sticking her tongue out. These images are not just photorealistic, they also don’t have the usual context clues we’ve come to expect from nonconsensual images. I think I am very good at spotting AI generated images because it’s a big part of what we do here, and I don’t think I could tell these were AI generated if I was just scrolling on X. These images are almost always posted by verified engagement farming accounts on X, which pays the people who operate them if their posts get enough impressions. X obviously doesn't care if people are doing this and a bunch of other stuff that’s as bad or worse. I’ve seen at least one account that shared a few of these subtle deepfakes, but when I scrolled down their post history I found nonconsensual fully nude and sexual images that they monetized via other platforms, so that’s another reason why X accounts might be doing this. There are many, many accounts like this and they are earning millions of views. I think it’s interesting because it’s a new, and as far as I know, yet to be covered use of nonconsensual AI images. It’s also important and in the public’s interest to know it’s happening because people should know what kind of AI content is targeting them, especially if it’s at the expense of other people. It’s also possibly a warning sign for how people’s AI use is evolving from abusive but obviously fake, to more subtle, less offensive, but much harder to detect. The other reason these types of images are going to be harder to moderate is that it’s easier to produce them. There are plenty of services that let people easily AI generate nonconsensual nudes of anyone, but they almost always involve some friction. People have to go to a sketchy site, install a sketchy app, or pay a small fee. As we’ve reported many times in the past, more easily accessible AI generators from tech giants can be manipulated to produce nonconsensual images as well, but generally speaking, most companies are trying to prevent people from making those images with their tools. While it’s not impossible, it does require some effort to bypass those guardrails. It is not so hard to bypass those guardrails if the nonconsual images people are producing don’t include any nudity or sexual acts, assuming the AI generator is even design to prevent those type of images. Most mainstream AI generation tools at least try to prevent people from generating any type of nudity. Fewer of them try to prevent people from generating the likeness of a real person. I can’t say for certain where most of the subtlefakes I’ve seen come from, but I know that at least some of them were made with X’s own Grok because some images included the Grok watermark. X has repeatedly been caught enabling nonconsensual images and only attempting to do something about it after public outrage. X did not respond to a request for comment. “Large social media platforms that are really struggling under the weight of this for the first time,” Hany Farid, an image forensics expert and cofounder of deepfake detection firm GetReal, told me. You can see my full interview with Farid here . “They've always had a problem, but this seems to be the first time where they're like, ‘Oh man, this is really bad.’ And I think it's because the volume and the sophistication of the fakes is nothing we've seen before.” The primary targets of deepfakes always were and continue to be women, but Farid said other nefarious uses of the technology are also becoming harder to deal with. “If you go back 10 years, it was pretty clumsy,” Farid said. “It's getting more and more sophisticated. They're evolving. Cyber criminals evolve and that's going to make everybody's job much much harder as the years go on.” Our reporting on deepfakes over the years has taught us that people make them because they feel entitled to women’s bodies, because they are intentionally trying to hurt them, or because it’s profitable. I’m sure all those reasons apply to subtlefakes, but it appears to me that the current state of X is pouring gasoline on that fire. People use subtlefakes as advertisements for accounts other platforms where they monetize nonconsensual images, but that is rare. It seems that the more logical scheme is to monetize pure engagement via X’s ads revenue sharing program, which pays X users for engagement. It’s not a lot of money, but it’s easy money. Since X is not interested in moderating this content, one person can also operate multiple accounts, posting dozens of times a day. In general, my X feed is mostly different types of engagement farming. Another type of subtlefake I’ve seen a lot of doesn’t require AI at all. For example, t