Family fun with deepfakes. Or how I got my wife onto the Tonight Show

35 Comments

[Also on Medium]

[ Update 3 Feb 2018: added two new creations at the bottom of this post. Last one turned out really well ]

I’ve first heard of deepfakes a good week ago. Thanks Twitter. Thanks Tim Soret.

Yes, it’s pretty damn cyberpunk. But from a superficial point of view, /r/deepfakes (extremely NSFW! You have been warned) consists of people using an app created by user “deepfakes” to create fake celebrity porn.

This has caused a shitstorm on the Internet, media discussing the legality of it all, websites taking down the deepfake creations, and people panicking as they realise AI is going to screw us all up (newsflash: it’s already been happening in much less obvious ways). And meanwhile, Nicolas Cage is taking over Hollywood.

While everyone’s debating whether this is good or bad, I just had to find out more. First thing that came into my mind? How can I apply this to everyone I know (in a non-porn way, in case you wondered).

How does it work?

The deepfakes app is a deep learning algorithm that learns how to reconstruct faces. Give it a bunch of pictures, let it run a few hours, and it spits out fuzzy copies of those images. Do note, it doesn’t create a copy. It learns what a face looks like, in different expressions, and is able to output that face solely based on that. There’s a detailed explanation on Reddit but let me try and dumb it down.

Think of it like this: imagine if you could stare at someone for 12 hours straight, observing all their expressions and absorbing that into your brain. Then that person asks you to sketch his face on paper, smiling, crying, any expression you’ve observed. What do you do? You immediately generate a photographic quality sketch on paper, from the mind! (using a pencil)

It’s insane!

While that’s pretty cool, it only gets better. See that “encoder” part? The FakeApp uses one encoder for all faces. It’s the decoder that’s kept face specific. And here comes the really cool part. Let it learn two faces, and things become more interesting.

Right, now see how this works. The encoder takes an image of a face, let’s it run through its “brain”, and the decoder spits it out again. In the example above, it can do so with faces of Anne Hathaway, and Elke, my wife. Ok, so far so good. But now let’s feed it a picture of Anne, but use the decoder that generates Elke’s face!

You just created a new photo of Elke. A photo that never existed, in the same angle, the same expression, as Anne! Amazing!

Family fun

Sure, putting celebrities’ faces on your favorite porn stars is an interesting use case. But we can leverage these celebrities for other things, such as inserting your friends and family into blockbuster movies and shows!

For the best result, you must first find an actor/actress that has a similar head shape as the person you wish to insert. In case of Elke (my wife) I accidentally noticed, while watching the Dark Knight Rises, that Anne Hathaway might be a good fit. I guess you know Anne, so here’s Elke:

All I needed was about 850 photos of Elke, a few 1000 of Anne, a lot of computing time, et voila: Elke’s on the Tonight Show starring Jimmy Fallon.

Bonus effect: now we know what Elke looks like with short hair :D

Here’s a little comparison gif:

I personally think it’s fun, can be innocent, and even makes for a nice surprise/gift. Remember, any tool can be used for evil. And as long as we’re not banning guns, this should not be a high priority, amirite?

There’s so much you can do with this technology. You know those dumb emails people send around where they replaced dancing elves heads with their own, or even worse, yours? Well, now you can put your best friend into his favourite movie: have her dance with Patrick Swayze and have the time of her life, or have an alien burst out of his stomach. It’s all within your reach now!

Beyond just pure fun, I can only imagine how people will start turning this tech into business ideas. Fashion will be huge (what would I look like with this kind of hair, this kind of dress…), fitness could be interesting (do I look good with muscles, will I really look better skinny), travel (this is you standing on a beach is going to be quite convincing). It’ll bring advertising to a whole new level. No need to imagine what if, they’ll tell you what your “better” life will look like! And it’ll be hard to get that picture out of your head…

Update: in the mean time, I’ve created two more. Elke’s a huge fan of Steve Carell, and I suddenly realized Anne Hathaway co-starts with him in Get Smart. First attempt was okay:

Then I wanted to try this one (original video):

And I think it turned out great:

 

Help us augment live streams of your game! For science!

Leave a comment

TL;DR: HCI research group at KU Leuven looking for game studios to provide access to live game logs for 2 Master theses, or even collaborate with our group around data visualisation for streaming research

Every year our HCI research group defines a number of Master thesis topics for Computer Science and Engineering students. Our topics include data visualisation, recommender systems, augmented reality, learning analytics, and e-health.

As my expertise is data visualisation, and video games have always held a special place in my heart, it only made sense to merge the two. That’s why this year we have two Master students working on  “Designing live data visualisations for the new spectator sport: video games”.

The general idea is to use live (interactive) visualisations during a game to help a specific audience get a better understanding of and new insights in what is going on. Some examples:

  • help people who are new to the game to get a better understanding of the game
  • assist the streamer in informing their audience better
  • help the streamer’s fans get more insights into their specific idol’s activities
  • facilitate a better overview to assist commentators

… anything goes, but we’ll be narrowing it down ASAP, as both theses must be finished by June 2018 ;)

For this topic my students require access to activity logs of a game, preferably during live play. The availability and type of data will play a big part in the choice of game (type) and target audience of the visualisation.

So what we need is a game studio (big or small) that is developing a game with Twitch/e-sports in mind, or a studio that has an established game with e-sports/twitch popularity. What we want from you is access to detailed game logs or an API, for science!

If you think you could help in any way, let us know! We can work independently, sharing our research results in a way that can help you, or we can even look for official collaboration options (we could even apply for funding, certainly if you have offices in Belgium), within and beyond the scope of these theses (this might actually be very interesting for small Flemish indie studios because of government funding possibilities).

Interested? Want to know more? Shoot me an email, or leave a comment below!

In case you wish to learn more about us, read on.

A  Master thesis at the Human-Computer Interaction group.
Our process is that of design-based research, iteratively designing prototypes with a heavy focus on the user, involving them in the design process and the evaluations. The result should be a fully working prototype that has been proven to be an added value to the target audience, by being usable, useful, and generating meaningful insights. The student also writes a 50-70 page thesis text, reporting on the current state of the art, the thesis process and results, and the research contribution. To pass the thesis (a requirement to finish the Master program) the student has to successfully defend the work in front of a jury of academics.

About our research group:
http://augment.cs.kuleuven.be

About our university:
https://www.kuleuven.be/english/

About me:
http://svencharleer.com/blog/about/

My PhD Acknowledgements <3

Leave a comment

Four point five years. Summarised in 131 pages and 18 scientific publications. That and a new title.

While the new title does sound cool (how else can one be taken seriously when announcing world domination), the biggest result is the little book: the dissertation, in my case, on Learning Dashboards. But while I hope the pages reach the intended audience (the Learning Analytics community), I wonder if anyone beyond that will read it.

Probably not. And that’s fine.

But more than just research and writing went into this achievement. A lot of people, friends, family, and colleagues, were part of the process. Pulling and pushing me along the path that led me here. So even if the main contribution doesn’t reach a large audience, I hope the acknowledgements in my book do. Without these people, I would not be where I am today, nor would this small advancement in Learning Analytics research.

So I could be ignorant and assume you’ll download and read the book ;) But I’ll just leave you with the first few pages, the important ones about.. the important ones.

Here we go…

Twenty years ago, I figured, let’s do Physics. I loved all things space and dinosaur related, and had an awesome Physics teacher. However, that same teacher told me studying Physics would most likely land me in Finance. So, I followed my other passion, Computer Science.

 

But after completing my degree and spending years in the private sector, I wondered if I had made the right decision. While I had a passion for programming, the lack of creativity that comes with a software engineering job (beyond the code) was killing me. Moving to Nottingham to pursue a career in games did not improve the situation either: a developer just develops, it seems. Side projects (indie game development and art academy) were an attempt to bring some creativity back into my life, but I needed a serious change where I spent most of my time: the day job.

 

The PhD was an unexpected opportunity that presented itself in my mailbox. Two weeks it sat there until I finally decided to reply. What followed was life changing. Research meant I could explore the unknown, build things no one had before, and join the user in their experience with our new creations. I got paid to create visualisations, play with new technology, and spend time thinking of all the crazy things we could accomplish with it. Both the nerd and the artist in me were satisfied. This might just be where I belong. It only took me 36 years to figure it out…

 

Thirty-six years is a long time. I owe where I am today to a lot of amazing people: for the opportunities, the support, the patience, and the listening.

 

I would like to thank Erik. In 2013, I somehow convinced him I would be the right guy for the job. When times got tough, he would keep convincing me I was. “I wasn’t that smart either and look where I am now”, a pep talk I will never forget. From all the “bosses” I’ve had, he was one of the few who genuinely cared about his people, at a personal, family, and career level. Thanks for letting me get to know you and your amazing family. Your awesome ideas will live on in our work, we will all make sure of that.

 

If it wasn’t for Bert, I would have never even considered a PhD. But it was his better half, Katrien, who got me in the room with Erik. I owe a lot to her. She stepped in just as I was close to jumping ship. She was the motivating force I needed, and pushed me across the finish line. Thanks to both Katrien and Tinne, this last year and a half of the PhD has been amazing. We’ve published great papers, made a name in the community, and put our stamp on student advice at the university.

 

Joris. What was it Jose said, thanks for the coffee? I’ll do one better: bourbon at Harvard, such an amazing trip! He never once doubted me, and I will never forget his endless “het komt goed” (it will be all right). And who’d have thought, it did! (I guess this calls for another round of drinks in Boston!).

 

I would also like to thank Andrew, Bieke, Yolande, and Martin, for taking the time to read my dissertation, providing valuable feedback, and a memorable private defence.

 

But my biggest thanks goes to Elke. I could not have achieved this without the love of my life (15 years this year, 10 as my gorgeous wife). Always supporting my crazy decisions. She quit a promising career to follow me abroad and let me pursue my game developer dream back in 2008. And in 2013 she supported me again in my biggest career change, when I gave up a well-paid, secure job to become a student once again. We were not expecting the PhD to be such a roller-coaster. I experienced moments of joy and despair, feelings that would affect her as much as it did me. But she always had my back, endured the after-work rants, and supported me in every way possible. Without her, I would never have managed.

 

Hazel joined us (in the womb) at the start of the PhD. Kids do not make things easier. But they do give you a reason to keep going. During dark periods of the PhD, she was always there to put a smile on my face (or add to the misery with sleepless nights. She’s a little monster like that). Hazel, if you read this when you are older, we love you and we will make sure you get to follow your dreams just as we could.

 

My parents, Marinette and Guy: they have always been there for me, supported me, and believed in me (and also provided me with all the nerdy hardware a kid needs to keep his technology interest going). And my grandparents, Meke, Vake, Peter Wieke, and Bobonne. Meke is not here to see this, but if someone believed I could pull this off, it was her.

 

My parents-in-law, Monique and Daniel, and the “Moekes”, for treating me like one of their own. Monique, the things you have missed out on, it is not fair. We miss you, words cannot describe.

 

Kurt, thanks for showing up at the defence (if you didn’t, this is going to be awkward). You’ve been that one true best friend. Always there in time of needs. And always making me look good at the board game table.

 

Franky, the bastard who pulled me out of the Flemish, secure mindset, and lured me to England. I ended up working long unpaid hours in Nottingham and lived amongst criminals and drunks. But I regret nothing!

 

José, for his unique perspectives on things, telling it like it is, and your attempt at keeping me sane through the PhD (it did not work).

 

Thomas for the babysitting and being Hazel’s coolest uncle. And “Tantan”, for taking care of Hazel all those Mondays, and for all the things you have done for Elke.

 

Kris, for bringing that new addiction into my life. The Nets won’t run themselves! Lies, for being Hazel’s awesome godmother. Jim, thanks for letting me win sometimes. Wait, no.

 

Sean, Jenna, Greg, and Johnny. One day Rad Lab Games will rise again!

 

François and Denis, we will make that dinner happen and bathe in Brasschaat’s sushi! Until then we will just shoot people online.

 

NorthgateArinso: Fred, Schtroumpf, Maarten, Karo, and Tom. My first and fourth job (thanks Samir), and also my last job before I ran off to academia (I am not implying anything!).

 

Everyone at Monumental Games and iChoosr, even though the stops were short, they were life changing.

 

My current, former, and visiting colleagues at the coolest lab of the Computer Science department: Yucheng, Karsten, Francisco, Gonzalo, Robin, Tom, Sam, Bruno, Victor, Gayane, Chen, Sten, Till, Frans, Samara, and Oana. And all the amazing people of the weSPOT, eCloud, and ABLE projects.

 

The Blade Runner soundtrack, for getting me through numerous paper deadlines.

 

And Bert. For getting me into this mess in the first place.

 

Dude, sucking at something is the first step towards being sorta good at something.

– Jake the Dog, Adventure Time

I love you, Pumpkin.

– Honey Bunny, Pulp Fiction

PhD in CS. Release Date: July!

Leave a comment

It has been a tough 4.5 years, but the end is finally near! Last Tuesday I had my private defence, or “preliminary defence” (or viva for the Brits, I hear). In Belgium, this part is pretty much the scariest moment of your doctoral career: it’s the proper, final, oral exam. Sometimes it is an enjoyable chat where they (4 professors) treat you as peers for the first time, sometimes it’s a soul-crushing, head-ache inducing, exhausting emotional rollercoaster. Mine jumped around between these two extremes, with some of this:

Still, the experience was a good one. They do a good effort in making you doubt your work, and I walked away feeling more confident and proud of what we have accomplished these long years.

The public defence, where I get to bore the audience with 45 minutes of presentation, is planned on 6 July 2017. If you are in any way interested in Learning Dashboards, Learning Analytics, or just like free drinks and snacks, I suggest you come watch! If interested, do shoot me a message for the details (it’ll be in Leuven, Belgium)! Official invites will be sent in a few weeks…

Meanwhile, enjoy the slides of the private defence:

Deux Wallons et un Flamand play Overwatch

Leave a comment

To those of you who don’t like to read, a summary:
Three Belgians (two Walloons and one Flemish, and sometimes a Mexican) will stream Overwatch every Wednesday, from 9pm CET until 11pm CET. So head over to Twitch and subscribe!
(yes this is an activity that’s helping me finish my PhD dissertation!)

Belgium is an interesting country. Super tiny, only 11 million people, yet home of chocolate and beer, fries, Jean-Claude Van Damme, Tintin, the Smurfs, the saxophone, the contraceptive pill (the Catholic church loves us for that one), we invented both asphalt AND roller skates, and so much more. We do all right for an insignificant speck on this planet.

But what makes Belgians really different, is their attitude towards political correctness. We tend not to do that. Except for politicians and hipsters, maybe. But in general, we “respectfully” make fun of everything. Nothing is sacred.

And that’s fine. We don’t mind being the target of laughter and ridicule either (we’ve been through enough misery that we can see the funny side of everything) . Hell, we do it to ourselves on a daily basis!

You see, Belgium is split into several parts, but the most important ones (sorry Brussels, no one likes you) are Flanders and Wallonia. Flanders (the North, where I live) speaks Dutch, known by the South for extreme racism and intolerance. But at least we cycle and feel better about ourselves because of it. Wallonia, the South, speaks French, known by the Flemish as leeches who contribute nothing to society. What they lack in civilisation, they make up for in hospitality and cleaner air (yet they don’t cycle!). Before I get lynched by either side, these are stereotypical jokes we throw at each other. Jokes, people!

Yet, media and politicians seem to take these seriously, fuel the hate (which.. isn’t really a thing), and want to split this country in two.

But in reality, calling a Walloon lazy, or a Flemish a racist, is just a thing we do. “No offence”. Both sides know that. If they can’t take the joke, it usually means they’re Bruxellois (or lazy, or racist..).

So what does this have to do with anything? Overwatch, believe it or not. Together with a couple of Walloon friends (and the occasional Mexican), we’ll do a weekly stream to prove everyone wrong. That we DO get along. As long as we get to shoot other people in the face. Isn’t it beautiful how violence brings us together?

We’ll show them those stereotypes are as true as the lies about French driving skills, Dutch cheapness, British food, American obesity…

*cough*

We’ll show the true love that lives between the North and the South! No ham will come between us (ok that’s a terrible inside joke).

The stream will be in English, of course, and once we hit critical mass and make money, we’ll hire someone to subtitle the Walloons ;) ..

Jokes aside (I have to save them up for the stream), we’re planning on a weekly stream, every Wednesday, from 9pm CET until 11pm CET. So head over to Twitch and subscribe! You might not be disappointed!