Judith van Stegeren

8 tips for self-employed designers

2013-08-30T00:00:00+00:00

Getting started as a green freelancer can be a daunting task. There is so much to do and even more to learn. Where do you start?

In this posts you can find some tips on freelancing or designing that I picked up over the years. They helped me and I’d like to share them with you!

Think of your dream day as an entrepreneur or freelancer

What would you do if you had all the time of the world? Which projects would you take on? Who is your dream customer? Write down the answers to these questions and re-read them after some time. This really helps with getting a clearer picture of what kind of service you want to provide and to whom. Secondly, this record makes it very easy to track your development as a freelancer or entrepreneur.

Spend the time that you are planning to work on actual work

Don’t waste your precious time on a facebook-twitter-9GAG-galore. Looking at Pinterest or Tumblr can be a good source of inspiration. However, remember that if you do this for three hours it can make a serious dent in your time… and self-esteem. If you can’t snap out of it, you’ll have a very busy but unproductive day. (And that’s a shame, isn’t it? After all, our work’s our hobby!)

Get up early

Get dressed as if you’re headed for the office or an important client. Have a relaxed cup of coffee. Take a walk. In short: take some time to boot your creativity. Be up and running for your day around 8 and get to it (fo’ real) around 9. This way, you can do the largest part of any serious work before noon.

Organize your projects

Make a list of all running projects. Save design briefs (yes, even for bigger personal projects) so that even after a dusty year on your to-do-shelf you’re always ready to pick up the project where you left it off. Be sure to store any important sketches or notes in the near vicinity of the design brief! Documenting your process is a healthy habit. Consider keeping a design journal, where you jot down notes on the status of the project, problems, solutions and thoughts on the process. This can be a handy reference, for yourself (and if you choose to publish it, for others).

Bonus tip: read about Photoshop Etiquette. Keep your PSDs as organized as your projects and make your printer, clients ánd future-you very happy.

Gather inspiration

Every day, spend a short amount of time (for example: one hour) gathering inspiration or new input. You can do this in multiple ways. You can read magazines or books on subjects you like (you don’t have to ready specifically work-related stuff!) You can follow some blogs and catch up on their RSS-feeds. Check out random tumblr-accounts, you can hang around on Pinterest, post in the DeviantArt forums, post some random friendly twitter-messages. Interact with people and communities on the internet that interest you. Try out a recipe of a dish that you’ve never tried before. Get your input from things that inspire you personally.

Experiment

Be aware of the fact that there is always stuff to learn. Don’t be afraid to experiment. Do stuff you’ve never tried or that you find a bit scary. Reserve a tiny amount of your budget for your own art-lab and invest in art supplies that you’ve never used, a course in a graphic technique they didn’t teach you in school or plan an excursion to an art gallery or museum near you.

Don’t be afraid to emulate

Sometimes to best way to learn something new is to emulate others. Famous painters like Van Gogh started out the same way — Vincent van Gogh copied paintings of painters he admired, such as Millet and Hiroshige. Read the book Steal like an artist by Austin Kleon.

Don't discard your ideas

Jot down all crazy ideas you might have. Be sure to carry around a small notebook in which you can scribble and doodle away. You don’t have to show anybody, so what do you have to lose? Everyone knows that the best ideas always come in the middle of the night, while riding a bicycle or under the shower or in the bathroom. Turn off your inner critic when brainstorming. It doesn’t contribute to the creative process anyway.

And above all: enjoy what you’re doing! Being able to choose your own living is the most beautiful thing there is!

Bonus link: why working at home is both awesome and horrible, a comic by The Oatmeal.

How to get started with chainmail

2013-09-10T00:00:00+00:00

When I was visiting Castlefest this year, I had the great opportunity to visit Kunst Op Straat. This artful collective gave some really interesting workshops and lectures about a variety of creative things, ranging from painting and song-writing to historical clothing. I signed up for a workshop on chainmail.

Since I was not entirely new to chainmail the guy conducting the workshop was kind enough to show me some advanced stuff — by which I mean something different than the planned ‘make your own keychain’.

The workshop was lots of fun, so I went home armed with my own private supply of pliers and aluminium rings. After making a nice bracelet with a very pretty looking byzantine weave, I decided to share some tips and tricks on how to get started with chainmail.

Step one: Gather some inspiration

Before you can start making your own chainmail jewelry or armor, you need to have a attainable goal. I recommend fooling around with a few weaves before you start building a complete hauberk out of nothing. This part is fun and easy: just browse around on Etsy, Flickr, DeviantArt or the M.A.I.L. website and gather some inspiration for what you want to make.

Step two: Obtain the right materials

Rings

If you have a general idea of the item you want to make, think about the right materials for the job. Aluminium is light and easy to handle, but not very strong. Galvanized rings come in lots of different colours but might scratch easily.

Steel is really tough, but you need more strength in your hands to work with it. Silver is neat for jewelry, but not very cheap to start with. And rubber rings can give your wearables a stretch effect — just don’t forget you can only use them as ‘closed rings’ in a weave! Look to other mailers to see what type of ring they use for different items.

The size of the rings, the thickness of the wire and the aspect ratio between the size and thickness of the ring all determine the kind of weave you can use. To give an example: European 4-1 works with almost all rings, but for the Helm chain, you need two different sizes:

You can obtain most standard sizes rings in all kinds of materials from specialized webshops such as TheRinglord.com (USA) or The Queen Ring (NL).

Pliers

You need two pliers to open and close the rings while weaving. Some people have different kinds depending on the type of ring they’re working with: pliers with tooths for lots of grips for the tougher rings and smooth smaller pliers for jewelry making. Mind that the bigger or broader the pliers, the harder it is to see what you’re doing but the better the grip. Toothed pliers might scratch galvanized rings.

Other materials

You can combine chainmail with scale mail to make really cool and shiny armor. Upgrade jewelry with wirework, beads and precious stones. The sky’s the limit. Don’t forget to buy closures: leather straps for armor and silver or gold coloured clasps for bracelets and necklaces.

Step three: Find a tutorial and learn to weave

You have a good idea of the thing you want to make. You have heaps and heaps of rings in shiny colours. You are dual wielding your mighty pliers with unparalleled prowess. Now what?

You have to learn how to weave. Lucky for you, there are numerous good sources on the internet. A few of my favourites:

CGMaille.com
The extensive weave library of Mailleartisans.org
Derakon’s library of instructions (theringlord.com)

Is this all a bit too much for you? Start with this tutorial on European 4-1. It will teach you how to make a sheet of chainmail for armor and simple bracelets.

I’m a master of European 4-1. What’s next?

If you can weave European 4-1 in your sleep, try your hand on one of the following things:

Try more difficult weaves.
Try working with scale mail.
Make a chainmail ‘inlay’ from different coloured rings.
Take on a big project. Make a big piece of armor, such as a hauberk. Hauberks generally consist of 7000+ rings.
Make your own custom rings.
Combine chainmail with intricate beading-schemes.
Try making something out of micro-maille.

If you have made something after reading this tutorial, be sure to share it in the comments. I’d love to see your creations. Happy weaving!

Summer read for motivation: Brick by Brick by Stephen McCranie

2014-07-14T00:00:00+00:00

I have a hard time finding the motivation to finish my huge list of projects during the summer break.

I survived my first year of my masters studies and the summer holidays are finally here. After the exams in June, I desperately needed some downtime so I took a week off. But now, after a week of no deadlines, no obligations, only parties and days of leisurely strolling around the city, I find that it is very hard start working again. It’s not a lack of discipline, mind you — although I think the lack of deadlines certainly doesn’t help.

But something happened to me today when I found the website Doodle-alley by Stephen McCranie. The website contains an online book, “Brick by Brick”, that contains motivational essays and comics about being a creator, based on McCranie’s experience as a cartoon artist. I can definitely relate to the feelings and pitfalls described in the book and the comics are both well-drawn and inspirational. Read the the first chapter of Brick by Brick.

I can recommend this book as a inspirational summer read that will also (hopefully) provide some motivation. I hope you will be as inspired and motivated by Brick-by-Brick as me. Enjoy!

Stop motion short: the history of typography

2014-07-22T00:00:00+00:00

I’m working on a Skillshare project at the moment (more about that here) and I came across this handy stop motion video that explains the history of typography in a mere 5 minutes! I think it’s really handy for when you want to search for a specific font but you don’t know the exact category — knowing the different between humanist sans and geometric sans is really useful when browing sites such as myfont and fontstore.com! This video makes it really easy to find out what kind of font you’re looking for.

If you like this kind of stuff and you want to learn more about typography, be sure to check out the book Thinking with Type by Ellen Lupton. It’s quite a good primer on everything font-related.

Handlettering course on Skillshare

2014-07-30T00:00:00+00:00

In January 2014 I bought access to an online course on Skillshare. I started with the course on hand-lettering in January but couldn’t continue because of exams. Then when the summer holidays began I could start right where I left off.

Practice for the course on hand-lettering

For those of you who don’t know about Skillshare: it’s an online learning community, a bit like Coursera, where you can take online classes on different subjects. Where Coursera specializes in academic courses from universities, Skillshare mostly caters to creatives and entrepreneurs. Some examples of courses you can find on Skillshare are calligraphy, logo design, entrepreneurship, creative thinking and writing film scripts.

This is what Skillshare looks like if you’re taking a course.

You can either pay a monthly fee for full access, or you can buy access to specific courses, which is what I did since I don’t always have time. Once you have paid for a specific course, you can always access the course materials.

The course I bought is called “Lettering 1: The First Steps of Hand-Lettering: Concept to Sketch”. The teacher, Mary Kate McDevitt, gives short video lectures and exercises and shows us tips and tricks on handlettering. It’s really cool and for 25 dollar for the whole course it was an absolute bargain.

I’ll keep you guys up to date about my progress. I’m having loads of fun drawing all kinds of funky letters and practicing fancy handwriting. For those of you who can’t wait for blogposts: you can take a look at my project page on Skillshare.

Introduction to cartoons and comics

2015-01-12T00:00:00+00:00

In the wake of the tragedy in Paris of last week, there has been a lot of attention for comics and cartoons, especially as a medium to voice an opinion or tell a personal story.
Sarah McIntyre has written this nice introduction to drawing cartoons and comics. It includes answers to frequently asked questions, some cool project ideas and lots of examples from numerous web comics.

Biographical comics

For those that only know comics as a source of entertainment for children, be sure to look into serious graphic novels like Maus and Persepolis. Comics are an excellent medium to tell a story in a personal manner. Telling a story in panels and quirky illustration can also serve as an easy approach to heavy or difficult subjects.

Maus is a biographical graphic novel by Art Spiegelman. It tells the story of the father of the artist, a Polish Jew, who lived in Poland during the Second World War. It deals with the war itself but also the long-term personal consequences for those that were in it. I believe it is a required read for first-year history students at the University of Amsterdam.

One of my personal favorites is the book Achter de Kawat (Behind the Fence) by Charles Burki. Burki was interned on Java in a Japanese prisoner-of-war camp during the Second World War. During his stay he kept a diary with illustrations and humorous notes, which he published when he returned to the Netherlands after the war. The illustrations depict the difficulties of daily life in the war camp, and the book is interesting from both historical and artistic perspective.

Preparing an exposition in the library of Nijmegen

2015-04-04T00:00:00+00:00

This blog post describes my preparation for my first exposition, in a local library in Nijmegen. You can read about the exposition itself here.

Part one: April 4

I’m so excited! My first exposition is coming up. If you are in the neighbourhood, you can visit an exposition of my children’s book illustrations in the local library, Bibliotheek Hatert, from next week onwards. Last week I’ve been busy collecting all kinds of props (mostly stuffed animals) and getting my artwork ready for the printer.

Here’s a sneak preview:

Part two: April 9

Yesterday I went to the city centre and bought a huge folder for transporting the printed illustrations for the exposition. The folder is large enough for A2 poster prints (75 cm x 53 cm) and it even has a handle so I can take it with me on my bike — although I must say that cycling with the folder gave me the impression I was kite-surfing through the streets of Nijmegen.
Next, I visited the printer and had all the smaller prints (A3 and A4) printed on beautiful glossy card stock! The two larger A2 posters I already printed on the A0 poster printer at the university.

The exposition in library Hatert is here!

2015-04-15T00:00:00+00:00

And finally, my first exposition is a fact. I had lots of fun last friday filling a huge glass cabinet with my children’s book illustrations and lots of cute props!

Here are some photos of the result:

The library asked me to fill a huge 2 by 2 glass cabinet near the entrance.

I even reprinted some of my older work — I used to draw childrens book illustrations when I was still in high school. The little black sheep is a recurring character. Recently, I discovered the animations of Shawn the Sheep and I realised how much they look alike!

“De Raaf en de Vos” (The Fox and the Raven) is still one of my favorites. I think it really looks good as this glossy A2 poster! I translated the poem (the original is in french) to Dutch prose, so that children who don’t know the story can read it in the library.

The penguins! I love the penguins. I have friend who dreams of becoming a professor. He loves penguins, so I made him this poster years ago. Now I could borrow his collection of stuffed penguins for the exposition. Another friend loaned me his Lego model of the Mars Rover Curiosity (which is very, very cool. I cannot stress this enough. Lego & science = awesome!)

My sister picked up the stuffed hedgehog for me so I could recreate the “for science!” t-shirt design with stuffed animals and a real Erlenmeyer flask. Thanks, sis!

If you want to take a look at the exposition in real life, you can find it here:
Bibliotheek Hatert
Couwenbergstraat 20
6535RZ Nijmegen (The Netherlands)

The library is open on the following days:
monday: 13:00 – 17:00
wednesday: 11:00 – 17:00
thursday: 13:00 – 17:00
friday: 13:00 – 17:00

How to learn to use Adobe Illustrator

2015-04-20T00:00:00+00:00

Today, I want to give you some tips and tricks for when you want to learn how to use Adobe Illustrator. I’ve been using Illustrator for two years now. In the beginning, I really struggled with the pen tool and the fact that the program is so different from Photoshop! I hope I can help some of you by sharing how I started out using Illustrator.

Adobe Illustrator is not Adobe Photoshop.

Forget everything you think you know about Illustrator since you know Photoshop. You’ll thank me later.

Gather some resources

Surf the internet. You can find many tutorials online that deal with illustrator, the pen tool and all other stuff you need to know to craft basic vector images. Some tutorials cover one tool, other are step-by-step explanations of the process of a particular image. There’s even a game for learning the pen tool!

Some resources I used to learn Illustrator:
Illustrator shapes tutorial by milkbun
Tutorial part 1: The basics of illustrator by modernreligion
Tutorial part 2: The basics of vectoring by modernreligion
Illustrator tutorial: gear wheel by surgio
The bézier game

Get your hands dirty.

You learn fastest by doing. Reading tutorials is fine, but step-by-step going through all the actions in a tutorial is better. As an example, here’s the first thing I created in Adobe Illustrator by following this tutorial by Milkbun.

I know, I know. It’s cute and ugly, but it showed me how stuff worked: how to use the pen tool, how to pick colors and how to combine shapes. When you know the basics, you can try to make more original stuff. Below is the second image I made in illustrator. I made a concept in Photoshop and then did a simple vector image in illustrator. I wanted to make something that looked like the warning signs in Portal:

Stop thinking in pixels and start thinking in shapes.

Digital photo’s, paintings and drawings are made up of colored pixels on a canvas. When you work in Photoshop, you work with pixels. However, vector images (Illustrator) are build of shapes, which means that if you want to master Illustrator, you have to think in shapes instead of pixels. Shapes can overlap each other, you can substract a shape from another shape, or combine several shapes to get a larger shape. Here’s an example:

If I want to draw a cloud, I could do it by painstakingly drawing the outline of the cloud:

But I could also realise that a cloud shape is made up of a number of circles:

In Illustrator, there is a tool called ‘pathfinder’, with which you can add and substract shapes. If I click the ‘unite’ button, all the shapes I selected will be combined into one shape. Note that the shapes have to overlap, otherwise nothing happens:
Before:

After:

Learn. Practice. Repeat.

Try out more complex stuff as you go along. Play with the tools, the settings, the filters. Have fun. Make a poster, a logo or a t-shirt design. It doesn’t have to be perfect: it only has to be better than the last thing you made.

Bonus: the Bezier Game

When I started to learn Adobe Illustrator, it was a big challenge to learn how to wield the pen tool. The Bezier Game is a really cool web game that helps you to learn how all the weird hotkeys and such work in illustrator.

Good luck!

How to get started with poster design

2015-07-23T00:00:00+00:00

I would like to share with you what I learned in the past 6 years as a print designer for Karpe Noktem. If you want to get started with designing posters, you need to adhere to some basic rules.

I presume that you have some notion of how Photoshop works. If you use different software, that’s no problem: Paint Shop Pro or GIMP will do as well, as long as you know some basics such as choosing the correct canvas sizes, the behaviour of layers, etc. What I want to emphasize is that you don’t have to be a photo manipulation wizard or artist prodigy to make a good poster. If you don’t know how how to do something, be sure to google it. You can find plenty tutorials online.

1. Start out with a proper poster template

This is the absolute minimum required for a good print design.

When you start a new file in Photoshop, select the proper canvas size (A4, A3, …) and set the document resolution to at least 300 dpi. This ensures the quality of the poster when you try to print it.

Use guides to mark the middle of the document — this makes aligning things easier. Also consider placing extra guides at least 1 cm from the borders of the document. This’ll keep you from putting important elements, such as text, too close to the document borders. Proper margins give your poster a more sophisticated look.

If you have to use some kind of branding template or adhere to a style guide, put all compulsory elements in place first. Think of colours, logo’s, etc. Then you have the ‘bare bones’ of the poster, and you can continue from there.

2. Think of the goal and audience of the poster

Your poster probably has to communicate some kind of message. For a band night, that would be information like the date of the venue, ticket price, the artists and the location. Keep this message in mind at all times. It will help you design a poster that both resonates with your intended audience and that communicates the message clearly.

3. Communicate your message clearly

Make sure your audience doesn’t have to search for the information they’re looking for and pay attention to readability. This means: don’t use ornamental fonts that no one can read and don’t use clashing colours for the background and text. Nobody likes reading red text on a green background.

4. Attract attention

The content of your poster should be a combination of clear text and eyecatching visuals. Choose graphics with care. Choose images, colours and fonts that speak to the particular audience of your poster. For example, look at this poster for ‘metal band night’:

Finally, use fonts wisely. Don’t try to put your entire font collection in one poster. Just stick with a maximum of 3 fonts. If you think you don’t have an appropriate font, check out the list with resources below. And please, PLEASE, don’t use comic sans — unless you need a poster for a children’s party.

4. Resources

Dafont.com: free fonts
Deviantart.com: for stock images and lots of inspiration
StockExchange: for (non-free) stock images
Colourlovers.com: colour palets

5. Tutorials and examples

If you have a question or even more tips for good poster design, let me know in the comments!

Poster design step-by-step

2015-07-23T00:00:00+00:00

Hey guys! As a continuation of my tutorial on poster design, I thought I’d write a blogpost with an example. Here’s the step-by-step design process of the poster “Pluk de nacht”.

Set up the document. This poster will be printed on A3 paper. To make sure final result looks nice and crisp, I’ve set the document resolution to 300 dpi.

Mark the middle and the margins with guides.

Insert all compulsory elements, in this case the name of the event and the logo. Add extra guides where neccessary.

Add all the text that should be in the final version of the poster. Now you know which amount of whitespace is left for decorative elements.

Decide how you want to fill in the rest of the poster. I decided to included some photographs of earlier versions of this event. To spice things up, I included polaroid frames for the photo’s. After fiddling around for a bit I came up with the following composition:

Finish things up. Tweak the poster until you are satisfied. The devil is in the details!

I hope this example has helped you with understanding poster design. Questions? Let me know in the comments!

The Value of Writing

2016-09-06T00:00:00+00:00

Hello world.

When I was still a student, I was mentally preparing myself for becoming a PhD student. I knew from the experiences of some acquaintences that the life of a PhD student can be hard and stressful. A significant amount of PhD students never finishes their project. There's the publish-or-perish-pressure to cope with, and then there's teaching duties, research topics that lose their novelty over time and sometimes non-compatible supervisors. (If this is all new to you, take a look at the webcomic PhD comics.)

In short, I decided that if I wanted to be a successful PhD student, I needed to become a prolific writer, unafraid of deadlines and enormous to-do-lists. I started reading everything I could find about doing research, productivity, life hacking and mindfulness. The blog of Matthew Might, a professor in computer science, and Leo Babauta's world-famous Zen Habits blog were my main sources of inspiration.

Both Matthew and Leo cite the benefits of writing often. I agree with them.

Writing forces you to order your thoughts in a structured manner;
Writing forces you to think about your audience and the message you want to get across, (hopefully) making you a better communicator;
Writing can help you find the flaws in your logic;
Writing in English can help you share knowledge with a broad audience;
Writing is a way of storing knowledge, knowledge that can be used to advance your field.

I work as a researcher and analyst in the area of digital security, a field that is highly dependent on information sharing between experts. Many security researchers share their findings online, for example via articles, tweets or open-source tooling. By publishing blog posts, I want to do the same.

I will share posts that cover various topics. I've had numerous blogs in the past, but I always tried to focus on one subject. Being a person with many interests and projects, I always fail to keep the blogs alive in the end. Thus, I have decided not to limit myself to one topic or field with this blog. As time progresses, I will translate some of my earlier articles from Dutch to English, and post them here.

How to write a master's thesis

2016-09-14T00:00:00+00:00

When I was writing my master's thesis, I discovered that to finish the thesis I needed more than divine inspiration and mathematical skill. I needed discipline and a good dose of tenacity. I started collecting productivity hacks and tips and tricks on writing research papers. This article summarizes the best tips I found.

Optimise your productivity
Use a back-up system
Keep a research log
Ask for help when you're stuck
Organise your sources
Separate the phases of doing research, writing and editing
Just start
Write well

1. Optimise your productivity

Amber Davis is a researcher and coach of PhD-students. She wrote three guest posts for the blog PhD Talk. In her articles, she focuses on taking good care of yourself when you're doing research by making sure you get enough sleep, exercise and time off. She also stresses the importance of staying positive, even if you're stuck with your research. Given that a lot of graduate students live on nightly writing streaks, self-help books and ramen noodles, reading these articles can't hurt.

Don't expect to be able to do difficult work for 6 to 12 hours per day. Experience learns that we can concentrate intensely on a difficult task for about 4 hours a day before our mental energy is depleted.

Make your work environment fit for concentrated work. Surround yourself with tools and people that promote productivity.

I also like the productivity posts by associate professor Matt Might.

If you are interested in forming new habits of productivity, Leo Babauta has written a lot on this topic on Zen Habits.

If you want to know how and why to train your concentration, take a look at the non-fiction book Deep Work by Cal Newport.

2. Use a back-up system

Stolen laptops, power surges, broken hard drives and fires are part of reality. When you're doing research, take precautions so that you don't lose the products of your blood, sweat and tears. Back-up regularly and often. Use USBs, cloud storage, or an old laptop.

Even better than a back-up system is a version control system. This allows you to experiment with your work, and set it back to an earlier version if your experimenting doesn't work out. Even better: you can try different paths of coding/research in different branches. Although cloud storage frameworks like dropbox sometimes have versioning capabilities, I prefer git. If git scares you, try learning it with the resources on Github.

3. Keep a research log

Keeping a research diary has many advantages. It serves as a log of everything you encounter, read, do and ponder upon during your research project.

If for whatever reason you cannot work on your project for a while, your research log will tell you exactly where you left off.

Writing things down when you're stuck will force you to formulate your problem clearly, which is often the first step on the road to finding the answer.

I find that a normal office document does the trick, but some people prefer to use fancy pens and moleskines to motivate them to write things down neatly.

I take my inspiration from Edsger Dijkstra, a famous Dutch computer scientist, who wrote an enormous amount of technical notes and essays ("EWDs") on a wide variety of topics. You can browse the archive of all EWDs here.

4. Ask for help when you're stuck

When I was an undergraduate student, the thought of asking other people for help filled me with fear. I had to learn the hard way that asking for people does not make you a failure. Besides, most people are happy to help -- just formulate your questions clearly and ask them politely if they have the time.

Asking for help has two major advantages.
Whenever you're stuck, you might get unstuck a little sooner, which makes you more efficient. Secondly, talking about your problems forces you to clearly formulate your thoughts -- which might lead to new insights as well.

5. Organise your sources

When you read large amounts of texts for your research, consider organising these sources in a meaningful way. Doing this consistently during the research phase can save you a lot of time during the writing phase.

I like to maintain a list of all the texts I have read. In this list I keep track of the following properties:

Where have I found this text?
What are the main ideas of the texts? (3-5 lines max)
How is this text relevant for my research?

If a certain text is particularly important for my research, I will also annotate all paragraphs with a short description ('proof sketch', 'description used hardware', or 'research validation'. Note that this only works for well-written papers, where every paragraph has at most one goal. An alternative is highlighting the most important sentences with brightly coloured markers.

These annotations save me a lot of time when I am citing my sources during the writing phase, since I can find important pieces of information back in a matter of seconds.

6. Separate the phases of doing research, writing and editing

My favourite workflow for doing research consists of separate phases of doing actual research, writing and editing. The first part, doing actual research, makes up the largest part of the work. It encompasses 'getting your hands dirty' in an academic way: reading through enormous amounts of research papers, installing tools, testing tools, using tools, writing code, experimenting with mathematical proofs, etc. The main problem is that most people put off writing until they are done with doing research. I think this is a mistake.

In my experience, it is better to do a few hours (1-4) of research, and afterwards process everything you have learned by writing it down. Note that this doesn't mean that you have to use this written material literally in your final research paper. However, it can be wise to record a few important citations (if you've been reading a paper), to write an evaluation of the proof strategies you've chosen so far (when proofwriting), or to describe why you've made a certain choice in your implementation (when coding).

During the writing phase, I try not to judge too harshly how I phrase things. In my experience, this gets in the way of writing efficiently and effectively. For writing you need a creative mindset, while for editing you need a judging mindset. Because the two phases require two different mindsets, I postpone judging the quality of the text to the editing phase.

7. Just do it

Seriously, just start writing.

If you have trouble with starting (and continuing) to write, try the web-app Write Or Die. I have used it in the past to write 50,000 words in 30 days.

Another nice strategy for writing is to structure your research paper like a tree object: the paper itself is the root node and all sections and subsections are child nodes. The leaf nodes of the tree should become the paragraphs of your paper. Each node in the tree should have a main message or a communication goal. If you have written down the argumentation structure of your research paper as a tree object, the only thing left to do is to transform each node into a neat paragraph.

8. Write well

Maybe I will someday write another blogpost on writing well. Until that time, enjoy this hilarious article by Steven Pinker:

Why Academics Stink at Writing

Tuvok picks high-rated episodes of tv-series

2016-09-15T00:00:00+00:00

Last week, when I was visiting the SANS SEC560 course in Brussels, I wrote a small tool in Python 3 called tuvok. This tool takes a tv-series and a minimum rating and then outputs a list of all tv-episodes of that series that have a rating equal to or higher than the minimum rating.

Why

I wrote this tool since I wanted to start watching the classic Star Trek series, but I don't have time to watch all episodes. Instead I decided to watch only The Very Best Of Star Trek: The Original Series. I needed a list of the episodes with an IMDB-rating of 7.8/10 or higher.

The Internet Movie Database allows you to view separate episodes and their properties. However, I found that IMDB does not provide a handy interface of listing all episodes and their ratings. Filtering episodes based on their rating is also not an option.

Thus, tuvok was born.

How

The tool uses the API of the Open Movie Database (OMDb) API, which outputs neat JSON-files. The nice thing about OMDb is that you don't need an API key or developer registration to query the database, which makes it very accessible to developers. Of course, it is important to be 'nice' when using the API: tuvok only uses (1+ nr_seasons) queries for obtaining all information it needs.

To start, tuvok needs an IMDB-identifier of a series (string) and minimum rating (float). It then queries OMDb for information about the tv-series and all its seasons, collecting information about all the episodes in the process. The JSON-reply to a season-query also contains the basic properties of all its episodes. I like this, as it keeps the number of requests very low.

The IMDB-identifier is easy to find: just surf to the series page on IMDB and look at the URL. The IMDB-identifier of a movie object is of the form 'ttXXXXXXX'. Note that there are also identifiers that start with different letters, such as 'nm' (name) and 'ch' (character).

Here is a sample input/output of tuvok:

$ python3
>>> import tuvok
>>> tuvok.main("tt0060028",8.3)
Season 1
----------
8.5      11 The Menagerie: Part I
8.4      12 The Menagerie: Part II
9.0      14 Balance of Terror
9.0      22 Space Seed
8.5      25 The Devil in the Dark
8.3      26 Errand of Mercy
9.3      28 The City on the Edge of Forever

Season 2
----------
8.8      1 Amok Time
9.2      4 Mirror, Mirror
8.8      6 The Doomsday Machine
8.6      10 Journey to Babel
9.0      15 The Trouble with Tribbles

Season 3
----------
8.6      2 The Enterprise Incident
8.3      23 All Our Yesterdays

Star Trek (1966-1969) has 14 episodes with rating >= 8.3.

Limitations

Of course, taking a rating-based approach to episode-picking has its limitations. Some series get such (skewed?) high ratings for each episode that filtering episodes purely on rating does not help in weeding the average episodes from the truly good ones. Take for example Downton Abbey: all 52 episodes have a rating of 8.2 or higher! Consequently, using tuvok to pick a subset of episodes will not yield interesting results.

Secondly, dealing with user ratings is hard to begin with. Sites like IMDB, Rotten Tomatoes and RateBeer all have their own algorithms to calculate ratings. The question of how to transform the ratings of thousands of users into a single number is a difficult one to answer.

We could start an endless discussion about the meaning and value of user ratings. I'll not go into that here, although it is a fascinating subject, especially for data scientists.

More info

You can find the source of tuvok on GitHub.

City Guide Brussels

2016-09-28T00:00:00+00:00

Food

Nuetnigenoughe (Rue du Lombard 25)

Tiny but cosy restaurant with lots of obscure great beers. I had a Saison Dupont draft beer with the "Lamsballekes" (meat balls of lamb in tomatosauce with goat cheese with fries and a light salad). Beautiful room with high ceiliings, wood work, an antigue mirror and large fin de siecle window has you eating bistrofood in style.

Houtsiplou (Place Rouppe 9)

Burger bar with relaxed atmosphere. Slightly alternative. They also have really good tonic and the burgers are huge. Their walls are covered in graffiti art that depicts the history of Belgium.

Anata (Boulevard Anspach 74)

Very decent take-away sushi.

Drinks

Moeder Lambic (Place Fontainas 8)

In my opinion the best beer cafe in Brussels. Close your eyes and pick a random beer from the menu or ask one of the waiters. Their snacks (try a mixed plate of sausage and cheese) are also really good.

Cafe Capitale (Rue du Midi 45)

Although most of the coffee in Brussels is bad, here you can sink in one of the vintage reading chairs and enjoy the sound of the vinyl player. Bar features coffee-themed art work and sells small presents suitable for your hipster friends. They close at 8:00 pm, so it's perfect for an after-dinner coffee.

Pop-up Sablon (Place du Grand Sablon 15-16)

Try a cocktail, some wine or a beer in this stylish bar next to the Eglise Notre-Dame du Sablon. The Caesar Salad is also decent. Good spot for an after-work-drink and people watching.

How I prepared for my GIAC GPEN exam

2016-12-06T00:00:00+00:00

Last month I took the GIAC Penetration Tester certification exam as a conclusion to the course in Network Pentesting & Ethical Hacking I took with SANS in september. At the final exam, I ended up with a score of 86%, with which I am quite content. :)

In this short article I want to share with you why I chose this certification and how I prepared for the exam.

Why the GIAC Penetration Tester certification?

The GIAC GPEN certification is my first infosec certification. Before taking the course and the exam, I made sure that my chosen certification was valued within the infosec community (and not just by HR people or C-level managers). What has helped me to decide which course/certification to take was reading other people's opinions and experiences.

Lesley Carhart has discussed the value of certifications in this blog post, which is part of her infosec career series.

Eventually I decided to take the SANS course and GIAC exam since the SANS courses consist of both theory and practice. I specifically decided to go with the network pentesting course for two reasons. After leaving university, it turned out that my networking knowledge could definitely use some improvement. Secondly, I find that the offensive side of security interests me tremendously.

Exam preparation

I followed these steps to prepare for the GPEN exam:

Read all official exam information from GIAC
Read about other people's experiences with GIAC exams
Schedule the exam
Make an index
Use your practice exams
Take the exam
Celebrate

1. Read all official exam information from GIAC

GIAC has a nice overview of the certification process. They also have a page about exam preparation. Needless to say, it is a good idea to read these before you start preparing for your exam!

2. Read about other people's experiences with GIAC exams

Many people have written down their experiences with taking a GIAC exam. A bit of smart Google-ing goes a long way. The following blogposts have helped me prepare for the exam:

JDMurray has described his study plan and exam experience in great detail.
Josh Armentrout has written multiple posts on how he took the GSEC and GCIA exams.
Lesley Carhart has included a page with tips for taking GIAC exams on her website. She holds 5 GIAC certifications, so she's had a lot of experience taking these exams.

3. Schedule the exam

When picking a date for the exam, make sure you have plenty of preparation time. If you are planning to make an index of the SANS lecture notes and you have a busy day job, a week is not enough!

You have to take the GIAC exam at a proctored exam center. On the website of Pearson-Vue you can check the location and availability of the exam centers. When you have selected the center of your choice, you can pick a date and a time slot. When I tried to schedule my exam, the scheduling website failed multiple times. Be sure to set some time apart for booking a time slot with Pearson. ;)

4. Make an index

Every person has their own way of studying for a test. Since GIAC tests are open book, the most common way of preparing for a GIAC exam is making an index for the books you want to bring. Most people will use the SANS course books. For my GPEN certification, this meant indexing over 1000 pages of lecture notes.

I took a month to make my index, taking one day a week to go through one of the books. After a whole day of cramming powershell commands, tackling the stack of SANS books can feel a bit overwhelming. Taking some time between study sessions was a good way for me to let the theory sink in for a bit.

My index was fairly simple and consisted of a spreadsheet with columns "concept", "book_nr", "page_nr" and "description". I tried to put as much information as possible in the description field, since this saves you a lot of time during the exam.

I brought two copies of my index to the exam: one with the concepts in alphabetical order, and one in the original page order of the SANS book. I did this because related terms are often mentioned in the same part of the same book, which makes it easier to find a term if you don't know the exact name.

5. Use your practice exams

When you register for a GIAC exam, you get access to two practice exams. You can take these online via the GIAC website. The practice exams are a great way to prepare for the real exam. The format and time limit is the same as you will get in the proctored exam center and the practice exams will also give you feedback about the different exam topics. During the practice exams, you can also get immediate feedback about incorrectly answered questions. Taking one of the practice exams with your index and books only (no internet) is also a good way of checking the quality of your index.

6. Take the exam

Don't forget to bring two forms of ID and your books + index, and don't worry too much. :) Just do it!

7. Celebrate

When you have passed the exam, it's time to celebrate! Don't forget this step! It's very important. ;) After I passed the exam, I treated my colleagues to traditional Dutch oliebollen.

Getting started with malware analysis for absolute beginners

2017-01-05T00:00:00+00:00

A few months ago I did some basic malware analysis as part of an incident responder training. As I was completely new to the field of malware analysis, I learned a lot of new things. This primer is my way of sharing my experiences as a beginner in the field. In this write-up, I want to describe some of the basics and point enthusiastic beginners like myself in the direction of more self-education resources.

I use the following broad, pragmatic definition of malware: malware is basically anything malicious that potentially has a negative impact on a system.

Why learn about malware analysis?

The goal of malware analysis is to answer all kinds of questions about it by looking at the malware and its behavior. But why start with malware analysis anyway? If you know how to analyse a piece of malware, it becomes much easier to discover malware-related problems. It allows you to find out the cause of the problem and assess what damage has been done. It also dramatically improves your chance of solving the problem!

As such, knowing about malware analysis is a useful skill for anyone that manages a system that might be exposed to malware. Think of security professionals such as incident responders and SOC analists, but also system administrators and web administrators.

It also gives you an edge when talking to your boss. If malware pops up in a business environment, your manager will want to know whether the company will lose customers, money or its good name. When you know malware analysis, you'll be able to assess the situation and answer these questions better.

It's all about hygiene

For people that know the foundations of computer science or computer security, malware analysis is not necessarily hard. But it requires some basic understanding of computer hygiene. It is important that you understand the risks of working with malware samples and know how to take proper precautions against the most common risks.

The environment in which you do malware analysis is often called malware lab. This word invokes images of researchers with safety goggles and lab coats, like in a pathogen lab, where they research diseases. Naturally, a malware sample can be as dangerous as a pathogen sample.

Like a pathogen lab, a malware lab needs a set of hygiene rules. We want to prevent the samples from spreading, while at the same time, we want them to infect the right hosts so we can observe the sample in action.

Learning the propper programming or reverse-engineering skills is easy, but without proper security awareness, you'll get nowhere

How to build a malware lab

To get good at malware analysis, you need to practice: experiment with tools, look at different kinds of malware and get familiar with working in an isolated environment. The best thing to do is of course build your own malware lab!

To build a malware lab, I suggest you start with dedicated hardware. If you're just practicing, like me, just one system will do fine.

Just make sure the system has plenty of memory so virtualisation won't be a problem. Take 8 GB of memory as a minimum. You will probably want to run multiple virtual machines at the same time.

You also need enough disk space to save all the different virtual machines and snapsnots of those machines so you can return them to a clean state after running malware.

Virtualisation has some advantages which makes it very usable for malware analysts. It allows you to quickly load a malware sample on multiple operating systems. Secondly, you can save so-called snapshots of virtual machines to quickly restore a virtual machine to a previous, un-infected state.

Make sure your malware system is isolated ("airgapped") from all other digital systems and networks. You don't want to accidentally infect your personal computer, web server or backup drive. Don't connect your malware lab to the internet, because active malware can do all kinds of nasty stuff online. It can try to connect back to the system of original creator, connect to a command and control server or contribute to a DDoS attack on a website.

Static and dynamic analysis

There are two styles of malware analysis. Which one you choose depends on the questions you want to answer and the tools and techniques you're familiar with.

We can investigate malware by looking at the sample, its code and its properties without running it. This is called static analysis. Some examples of static analysis:

Comparing the hash of the malware sample to a database of known malware samples (such as VirusTotal)
Looking at the filetype of the sample
Extracting all ascii-strings from the sample and searching them for clues
Putting the sample in a disassembler and going through the whole assembly by hand to see what it does

The other form of malware analysis is dynamic analysis or behavioral analysis. This form tries to answer questions about the malware by running it in a controlled environment and looking at its behaviour. Some examples:

Running the malware and comparing the 'before' and 'after' states of the virtual machine afterwards
Running the malware while tracking its activities with a process monitor and a network sniffer
Hooking up an infected virtual machine to a fake virtual network to trick the malware into connecting to a malicious server
Going through the malware step-by-step with a debugger to investigate its internal states while running

To do a complete analysis of a sample, you will have to use both static and dynamic analysis techniques. Iterate this process to move from the easy basics to the more complex stuff.

Additional resources for learning

The following resources can be helpful if you want to learn more about malware analysis:

Practical malware analysis, by Honig & Sikorski (book and practice material)
A curated list of awesome malware analysis tools and resources
Open Courseware by RPISEC is a course on malware analysis based on the book `Practical malware analysis'
Anything written by Lenny Zeltser (webcasts, blog articles)
The SANS Digital Forensics Blog has some nice introductory articles
Tuts4you has a few tutorials on reverse engineering
On Crackmes.de you can find crackme and reverseme executables to reverse

Final remarks

When you start out doing malware analysis, the greatest part of the time will be spent tweaking your lab and setup. You will struggle with dependency hell and badly maintained github repo's, you will fight with your virtualisation software... Don't give up, once you have your infrastructure in place, malware analysis can be a lot of fun and very rewarding.

At this point, you know the basics of malware analysis. If you want to learn more, it's time to start practicing.

Rinse and Repeat: threat hunting with CapLoader and Wireshark

2017-04-05T00:00:00+00:00

When I was visiting TROOPERS17 in Heidelberg, Germany, I had the chance to follow a 2-day training in Network Forensics by Erik Hjelmvik. I'm glad I did! In this post I want to describe a technique I learned there. This technique is a nice way to investigate captured network traffic to find suspicious or malicious traffic. Although I dislike the term 'threat hunting', this is actually a neat and fast way to hunt for the weird and interesting stuff in a large PCAP-file!

The technique: Rinse and repeat

The workflow I want to describe could be called "Rinse and repeat". I like it because it is elegant: simple and flexible, but powerful if used right.

The idea is as follows:

Start with a large amount of network packets in a PCAP-file.
Define "normal" network traffic. Use Caploader to "rinse" (clean, remove) the normal traffic from the file.
Afterwards, investigate the remaining traffic with Wireshark.

Repeat step 2 and 3 and you'll end up with a PCAP-file that only contains unusual (ie. unexpected, unwanted, suspicious or malicious) traffic.

Of course, the way you go about your actual indepth analysis and what you do with the results of that analysis afterwards is entirely up to you.

What is 'normal'?

What you define as normal is completely dependent on the scenario you're investigating. Are you analyzing network traffic from a web server? Then, the traffic on port 80 will probably be normal, since this is where most web server applications run. Are you looking at a desktop computer that's not supposed to run a web server? Then, host traffic on port 80 might be suspect. For example, an attacker could be using a pop-up on-the-fly webserver to exfiltrate data.

Need to brush up on your basic knowledge of networking, protocols, etc? Don't worry, no one knows everything. Be sure to check out Julia Evans great comics, zines and blog articles. She's written a zine about basic networking concepts in a very accessible, fun and clear way.

Can I use flow data for this technique?

When you capture traffic for later analysis, I recommend you do full network packet sniffing. Flow data contains only parts of network traffic, such as it does not save the whole contents of a network packet, but only the interface, source, destination, protocol, ports, timestamps, etc. When we have the full network packet, including contents, we can use tools like Wireshark or NetworkMiner to extract all kinds of useful information for our analysis.

Why not just use Wireshark?

Caploader has a few advantages over Wireshark, which make it very suitable for this workflow.

If you have defined an aspect of normal behaviour, you can use CapLoader to hide all flows that show that behavior. This leaves us with a subset of network traffic that is not normal according to our definition. The cool thing is that CapLoader can automatically generate a new PCAP-file from this subset. By clicking the purple PCAP-icon in the top right of the screen, we can open this generated PCAP in Wireshark (or another program of our choosing for further investigation.

What I also like about this technique is that we don't have to use the Wireshark syntax for filtering. Both the Wireshark syntax and the Wireshark UI are not optimized for long queries, and editing Wireshark queries by hand can be a real pain. CapLoader allows us to hide entire groups of flows with one click of a button and generates a new PCAP for us to boot! YAY!

My own experiences

During the Network Forensics course, all participants received a 6-month license for CapLoader so we could play around with it some more. When I returned home after the training, I tried out this technique on my own web server. I definitely found some interesting stuff: stuff that I wouldn't have found going through my log files by hand.

So far, I found two drawbacks to CapLoader. Firstly, it is a Windows program, so I ended up transferring PCAP files between a Windows host and a Linux guest computer when I was doing my analysis. Secondly, a CapLoader license is a bit too expensive to buy for personal use, and I will be sad when my license has expired -- I really like this workflow and I still have to find an free and/or open source project that has the same functionality. However, if you just want to try out CapLoader, you can download a trial version with a 30-day license on the website, which has all the functionality you need for the workflow I describe in this article.

More resources:

You can download a 30-day trial version of CapLoader on the NETRESEC website.
CapLoader demo video that shows the basic usage
Check out Veronica Valeros talk on threat hunting (TROOPERS17). Although she doesn't name specific tools in her talk, she describes a similar workflow for threat hunting.

Some thoughts on responsible vulnerability disclosure

2017-04-09T00:00:00+00:00

It's hard for system owners to keep their systems secure. Servers and devices are scanned, defaced, compromised by the minute, by newbies playing around with github scripts, nation state actors and everything in between. Lucky for us, a growing group of ethical hackers spends time hunting and reporting vulnerabilities.

It appears ‘bug bounty’ surpassed ‘vulnerability assessment’ in search volume and gaining on 'penetration testing.' pic.twitter.com/ftDkIaVMMx
— Jeremiah Grossman (@jeremiahg) 17 August 2016

In this post, I want to shine some light on "responsible vulnerability disclosure". In some communities, this practice is known under a different name: Coordinated Vulnerability Disclosure (CVD). I think it would be a good idea if this practice got a little more publicity and recognition. Although it is slowly becoming more common, we are nowhere near done with defining and promoting responsible disclosure.

So what do I mean by responsible disclosure? My attempt at a definition is as follows:

Responsible disclosure is a form of vulnerability disclosure in which both the reporting ethical hacker and the owner of a vulnerable system try to solve the vulnerability together, by collaborating, sharing knowledge, communicating clearly with each other and adhering to some basic nettiquete rules.

As a security community, we need to keep sharing our experiences, both from the perspective of ethical hackers and vulnerable parties. We need to keep asking ourselves: 'What practices make vulnerability disclosure responsible?'

Where vulnerability disclosure goes wrong

There are plenty of examples where a security researcher has found a vulnerability but the disclosure of the vulnerability doesn't exactly happen in a responsible manner.

Some examples:

A vulnerability has been found, but it's unclear who the system owner is
A vulnerability has been found, but it's unclear how to contact the system owner
The system owner does not respond to messages
The system owner feels threatened and/or blackmailed by the security researcher
The systems owner threatens with legal consequences, lawyers or fines
The security researcher does not allow the systems owner to solve the vulnerability within a reasonable time frame
The security researcher does not provide enough technical details to reproduce or solve the problem
The security researcher demands money in exchange for the vulnerability report

What can ethical hackers do?

When you report a vulnerability, give enough information to reproduce the problem. It's also a good idea to list possible ways to mitigate or solve the problem, if these exist. In my opinion, this is both a sign of goodwill and extremely helpful to the person receiving this report.

Do not download or manipulate any data you might have encountered during your investigation. Don't make any changes to the vulnerable system.

Give the system owner enough time to respond. Take into account: office hours, time zones, different communication channels, and the fact that the systems owner is probably human (and not a time-bending superhero).

What can organisations do?

Create a Responsible Disclosure Policy and make sure it is easy to find on your website. Luckily, there are a few good policy examples online that you can use to design your own:

The National Cyber Security Centre of the Netherlands (NCSC-NL) has published a responsible disclosure guideline in English.
Floor Terra, a Dutch security researcher, has published an example responsible disclosure policy under a Creative Commons license which anyone can use.

Communicate clearly with people who want to report a vulnerability in one of your systems. Clearly state in your policy what the disclosing party can expect from you. For example: put down a timeframe in which the disclosure can expect a response to their vulnerability disclosure.

Take disclosures seriously. Keep ethical hacker up-to-date during the process. If you publish anything about the vulnerability, be sure to give proper credit where it is due.

My own experiences

I'm lucky that I had only neutral or positive experiences with vulnerability disclosure. In one of the better cases, I got an email of the vulnerable systems owner within hours, thanking me for my efforts. The vulnerability had been fixed.

However, there have been many cases where I could not determine the owner of a system, which makes responsible disclosure so much harder to do right.

My experiences at infosec conference TROOPERS 2017

2017-04-14T00:00:00+00:00

Last month, I visited TROOPERS17, a conference in Heidelberg, Germany. It was the first infosec conference I've attended, so I didn't really know what to expect. The website showed photos of geeky people in hoodies, a soldering table, lots of club mate and weird LED badges. I figured I could fit right in, and when I saw the training program I was even more motivated to register.

In the two days before the conference, I followed a training about network forensics. Earlier this week I wrote about the Rinse and Repeat technique I learned there, which can help you find the interesting network activity in a large PCAP-file.

Beautiful view and smart students in the #TR17 Network Forensics Training at 11'th floor of the PMA. pic.twitter.com/ngfyMtwvsi
— NETRESEC (@netresec) March 21, 2017

Atmosphere

With only 400 visitors, TROOPERS is a friendly and casual conference. What really surprised me: I met many different people and everybody was SO approachable and NICE! As one of the attendees put it: "TROOPERS is the only family where your parents don't ask you to fix the WiFi". Networking, which (in my head) evokes images of smooth talking and sprinkling business cards, was never so easy. I ended up sharing stories with incident responders, CEO's, security officers, programmers, policy makers, students, consultants and pentesters.

The open and welcoming atmosphere reminded me of the Chaos Computer Congress in Hamburg, although TROOPERS focuses specifically on information security.

THX 2 ALL!#TR17 pic.twitter.com/jWalWB9Le2
— Giaco (@giaco127001) March 23, 2017

Talks

The tracks consisted of a nice mixture of the different aspects of computer security. There was a track on attack and research, a track on defense and management, a sponsored track that focused on the security of SAP products (day 1) and a wildcard track for talks that don't fit a specific category (day 2).

I didn't attend a talk at every possible time slot. I tried to find the right balance between talks, hanging out at "lobby-con", talking to people, exploring the various extra activities at the conference and enjoying a nice cappuccino every now and then.

.@verovaleros found >70% of companies are infected by adware, and don't understand the extent of data exfil'd or how much it's worth #TR17 pic.twitter.com/XRjWncYb63
— Kelly Shortridge (@swagitda_) March 22, 2017

I can definitely recommend the following talks:

Veronica Valeros talked about threat hunting. Valeros works at Cognitive Threat Analytics, a research department that is part of Cisco Systems. In her presentation, she talked about her methods for finding the needle (malicious activity) in a haystack (large amount of network traffic). She also shared a case study from her own work: it appears that adware can have a higher negative impact than most people expect, since adware is often used to exfiltrate sensitive data (such as browsing history) from infected computers. In some cases, the adware will communicate unencrypted with the ad server, which results in sensitive data being transmitted in plain text over the network!
Matthew Domko gave this talk about BroPy, a Python tool that can automatically generate a baseline for network traffic that. This baseline can subsequently be used as input for Bro, as a definition of normal network activity. Of course, it's required that the system is completely free of malware when you run BroPy, otherwise malicious activity will end up in your baseline. I liked this idea, but I'm sceptical whether this approach will work for existing systems.
There was a very good presentation about the North Korean Red Star operating system and the way it can be used to track unwanted media files within the country to monitor dissidents.
@TheGrugq gave a surprise talk about intelligence services.
A surprisingly good and critical talk by Mara Tam about cybersecurity policy. Worth watching for her story about submarines.
Pentester @_Staaldraad gave a great talk in which he explains a very interesting persistence and post-exploitation technique: tackling persistence in a organisation after compromising one of their workstations by burrowing in the organisations Exchange server. I was impressed by the way he explained the technique: it was accessible and clear without being overly simple. Worth watching if you ever use Exchange. This talk will make you freaking paranoid. ;)

Events and extras

On Wednesday night there was a social dinner, with Packet Wars (CTF) immediately afterwards. I ended up joining the WizardsOfDos, CTF team of Hackerspace Darmstadt, who were one person short for their team. It certainly was a lot of fun and we ended up winning the Packet Wars!

The #packetwars at #TR17 has begun ... pic.twitter.com/E9dXBo2vXD
— Sam Flynn (@l0x2a) March 22, 2017

The CTF challenges revolved around a fake Donald Trump twitter account, which we had to compromise in order to obtain different kinds of evidence. The first challenge was OSINT-based. A writeup is available here. The second challenge consisted of sending a spearphishing email to the fake account to compromise the underlying workstation. If you want to see some of the hilarious phishing emails that were sent during this challenge, be sure to check the closing talk of TROOPERS17 on Youtube.

Other activities and stuff worth mentioning:

Good food.

Decent coffee.

Refridgerators with club mate and club mate cola everywhere.

A vintage pinball machine and donkey kong arcade machine (AWESOME)!

My very first weird geeky conference badge gadget! The TROOPERS17 badge was made by @BadgeWizard and consisted of a custom Arduino-board paired with an (ancient & refurbished) Nokia 3310. Sadly, my battery was as good as dead, so I didn't use it much during the conference. However, the badge was a great conversation starter and also doubled as an attribute for one of the conference challenges. Nicely done, TROOPERS!

Just a random heap of #TR17 badges prior @WEareTROOPERS . @0x4045494650 @net0SKi @welectronic pic.twitter.com/sFsMLY8rxw
— Brian (@BadgeWizard) April 5, 2017

Verdict

Long story short: I would definitely recommend visiting TROOPERS. I hope to visit again next year and meet even more interesting people.

The end...The end of a great conference! See you in TR18 #tr17 pic.twitter.com/OgWxcsw4PE
— Cr0wTom (@serbinhio) March 23, 2017

Travel tips from my shoestring days

2017-04-21T00:00:00+00:00

When I was in university, I didn't have a lot of money for traveling. Lucky for me, I had an adventurous friend who wasn't afraid of traveling on a budget. We had a lot of fun adventures together throughout Europe, buying the cheap train tickets and sleeping at CouchSurfing hosts to save money on accommodation. I found that the art of slow travel can give an extra dimension to traveling, which makes it a lot more enjoyable. This is what I learned during those trips.

Don't lug around too much stuff

Traveling light means you have more energy left to enjoy your travels. Also: checking in and out of airports becomes a matter of minutes.

Take your time

It's better to stay too long in one place than to run through all items on your "5000 things to see before you die" list.

When traveling in Rome, I met four girls from the US who where viewing Europe in four days. Europe. Four days. That meant hardly any sleep, long train trips with only distant views of a few very beautiful cities, and never sitting down for a proper meal. Another friend did "Paris in 24 hours", which included the bustrip to and from.

This all sounds very exhausting to me, and it's definitely not the way I want to travel. It leaves no room for the stuff that I like best: enjoying the food, getting lost, meeting new people, trying new things.

Embrace not being in control

When I travel, I find that not being in control gives me great sense of peace. Not knowing where I'll eat or sleep next, I have to focus on getting around and surviving in a strange new city. This means I don't have any time left to worry, over-think and over-analyze everything, which (ironically) means that I end up making better decisions.

The first thing to do when in a new city

This is the best travel tip I've ever read on a blog: when you arrive in a new city, take a break immediately upon arrival. Sit down for 30 minutes. Take some time to settle. Grab a cup of coffee and recharge from traveling by train or airplane. Observe the people that walk by. Find a free tourist map. Chat with the barista and get your first travel recommendation.

Do as the Romans do

In my experience, asking recommendations from locals is the key to comfortable, authentic travel. They can tell you about the best food in town, the tourist traps you need to avoid and the best way to get around.

When I was couchsurfing in Spain and Sweden, the couchsurfing hosts provided me with great tips: the best spots for a barbecue, a hidden zoo in the middle of town, a local battle-of-the-bands-event (which was the best activity we did for that particular town), a beautiful garden which was also accessible at night and the location of the best coffee bar in town.

Some ideas for obtaining this super secret information:

Just ask someone! Talk to random people on the street or in shops. Ask them whether they live around here, and what they think is the best place for dinner, a drink, or some fun.
I always check the local Wikitravel page before I travel. This has helped me in the past with avoiding tourist scams and other negative experiences. It has also saved me a lot of money on subway tickets in Paris and London. ;)
Try to find local guides to the city. Google for blogposts describing the local hotspots and check out the local tourist office. In Brussels, I found this really cool map made for students, by students. Me and my travel companions ended up having dinner for less than 5 euros in a restaurant that catered to the local Chinese population.

Do the normal stuff

Don't forget to have a taste of daily life in the country you're visiting. Visit a supermarket. Celebrate a national holiday together with locals. Have a barbecue. Watch a sports game and root for the local team. Join your CouchSurfing host for a cup of coffee. View life through the eyes of the local population, and you'll really get to know a country.

When I was London, of the most fun things I did was go to a Waitrose supermarket and buy EVERYTHING I wanted to try. I ended up with a dragons hoard of chutney, Fentiman's soda, tea and chocolate. In Germany, I went crazy with buying all kinds of mate soda. And of course, the best place to buy Belgian beers and chocolate is in a Belgian supermarket.

Dare to wander

Dare to just go somewhere and see where you end up. Dare to be lost.

When I was in Rome, me and a friend rented an appartment in Monti. Every morning we would leave our appartment, choose a random direction and just start walking. We would have breakfast at the first inviting-looking coffee bar, find some sandwiches or pizza al taglia and continue walking. After a week, we knew the part of the city where we stayed by heart, and we didn't need a map anymore. It feels great to be in the largest cities of Europe and to know exactly where you are -- and where the nearest coffee bar is.

Don't hide behind your camera

I once took a camera with me on my trip and ended up with 5000 photo's of cows, buildings, views and bicycle lanes and absolutely NO MEMORY of the trip itself whatsoever. If a landmark or view is worth making a photo of with your smartphone, it's worth looking at it with your own eyes. I figured I can always buy a postcard that'll look more picturesque than I'll be ever able to take.

Write in your travel journal

Write about your experiences! My travel journals are a better keepsake than all the souvernirs I could have bought on my travels.

I used to tack five blank A4 sheets together, and toss them in my bag with 2 pens. My travel journal was not a fancy molekine, but it weighed almost nothing and hardly took up any space -- very important when you're traveling with only one small backpack. I used the journal to scribble down everything, from bus schedules and phone numbers of couchsurfing hosts to handdrawn maps to the places we wanted to visit. The sheets that remained, I filled with random drawings and observations.

Don't forget to take time off

It's really important to recharge your batteries every now and then. This is especially important to keep in mind when you're traveling for business or for a longer period of time. Business trips, with activities like long meetings, conferences or training sessions can be super-tiring. If you feel obligated to spend your evenings off racing around town to get all the tourist things, you can easily feel overwhelmed.

I heard from many people that they find it hard to slow down during travel and take some time to do nothing. Some people even feel really guilty when they decide stay in to relax.

If you feel like spending your evening in pyjama's watching a series while eating popcorn: go ahead and enjoy it! Stock up on tasty food in the local supermarket and do absolutely nothing for a while. Traveling should be fun, don't forget that.

I hope these tips help you. Happy traveling and enjoy the journey! :)

Searching for a job after graduation

2017-04-26T00:00:00+00:00

As I was finishing the last chapters of my thesis in September 2015, preparing slides for my final presentation and fighting with a particular tricky LaTeX module, a voice in the back of my mind reminded me that I needed to find a job.

Part of me believed that a suitable PhD position would turn up. But the more time I spent doing research for my master thesis, the more I wanted to do something else for a while. Something not quite like research. But what?

Too many options

It turned out I had only a vague picture in my mind about Life After Graduation. I halfheartedly looked up a few vacancies online, but that didn't help me at all. As someone who never had a job in IT before, the job titles, descriptions and lists of requirements didn't really mean anything to me.

As a starting point, I decided to research companies that I knew from word of mouth. However, I was only familiar with organisations where I knew people, the companies that sponsor the student associations and the companies that give lectures at university, mostly for recruiting purposes. Many of these organisations either had nothing to do with computer science, or were small (web) development companies looking for junior coders. Not really my cup of tea.

Demand-driven job search

This was the point where I decided to do things differently. As I learned in a training recently, when you have a to process a lot of information to make an informed choice, it's better to search for information based on your objectives, instead of trying to process everything there is on offer. If you try to process all information that you can possibly get, your brain easily gets overloaded by all the options and the decision-making process becomes less effective.

In other words: it was time for a demand-driven job search. I had to find out what I wanted from a job, and then find out which employer could provide this.

However, when you're knee-deep in mathematical proofs and you've never had a full-time job in your life, it's kind of hard to come up with a complete picture of your own wishes. I decided to make a list to help me with this.

I started out by listing my strong points, the things that make me stand out from others, and the things I like to do best.

I also wrote down what I wanted my life to look like after graduation. I figured that if you spend 30-50 hours a week at your work, it's going to leave a big mark on your daily life, right?

Mind mapping different career paths

I also made a mind map of different possible career paths. Since I have many different interests, there were a lot of different career paths I could choose, in many different fields. I decided to draw a mind map to plot all the different courses of action. I hoped it would help help me choose between them.

I wrote my four major options in the corners of a large blank paper. For each option, I wrote down possible employers, the things I needed to learn to work in that field, the most interesting subjects, the pros and cons of this career path, etc.

Possible career choices

Science journalism, philosophy of science, the role of science in society.
Have my own company
Computer science research in academia
Research and development in a commercial company

I thought I'd choose one of these careers, which would automatically rule the other ones out. Now that I have a little more work experience, I don't think it has to be so black and white. I think it's perfectly okay to have multiple interests and career paths. Actually, I know one full professor who changed his specialisation when he was in his fifties.

Example mind map for Career path "Science journalism"

Write for New Scientist (magazine)
Do research for Institute of Science In Society
Write for newspapers
Focus on writing
Scientific journalist?
Be the bridge between people and science
It is possible do an internship at a journal?
Ionica Smeets does this for math
Freelancing: no financial security?
Work from home (yay!)

I used the lists and the mind map as a compass for my job search. Now I could specify what I wanted, I could start searching specifically for jobs that had to do with my interests, or that come with the activities and work environment which I would like to have in a job.

I still have the lists and mind map. I review them occassionally, to see if my opinions have changed over time. In retrospect, the things that I wrote down were very useful. Nice colleages, freedom to manage my own time and an employer that values my expertise were the most important requirements for me to be happy in my job. I was lucky to find a job that had these requirements. I've been working happily at my first job for one and a half years now.

I hope this method helps you in your own journey towards your dream job. :) Good luck!

City guide Rotterdam

2017-04-30T00:00:00+00:00

These are the places I enjoyed when I was visiting Rotterdam. Note that this list is nowhere near complete, as it focuses on the things that I like (coffee, tacos, books).

New Fork (lunchroom)

Very close to the main city center and the maritime museum, this lunchroom has the best sandwiches of Rotterdam. The bread is hands down the best I've ever tasted. They have a lot of veggie options on offer, which is nice for travel groups with vegetarians. Apparently frequented by students a lot, so the prices are very decent as well.

www.newfork.nl, Coolsingel 141, 3012 AG Rotterdam

Donner (bookstore, giftstore, coffeebar, lunchroom)

Wherever I go, I always get drawn to the local bookstore. Donner is situated in an old bank building, which gives it a peculiar atmosphere. They have everything you can think of in terms of books, and also sell sheet music, board games, vinyl and gifts. There's a cute coffee corner in the back of the building, where you can have lunch or drink a nice cup of coffee with a slice of cake.

www.donner.nl, Coolsingel 119, 3012 AG Rotterdam

Water taxi

Rotterdam is split in two by the Nieuwe Maas river. If you need to go from the north side of Rotterdam to the south side (or vice versa), you can walk across one of the bridges, or... you can take a water taxi. These little speedboats will bring you from one side to the other in minutes, and it's a lot more fun than walking. A ticket of EUR 4,50 will cover a transfer between any two stops within the city centre. On the website you can find an overview of prices and locations.

www.watertaxirotterdam.nl/

Fenix Food Factory

The Fenix Food Factory is a group of indie stores and restaurants in an old facory hall. It consists of a variety of small food businesses with a large food court in the middle. You can buy something from each of the stores -- cheese shop, butcher, bakery, coffee-roastery, microbrewery -- and sit in the middle and enjoy it all. Be sure to go here with enough appetite for one or two meals. ;)

When I was here, there was enough to see, try and taste to warrant a stay or a few hours. We started out with lunch at Meneer Tanger, a counter that sells spices and all kinds of morrocan tapas. We had a nice cup of coffee afterwards. I then bought 5kg worth of locally sources honey, home-made chutneys, fresh dairy and jams. We wrapped up our visit with a few beers and a tray of cold cuts at the Kaapse Brouwers, a local brewery.

www.fenixfoodfactory.nl, Veerlaan 19D, 3072 AN Rotterdam. Note that all shops except for the brewery are closed on Mondays.

The Stirr (cocktail bar)

A cocktail bar that only serves custom cocktails. Just tell the bartenders what you like and they'll bring you a newly invented alcoholic beverage. The bar has a grungy vibe, with high windows and a bare brick wall. There's no loud music or bright lights and the high windows have a beautiful view, so this is the perfect place for having a drink, relaxing and talking. I paid EUR 23,00 for two cocktails, so it's on the expensive side, but the laid-back atmosphere made it money well-spent.

www.thestirr.nl, Eendrachtsweg 29B, Rotterdam

Xoco (taco bar)

This is a taco bar with the best tacos I ever had in the Netherlands! Although they call themselves as a grill restaurant, it has an informal atmosphere and no is more like a student cantina. The menu is simple but very tasty. You choose tacos, a burrito or a salad, choose between different meats or the veggie option, add toppings and sauces and you're good to go. Be sure to try the home-made peppersauces!

www.xoco.nl, Scheepstimmermanslaan 17, 3016 AD Rotterdam

Burger Club (burger restaurant)

Tiny burger restaurant in the north-west of Rotterdam. Apparently they won a prize for their burgers a while back. The burgers are pretty good, there's a lot of choice on the menu and eating here is not too expensive. Interesting is that they let you choose the kind of meat for your burger: local-produced beef, wagyu-beef or high quality pork. If you like your burger with a gin-and-tonic, they have about ten different gins and tonics on the menu.

www.burgerclub.nl, Nieuwe Binnenweg 139-a, 3014 GJ Rotterdam

Gys (organic restaurant)

I encountered this cute eaterie on my way back from the Burger Club. The design of the place was so great, I decided to go in and have a coffee as desert. I wish I was a freelancer in Rotterdam, so I could go here during the day and enjoy the coffee, beautiful design, the light and the view from the enormous windows. I will definitely return here and try the rest of the menu. They serve everything from breakfast to dinner, with great coffee to boot.

www.gysrotterdam.nl, Nieuwe Binnenweg 182, 3015 BJ Rotterdam

Things I learned after 18 months of working

2017-07-25T00:00:00+00:00

I was very lucky to get a job immediately after graduation. In one month, I went from hippy-vibed Nijmegen and a no-rules-just-write schedule to the imposing The Hague, to sit in an open-floor-plan style office from 9-to-6. It's been quite an experience.

When I started working, I quickly noticed there was more to learn than what they teach you at university. Since I'm approaching my one-and-a-half-year workiversairy, I figured it was time to write down some of the things I learned.

Here's some tongue-in-cheek advice for nearly-graduated folks.

When you leave university after graduation, you feel like you're ready to change the world, solve all the problems and stagger everyone with your genius. Then you get an actual job and you go work in an office where you meet all these old people (gasp!) with kids (gasp!) and their ancient knowledge of How Things Are Done Here. Organisations turn out to be these horrible structures with things called rules and processes and a fixed chain of command. Things go way slower than you imagined and overthinking this will make you sad. You're in for some culture shock. Be prepared.

When you're new at a job, don't expect to change the business overnight with your brilliant outsider insights. Although you might see things that could use some improvement, save your opinions until you've seen enough of the organisation to understand why things are the way they are. Pro-tip: asking "Why?" is generally more appreciated than remarking "You're totally doing it wrong!".

Don't try to do everything alone. There are people that can fix the printer, send mail, etc. and they do it way more efficient than you. Moreover, it's OK to not-know things. I hope you have some friendly colleagues (like I had), who can explain the stuff they don't teach you in school.

Don't let the constant stream of unneccessary emails drive you crazy. About 90% of the emails I receive (newsletters, system notifications, spam) are completely irrelevant to my work. If you want to keep your sanity, learn to build good filters.

When you start participating in Office Communications, you will learn quickly that most humans don't know how to write concise emails. As a result, a lot of people won't actually read their email: they glance at the subject and sender and try to decide who to forward this email to. I decided I don't want to be a part of that problem, so I learned How To Email. I particularly like Matt Might's article How to send and reply to email and Patrick McKenzie's General Tips for Emailing Busy People.

Prepare a phone conversation in advance if you have to explain difficult technical stuff or a tricky situation (like a security incident) over the phone. The order in which you say things on the phone determines whether people will panic or not.

The things that you don't like to do (because they are out of your comfort zone) are probably the things you're going to learn most from. If this scares you, go read Daring Greatly by Brene Brown, or watch her TED talk on vulnerability.

Final wisdom: don't eat all of the birthday cake.

Now I'm wondering: what would you tell your younger self about work, life in the office or careers? Please share in the comments, I'm really curious about your stories.

Digesting difficult books with Holiday's reading method

2017-09-06T00:00:00+00:00

Blogger and book master Ryan Holiday wrote an article in which he describes his personal way of reading books. His method builds heavily on note-taking and is optimized for recording thoughts, associations and interesting quotes during reading. These notes can then be used for further research or as writing material. After stumbling upon Holiday's blog post, I wanted to try this method for myself.

Using Holiday's method for reading non-fiction

What better way to test Holiday's method for reading books by testing it out on one of his own? So I went to the local bookstore and obtained a copy of "The Obstacle is the Way", a book about the philosophy of the Stoics. Then I followed the method as described in Holiday's blog post. Here are my experiences.

Reading a summary and reviews before you read the book helps with figuring out the structure of the book. This meant I had more energy left for taking in the actual contents of the books.
The whole note-taking-and-highlighting made it easier for me to concentrate on the book and stick with reading. I definitely got less distracted when I was reading with a highlighter in my hand. I also noticed that I was reading slower, but I finished the book as a whole faster.
I think I was taking in more of the content since I was hyper-aware of the structure of the text. This definitely helped me decide which parts to highlight.
It is also really easy to write a short summary of the book afterwards, since all the important passages where highlighted already.
Revisiting my notes feels like using a shortcut to the best parts of the book. I can now 'read' this book within 10 minutes, which is awesome.

I chose not to copy all my highlights to little note cards, since I don't have an immediate use for them. I can imagine that this might work great for research purposes, though.

In short: I really liked this method, especially for reading non-fiction. I might use this more often, especially when I need to read for research purposes, or when I want to write a review of the book.

Epilogue: my review of The Obstacle is the Way

After reading Holiday's book, I must say I understand the reviewers who faulted the book for the writing style.

It is interesting that you can clearly find traces of Holiday's research method in the book. The narrative leans heavily on quotes from famous people and other writers. Almost every paragraph follows the same structure: Holiday poses a statement and then backs it up with various quotes and anecdotes from literature/history. This can get repetitive quite fast. Some people might find it boring. I, being a big history nerd, was mostly amused by the various historical characters that Holiday uses to support his story. But, as Holiday himself says: "Read the classics." The Obstacle is the Way is an entertaining book, but it mainly serves as a springboard to better and more original reading material.

I'm curious as to whether you, my dear readers, have tried this method and what your experiences were. Do you have a different method for reading dense material or difficult books?

Roger monitors changes in HTTP status codes

2017-10-03T00:00:00+00:00

Some time ago I found sensitive personal data on a public web page. I notified the system owner, but was curious to see if my report would have any impact. I decided to monitor the web page to see whether the information was removed.

Since I only wanted to know whether the web page was still online, monitoring changes in the HTTP status code (instead of changes in page content) was enough for this use case. One way of keeping up with the status of a webpage is hitting Refresh until you get a different HTTP status code... But I decided to let my computer do this for me, so I could use my time for more interesting activities.

I decided to write roger, a Python tool for HTTP status monitoring. roger tries to load a given url, saves the current HTTP status code, and then sends you an email whenever that status changes.

How does it work?

Roger follows a fairly straight-forward algorithm:

Load the page for the first time and save the resulting HTTP status/error code:

# first time connecting to target to get initstatus
try:
	print("Connecting...")
	connection = urllib.request.urlopen(
	    urllib.request.Request(testurl), timeout=60)
	initstatus = connection.getcode()
	print("Initial statuscode", initstatus)
	connection.close()
except urllib.error.HTTPError as error:
    print(type(error), error)
    initstatus = error.code
    print("Initial statuscode", initstatus)

Then, we keep repeating this process every n minutes, until we get a different result:

while not status_changed:
    print("Entering test loop with timeout of", str(n), "minutes")
    time.sleep(n * 60)
    print("Testing...")
    try:
        connection = urllib.request.urlopen(
            urllib.request.Request(testurl), timeout=60)
        print("Statuscode", connection.getcode())
        if connection.getcode() != initstatus:
            newstatus = connection.getcode()
            print("New statuscode detected", initstatus, "->", newstatus)
            status_changed = True
        connection.close()
    except urllib.error.HTTPError as error:
        print(type(error), error)
        if error.code != initstatus:
            newstatus = error.getcode()
            print("New statuscode detected", initstatus, "->", newstatus)
            status_changed = True
return newstatus

If a status change occurs, Roger composes an email with the details and sends it to me via my local mailserver. You can expand this script with different forms of notification. For example, you can combine roger with my project zazu, to make a Twitter-bot that automatically posts updates about the status of your website. Or you could send the output of roger to an irc-bot, to give you notifications over irc. The possibilities are endless.

roger can be used for all kinds of OSINT activities. I'm considering expanding it later to include content monitoring. The Python code is available on Github.

A quick overview of my PhD research project

2018-02-12T00:00:00+00:00

I'm working as a PhD student at the University of Twente, where I research natural language generation for adapative games. People often ask me what my research is about, so I figured I should write a blogpost to explain a bit more about my research field.

What is a "natural language"?

A natural language is a language that is used by humans, such as English, Dutch or Japanese, as opposed to the formal languages of mathematics and logics. Note that I use this very informal definition, since I'm a computer scientist and not a linguist. If you ask a linguist to define a natural language, you get probably many different answers, such as the definition from Wikipedia.

What do you mean with natural language generation?

When I say "natural language generation", I mean the task of automatically generating (creating) human language text, for example with a piece of software. The software takes data as input, and outputs text in a natural language.

You can create all possible types of texts with natural language generation: dialogues, riddles, jokes, social media messages, novels, poetry or more formal texts like weather forecasts, product descriptions and sports match summaries.

In some sense, natural language generation (NLG) is the opposite or inverse of natural language processing (NLP); with NLG, you generate text from data (which can be text as well), with NLP you extract data from natural language texts. Some people hope that you can apply NLP techniques to NLG problems and vice versa, by designing algorithms that work in both directions.

Natural language generation is part of computational linguistics, the interdisciplinary research field where computer science and linguistics meet.

Can you give me an example of natural language generation?

An example of a natural language generation program is a program that greets the user by name, with a greeting that depends on the time. The input data in this case is a name and the current time, and the output is a sentence in English, which is a natural language.

Input		        Output
("Judith",11:06)        "Good morning, Judith!"
("Hector",14:45)        "Good afternoon, Hector!"
("Rachel",19:25)        "Good evening, Rachel!"

What is your research project about?

In my specific research project, I try to apply natural language generation techniques to adaptive training games, ie. training games that adapt their content to the player.

The two subjects (natural language generation and adaptive games) that make up my research project are both too extensive for one PhD project. This means I'll have to choose a more specific focus for the upcoming four years. At the moment (february 2018), I'm mostly reading (survey) papers and thinking up as many relevant and open problems as possible. Later this year I will decide on more specific research questions together with my supervisors and the rest of the project team.

Random-tony generates chocolate flavours from a context-free grammar

2018-10-03T00:00:00+00:00

One of my first experiments with natural language generation was writing random-tony, a program that could produce new chocolate flavours. The program uses a grammar as a model of a "valid" chocolate flavour. Here are some examples:

$ python ./generate_flavours.py 

blonde chocolate with marshmallows and cucumber
dark chocolate with orange
white chocolate with liquorice, rose petals and kiwi crumble
blonde chocolate with cardamom, soy beans and broth
dark milk chocolate with cola, mayonnaise and butterscotch
white chocolate with sesame seeds, pumpkin and black pepper
dark chocolate with jasmine and honey
extra dark chocolate with apricot and broth
dark milk chocolate with caramel and pear crumble
extra dark chocolate with cucumber

Parser and generator for context-free grammars

When we generate text based on a context-free grammar, the grammar acts as a model for valid language production. So before I could automatically generate text output, I needed a program that can parse and interpret a context-free grammar.

Nowadays, we can easily use Tracery, an excellent and very usable javascript framework by Kate Compton for doing language production with grammars. However, when I wrote random-tony, Tracery didn't exist yet. Instead, I wrote a parser and generator in Python 3 for context-free grammars, based on code from Eli Bendersky.

The grammar parser can parse any context-free grammar; the generator takes a grammar and a start symbol, and produces language by applying grammar rules to the start symbol until there are no further rewrites possible. If you want to learn more about grammars and rewriting, check out the Tracery tutorial.

Chocolate flavour grammar

As input for the parser/generator, I wrote a context-free grammar to describe chocolate flavours. I was inspired by the crazy limited edition flavours that Tony Chocolonely (a famous brand of fair-trade chocolate in the Netherlands) releases each year. The popularity of Tony's crazy flavours has caught on, and now almost every supermarket in the Netherlands sells chocolate with weird combinations of flavours.

The grammar defines valid chocolate flavours. I have defined it as a type of chocolate (dark, milk or white) and one, two or three special ingredients. A chocolate bar without any ingredients would also be a valid flavour in the real world, but since this would be a bit too boring, so I left these out of my definition.

LIMITED_EDITION -> CHOCOLATE INGREDIENTLIST
INGREDIENTLIST  -> INGREDIENT 
                 | INGREDIENT INGREDIENT 
                 | INGREDIENT INGREDIENT INGREDIENT

I looked at the flavours made by Tony's Chocolonely and decided that an ingredient can be a number of things, such as fruits, nuts, vegetables and an assortment of other random weird things.

INGREDIENT      -> FRUIT | FRUIT CRUMBLE 
                 | NUTS | DRINK | CANDY 
                 | SPICE | VEGETABLE | MISC

The fun part is filling in concrete examples of all the categories of ingredients specified above. For example, here is the list of 'MISC' ingredients.

MISC            -> honey | coffeecrunch 
                 | butterscotch | wasabi | red curry 
                 | green curry | jalapeno peppers 
                 | balsamic vinegar | lavender 
                 | rose petals | cornflakes
                 | mayonnaise | ketchup | soy sauce 
                 | broth | whipped cream | liquorice | salmon

Why is this interesting?

So, why bother with writing a context-free grammar for NLG? Mostly, it's fun. The random, combinatorial approach to generation can lead to weird, crazy and unexpected outputs. A similar approach is often used for procedural content generation for games. If you want some examples, take a look at the procedural histories of Dwarf Fortress, the loot in Diablo, and the NPCs of Caves of Qud.

As an unexpected bonus, random-tony has helped me land my current job. I developed it before I started my PhD in natural language generation, just for fun, but it helped me show my prospective supervisors that I was really interested in the main topic of research. This meant I had a headstart in the application procedure. Now, two years later, this little generator is helping me with my research too! Since my PhD project is about personalized (adaptive) natural language generation, I could probably use a modified version of the program to conduct experiments.

Choose the right bias for your text generator

2019-02-07T00:00:00+00:00

A suitable corpus, or dataset of text documents, is one of the main ingredients of many natural language generation projects. There are many general-purpose corpora available for download on the internet. For example, the natural language processing library nltk has a built-in download module, through which you can access various standard datasets. Anyone can download these datasets for free, which makes them a great NLP resource for programmers and language researchers.

However, sometimes you need a corpus specifically for your project. This is particularly the case for data-driven approaches to language generation, where, instead of manually coding a model of what you want to generate, you train a model on a dataset. This results in a model that reflects certain properties of the corpus, because the text in the dataset acts as an example of "valid output text". Depending on the chosen model and way of preprocessing, output texts will feature the same vocabulary, word-length, topics, sentence structure or grammar as the text from the corpus. If your input dataset is biased in any way, that same bias will end up in the results of the generator trained on that same dataset.

In other words, if I use weather reports as a basis for generating text, my output will also look like weather reports!

Some of the most popular textual datasets contain public domain books, news documents, Wikipedia pages, and product reviews.

If I use newspaper articles as starting point for generating new text, my output will probably contain terms related to politics and economics. Books from project Gutenberg infuse my output with the old-fashioned language found in 19th century novels. Using Wikipedia articles as a source might introduce crowd-sourced spelling mistakes in the text. And using a dataset of Amazon product reviews riddled with opinions will lead to output text with lots of subjective language.

This "contamination" is fine, as long as it is deliberate and it fits with the intended application.

Keep this in mind next time you choose a corpus for your NLG project.

Thanks to Gerdriaan Mulder for proofreading this post.

Video game corpora

2019-03-02T00:00:00+00:00

Last updated: October 2020

In this post I want to share some fun data sources for NLG and NLP. In an earlier post, I mentioned why a suitable corpus is important, especially when you're doing natural language generation. Since I'm working on natural language generation for video games, I try to find good datasets for that domain. However, finding datasets with text data from games is hard, which means that often I need to scrape and compile my own corpora.

I'm also collecting examples of games that use NLG, to see which techniques are used by developers and how these affect the quality of the game.

Game code

Finding information about NLG in games is hard, since game companies generally don't publish their game code. Luckily for me, it's easy to inspect the code of games that run on client-side javascript, such as CookieClicker and open source games like Max Kreminski's idle game Epitaph.

Both games feature NLG techniques, which is an added bonus. CookieClicker has a news ticker that shows weird headlines depending on your game process. Epitaph is text-based and procedurally generated, which means it runs completely on NLG. The game generates fictional societies, languages and Civilizations-like tech trees.

Fan-made resources

Popular games such as the RPGs Diablo, Deus Ex and Bioshock have crowd-sourced FANDOM wikis, where you can find collections of game text. The Deus Ex wiki contains the text of in-game ebooks and newpapers. The Bioshock wiki has transcriptions of all audio diaries that the player can find in the game. Warning: crawling these wikis might lead to spoilers. ;)

The Imperial Library has a database of all books (and other booklike items) from the Elder Scrolls games. Some of my students used it to generate new lore for Skyrim. I imagine this would be a nice technique for Skyrim modders that want to introduce new books into the game.

Thuum.org is a website about the Dragon language from Skyrim. Besides numerous language-learning resources, the Thuum Library includes a text-dump of Skyrim dialogue, which (interestingly enough) includes notes for voice actors about the prosody of some of the dialogue lines.

Github user efonte has extracted all texts from the text-heavy RPG game Disco Elysium. (If their link is broken, you can find a fork at my own Github profile.)

Javier Torres has written a manual for extracting dialogue from Morrowind using the Elder Scrolls toolkit.

Scripts

Film scripts do not contain video game text, but are still fun for my application domain, especially for generating dialogue.

I'm really impressed by the fan community of D&D webshow Critical Role, who have transcribed all episodes of Critical Role.

There are also transcripts of Star Trek Voyager episodes available online. I'm secretly planning on turning these into a Vulcan irc-bot.

I hope you liked this list of text resources. If you know other fun datasets that could be of use to NLG/NLP research in the context of video games, let me know!

Update October 2020

I've started to mine game files directly, because that's the best way to get high-quality video game text. I've published a paper about it, which can be read here. I've also released three datasets with text from video games, sourced from Star Wars: Knights of the Old Republic, The Elder Scrolls and Torchlight II.

Jakub Myśliwiec, whom I supervised for his bachelor thesis project, has also scraped a dataset of World of Warcraft quests, which can be found here in plaintext format. If you want to use the WoW quests for your own research, you need to do some preprocessing as this plaintext file is meant for fine-tuning a neural generation system.

City Guide The Hague

2019-03-03T00:00:00+00:00

I lived in the Hague for two years. These are some of my favorite spots for lunch, dinner, coffee and drinks.

Coffee etc.

Bijenkorf City Café

The ground floor of this huge upscale department store has a small cafe overlooking one of the busiest crossroads in the Hagues city center. It is my favorite place for people watching. Order a cappuccino and grab one of the chairs in front of the windows. They also have tasty cakes.

www.debijenkorf.nl, Wagenstraat 32, 2512 AX, Den Haag

8tea5

The best place for bubble tea, situated in the beautiful Passage.

8tea5.com, Passage 77, 2511 AC The Hague

Exki

My favorite coffee bar at the Turfmarkt. Lebkov and August (the other two coffeebars) are okay too, but Exki's lemon-drizzle cake with white-chocolate topping is just to die for. They also sell Palais de Thes tea.

www.exki.nl, Turfmarkt 224-226, Den Haag

Food

Minglemush

Food court with lots of cute little food stands. Minglemush has lots of different food options, plus craft beers, wine and gin & tonics. Minglemush lies on the east side of The Hague Central Station. It's almost never found by tourists, who tend to immediately exit the train station to the west upon arrival to go to the city centre. There also an upstairs, where you can find an arcade with old-skool games. Try the poke from the Pokébar or the genuine mexican tacos from Mixco.

www.minglemush.nl, Anna van Buerenplein 712, 2595 DK Den Haag

Bik Frietwerk

Get your fries (regular potato or sweet potato) from this haute cuisine snackbar, only 2 minutes away from The Hague Central Station. Great for lunch and dinner. I prefer the sweet potato fries with mango-mayonaise, or the regular fries with guacamole.

bikfrietwerk.nl, Turfmarkt 228, 2511 DJ Den Haag

Istana

The Hague is famous for its great indonesian food. If you have some time and money to spend and want to taste good rendang, go to Istana.

www.istana.nl, Wagenstraat 71-73, 2512 AR Den Haag

Warung mini

This tiny warung (small restaurant or cafe) is famous throughout The Hague. You might need to wait for a long time, as it is popular with locals. When I lived in the Hague, I went to eat there regularly. The roti leans to the salty side, but it's comfort food and comes for a reasonable price. They also do take-away, although you might try Moksi (a few streets further away, at Boekhorststraat 119) for shorter waiting times.

www.warungminidenhaag.nl, Amsterdamse Veerkade 57-A, 2512 AH Den Haag

Eazie wok

If you want a cheap and healthy supper, try the wok dishies of Eazie. The "indonesian cocos" with fried rice and fish is my number one favorite. Both The Hague Central Station and The Hague Hollandsch Spoor have an Eazie within walking distance.

eazie.nl, Gedempte Gracht 78 (near Central Station), Herengracht 26 (near Central Station), Stationsweg 136 (near Hollandsch Spoor)

Broodje van 's Gravenhage

Cheap but tasty roti-filled burrito's ("roti-wrap") for take-away. Perfect train food.

broodjevansgravenhage.nl, Rijnstraat 14, 2515 XP Den Haag

Visiting ICCC 2019

2019-07-03T00:00:00+00:00

This is a ~~short~~ report of my visit to ICCC, or the International Conference on Computational Creativity 2019, in Charlotte, NC. It's not meant as a summary, but as an overview of the things I found interesting during the conference. If you want more information about the talks, check the conference program, or read the live-tweets from Matthew Guzdial, Christoph Salge and me.

I've marked some places in this text with a (*), which means you can find an accompanying paper in the ICCC 2019 proceedings.

ICCC is a small conference: single track and about 150 participants. The conference concentrates on the automation of creativity in all its aspects, from creative computer science (how can we generate paintings, poems or dance?) to conceptual AI (how can we formalize the cognitive aspects of human creativity?), cultural sciences (does a computer generated text differ in cultural value from a human-written one?) and philosophy (can computers be creative? What does creativity even mean?). Despite the varying backgrounds of the participants, most of the talks were understandable and relevant for my own research. ICCC turned out to be a pretty good fit for my research projects, which deal with coherence and context of automatically generated texts.

I found that Computational Creativity doesn't just value a generative system for its raw output, but also pays attention to the conceptual idea behind a system. Attendees seemed to appreciate the thought process behind a project: what input do we use and why, which approach do we choose, what libraries and datasets are available, do we make the project open-source or not, etc. I visited ICCC to present a poster about Churnalist, a headline generator for creating fictional headlines around a context. Churnalist is certainly not state of the art when it comes to language generation, but the system is ideal for testing hypotheses about text generation for a specific context. At ICCC I noticed that this topic interested other participants as well, and that people could appreciate the conceptual side of Churnalist, even though its outputs are not that sophisticated yet.

I saw also a certain openness about negative results in research work at ICCC, which I think is very important, as transparency in research leads to more progress. If you know which approaches work and why, it's easy to adopt part of someone else's approach in your own research, even if their work is in a different application domain. I know this sounds like basic science methodology, but in practice researchers tend to hide their bad results or ugly academic code, and use the approaches that sound most promising to funding agencies. In other words, the ICCC community matched with my own ideas about science methodology and research best practices.

Workshop on Generative Deep Learning

On 18 June, I attended the workshop on deep learning and computational creativity (DL4CC). The workshop was an informal meeting with about 30 people and consisted of presentations and discussions. Even though I don't use deep learning in my current work, the quality of the discussions made it worthwhile to attend.

Highlights of the workshop:

AI-driven systems for co-creation should no longer be seen as unintelligent tools. Instead, they must be designed as articulate creative partners. It would be nice if these systems not only gave the user more creative possibilities, but also made suggestions and critical remarks during the creative process and corrected or even interrupted the user. The biggest problem in deep-generative system is a human-computer-interaction problem. Active collaborating with AI fits with the new focus on reflection and framing in Computational Creativity: "simply creating something" is no longer enough for a creative system. Systems should present generated output together a justification, so that the user can follow the underlying reasoning and generative process. See also the recent revival of "explainable AI".
Some speakers defined "creativity" of outputs using the parameter "surprise", but I was critical of this. Creativity judgments defined in terms of surprise can never be objective, because surprise is strongly linked to experience and consequentially to age, social environment, etc. Nick Montfort's keynote on day 3 (see below) made me slightly less sceptical.
Mikhail Jacob presented a project with virtual agents that play improv theatre in VR. He emphasized the importance of generating a "creative arc" instead of the optimal artifact. If a virtual agent would always choose the most creative artifact (for example, the weirdest or most surprising act), you would not get a coherent story over time. I can imagine that this is important for generating any artifact with a sequence, such as music, dance or animations. I recognize this problem from my research in language generation, where, for example, consecutively choosing the most beautiful word does not always lead to the most coherent text.

After the talks, there was a discussion session in which we brainstormed about the current challenges for deep generative systems, to find fruitful directions for new research. The following five main points emerged from this session:

How do we evaluate creative systems and generated artifacts? How do we define creativity?
How do we deal with human-computer co-creation? How should new generative systems behave to facilitate collaboration? How do we build the next generation of explainable, reflective, framing and collaborative generative systems?
What do we do with gaps in the latent space of generative systems? How do we find these gaps, how do we classify them and how do we address them? In other words, how do we "escape" from the latent space learned from a training set?
As deep learning is applied to more application domains, we run into domains for which no 'big data' is available (to use as training data). How do we deal with this?
How can we build personalized models in deep generative systems? What additional training data do we need from the user, how do we get that data and how do we incorporate it in the general model?

The results of this discussion have been published online for the rest of the scientific community.

Day 1

The first day of the conference was opened Mary Lou Maher. She explained ICCC's approach for welcoming new people to the conference (which I loved as a first-time attendee): first-time ICCC participants were explicitly invited to ask questions after a talk, and session chairs were asked to favor new people over acquaintances when picking audience members for questions.

Rebecca Fiebrink gave the first keynote. Her work centers on building usable creative systems that use supervised machine learning. Her approach to researching creative systems reminded me of Kate Compton, which also places the user first. Fiebrink gave a demo of The Wekinator, a supervised machine learning system that can be used for creating music. The system solves the problem of lack of training data by asking users on the fly for examples.

In her keynote, Fiebrink emphasized that sometimes it is better for a system to focus on the mistakes in the model. In machine learning, we tend to focus too much on positive metrics like accuracy. However, for a user it's more useful to know where the gaps and mistakes are in the underlying model, and where these mistakes come from. I recognize this difference from a CLIN talk from Erik Tjong Kim Sang about historians using ML to classify historical documents. Big surprise: the historians only wanted to use machine learning if it was 100% transparent where classification errors came from.

I also thought the HiBot was very cute: an arduino robot with motion sensor that cheerfully waves back whenever someone waves at it.

Kyle Booten presented work* which was inspired by Erasmus' book De Copia, a book with exercises for improving rhetorics and text variation. I like both the fact that Booten drew inspiration from a historical Dutch figure (and texts and methods) and that he conducted small, elegant crowdsourcing experiments to test his hypotheses.

Tony Veale discussed a new Twitterbot* that responds to any Twitter account that uses the phrase "Read me like a book". The bot analyses tweets from these accounts to build an 11-parameter personality and language model for that account, and uses the model to tweet book recommendations to the user. The bot builds on Veale's earlier work on metaphor generation and computational humor. An example conversation between Veale's Trumpbot and the book bot can be read here.

Although I like work related to Twitter bots and language generation, I'm critical of the bot's social behavior. It tends to tweet to unsuspecting users that have not explicitly invited the bot to interact with them, and the output can contain sensitive or controversial content, which is not in line with the best practices for Twitter bots.

Razvan Bunescu mentioned a paper by West and Horvitz, in which researchers turned a classic linguistic crowdsourcing experiment around: they presented headlines from satirical news site 'The Onion' to crowdsource workers and asked them to change one word in the humorous headline to change it into a serious headline, for example "BP ready to resume oil {spilling, drilling}". This is much easier for humans than asking them to come up with satirical headlines on the spot, but the result is the same: a high quality dataset that can be used to study satirical language.

Poster session and demos

At the poster session I presented my prototype of Churnalist*. The session coincided with the demo session (in another, adjacent room) and social event with good food and wine. I spoke to various people (Kyle Booten, Sarah Harmont, Eyal Gruss, Pablo Gervás and many others) about my work, their work, art, music, our experiences visiting the USA and the future of computational creativity, which I enjoyed immensely.

Kyle Booten presented a demo called Fragile Pulse*, which seemed a satirical version of meditation apps. I didn't have time to try it out, but the demo revolves around a text written by Booten that is paired with a microphone and motion sensor. If the reader of the text makes noise or moves a lot, the text is modified by a text generator to include all kinds of anxiety-related words. In other words: the only way to read the original text is by sitting still and being silent.

Day 2

On day two of the conference, artists Lilla LoCurto and Bill Outcault opened with a keynote about conceptual art that incorporates technology. The keynote started with a bit of contemporary art history, with lots of examples by other artists. I particularly liked Beuys Voice by Nam June Paik (a robot built from old television sets) and Stochastic Process Painting by Cheyney Thompson (a painting consisting of tiny hand-painted squares, each square's color is picked algorithmically by a computer). LoCurto and Outcault also disucssed their own works the willful marionette and cat's cradle -- it was fascinating to hear about the development of their artistic work. In order to realise their ideas, they needed to collaborate with organisations, programmers, and researchers throughout the entire process.

Mike Cook recorded a talk about his survey* on framing in computational creativity. Framing can be seen as the sign next to an artwork in a museum. It gives the audience extra context in which they can interpret the artwork: the name of the artist, the year in which it was made, methods and materials used and the story behind the piece. Cook argues that creative systems should provide something similar when presenting output to the user. I figure that providing some kind of framing for an output is the minimum requirement for creating more explainable generative systems. It can also help us take a step in the next direction: creating generative systems that can reflect on the things that they're generating. I have not read the accompanying paper yet, but I estimate that it will be important in my work -- and that of the rest of the Computational Creativity community -- in the coming years.

Alison Pease made an interesting observation about framing in academia: "Papers frame systems, talks frame papers." Very true, and it illustrates clearly why framing is such an important concept in Computational Creativity.

Day 3

Nick Montfort opened the third and final day of the conference. In his keynote he distinguished between H-creativity (global, universal) and P-creativity (personal, individual) and posed this question to the audience: what lies in between? Drawing on examples, Monfort argued that subjectivity plays a larger role in computational creativity than we (scientists) like to admit, and that it should get more attention, especially during evaluation. In the evaluation phase, we should keep in mind the cultural background of the evaluators. Similarly, the definition of "creativity" is a cultural artifact, and consequentially not something objective. We shouldn't abandon the notion of objective research, but we should be aware of the culturally determined aspects of creativity. Montfort finished with slides with advice for Computational Creativity researchers, which are well worth a read.

A presentation by Maya Ackerman about field work in computational creativity* listed creative ways of obtaining feedback during every phase of the research process. She also did a small reading of the published book with computer generated stories of the MEXICA story generation system by Rafael Perez y Perez.

Christoph Salge gave a short talk about the Generative Design in Minecraft Competition. I was happy to hear that this year there will be a bonus assignment focussing on chronicle generation*! The goal is to generate a chronicle for the procedurally generated Minecraft settlements, i.e. a text that describes the historical background of the settlement. It should be clear from the generated text which chronicle belongs to which settlement. I'm looking forward to the results of this new bonus challenge, as it has some overlap with my PhD topic: natural language generation for games. If you are interested in participating in the competition, be sure to check out the website. The organizing team made it really easy to participate. As long as you can code basic Python, you can experiment with their bootstrapping scripts and create a submission for the competition.

Pablo Gervás talked about storytelling system INES that can create multiplot stories*. I particularly liked the architecture of the system, which is based on multiple microservices. Secondly, I recognized Gervás' story about the development process from my own experience: although the goal was to design and implememt the system, the primary researcher ended up working on one tiny but complex part of the system during his PhD.

At some point in the talk, Gervás also mentioned the following dilemma for reseachers: are we going to build tiny systems that tackle one specific problem, or are we going to investigate large difficult problems that we encounter in many different problem spaces? Building solutions to tiny problems leads to completable projects with a clear scope and thus quicker/more publications. However, solving the large difficult problems is probably more valuable for science and society in the long run, but not as "profitable" (career-wise) for early career researchers because of the high pressure to publish. An interesting observation.

The final conference session was about computational humor and colorful language.

Khalid Alnajjar presented work on a creative headline generator built by a team from the University of Helsinki. Although their goal was different from that of Churnalist, our implementations used the same libraries and datasets -- an encouraging observation.

Thomas Winters gave a talk about joke generation for 'I like my X like I like my Y, Z' jokes*. The trick is to find nouns that have a particular (adjective) property in common, such as: "I like my coffee like I like my war, cold". He and his co-authors also created JokeJudger, an opensource platform in javascript for generating and evaluating this type of joke. It might be useful for evaluating other types of text too.

Conclusion

In short, I felt right at home at ICCC. The research community was very welcoming to newcomers, and the talks and social events were enjoyable, interesting and productive. Next year's conference is in Coimbra (Portugal) from June 29th--July 3 2020, and I'm definitely going to try to visit ICCC again. In 2021 the conference will be in Mexico City, Mexico.

If you want to talk about any of the stuff above, send me a message on Twitter!

(Thanks to Ruud de Jong for proofreading this humongous blog post!)

COG 2019 conference report

2019-09-03T00:00:00+00:00

By popular demand, another conference report! Please note that this report is very biased towards the topics that are relevant for my PhD research, i.e. text generation and procedural content generation for games.

Between August 24 and August 27 2019 I visited the IEEE Conference On Games (COG) in London, UK. In previous years, this conference was called Computational Intelligence and Games (CIG). The last edition of this name was in Maastricht in 2018.

As opposed to Foundations of Digital Games, where most of the presented research seems to come from the social sciences, COG is more focused on the artificial intelligence aspects of games. The conference had two sessions on procedural content generation (PCG) alone, and I met multiple people with a background in mathematics or computability theory. This was not surprising given the type of research that was presented: during the poster session I saw posters about search spaces, generator expressivity, modal logic, and optimization, among other topics. Other than that, the conference had special sessions about esports and opponent matchmaking.

The majority of the participants seem to be from the digital games industry, although there were some board game related talks. For example, Hamna Aslam gave a very useful tutorial about playtesting board games on Wednesday.

Industry day and demo session

The conference opened with the industry day on Tuesday, with about 450 visitors from both academia and the games industry. The subsequent days had about 250 visitors.

During this first day, the social media coverage was so extensive that I decided not to livetweet during this conference. However, in the days after, the amount of tweets decreased. It seems that on average games industry people are more active on Twitter than researchers.

I was particularly interested in meeting people during industry day, as I wanted to talk to people from the gaming industry to find out whether they would be interested in using text generation for writing game content. I’ve visited local game developer meetups in the past, but that was a very different crowd than I encountered at COG. I met people from game development companies from all over the world (albeit mostly from Europe), including giant game publishers like Microsoft and Sony.

I was at COG to demonstrate headline generator Churnalist, and the demo session was planned during industry day. Although I already presented a system description paper at ICCC this year, this was the first event where people could try out the system themselves.

I extended Churnalist with a Flask (Python) back-end and a Javascript front-end, to make sure that it was a bit more user-friendly than the bare-bones version. Although it is meant as a human-computer co-creation tool, I programmed the demo in such a way that it could run without any user interaction. On startup, the system generates 50 headlines and showcases them in a web page with a CNN-style news ticker. I used the call for papers of COG as input for the generator, so the headlines included phrases like 'high-profile industry', 'academic leaders', artificial intelligence' and 'procedural generation'. Whenever people wanted more details, I could show them the interactive elements that turn Churnalist into a writing support tool for game writers.

Visitors of the demo session seemed to appreciate Churnalist and its possible applications (the newspaper flavour text from Deus:Ex is one of my favorite examples), and I had multiple conversations with interested researchers and developers. Antonios Liapis stopped by and mentioned the Fantasy Newsroom Bot, a twitter bot that tweets fake news in a fantasy setting. If you stopped by during the demo session: it was great to talk to you, and thank you for all your suggestions and questions.

As an added bonus, the demo of the “fake news generator” made it easier for people with similar interests to spot me later at the conference. Well done to the organizers for planning the demo session relatively early in the conference.

A static version of the demo, with pre-generated content, can be viewed here. The headlines are based on a description of the bachelor programme Creative Technology at the University of Twente.

Research highlights

Javier Torres, whom I met last year at FDG, presented a poster about modeling NPCs (agents) and their speech goals with logic. The idea was that agents can’t just reason about their own goals and speech acts, but also about those of other agents. In other words, they can choose to lie, but also find out that other NPCs are lying because there are logical inconsistencies in their speech!

Guenter Wallner had a poster about analyzing Twitter data to find different categories of gamers on social media. I’m always interested in seeing NLP research applied to social media data (a popular topic also at ACL this year). Besides that, the poster mentioned a way of measuring topical coherence in texts, which has interested me ever since I tried to judge the coherence of NaNoGenMo novels.

Mike Cook (whom I finally met IRL at COG after many exchanges on Twitter) gave a very clear talk about smoothing for monotonic generator functions. I have to set aside some time to ponder how we can use this for text generators. The talk can be viewed here: it is the first talk in the session on PCG and AI for Game Design.

Ahmed Khalifa presented a word game called ELIMINATION, in which the challenges are generated with PCG. The talk was interesting, and I also loved the art style of both the game and the slides (see the paper for examples). I liked how the procedurally generated content was evaluated in a realistic game context -- as it should be! This talk made me realise that COG is probably a good fit for my future research, as I’m planning on implementing a game so I can evaluate my generated texts in a representative environment. The talk can be viewed here: it is the fourth talk in the session on PCG and AI for Game Design.

Joshua Sirota presented about procedurally generated grounded languages for NPCs. He created a set of agents (neural networks) that learn a language by playing a referential game, with reinforcement learning for the learning. Although the title reminded me of work by Mark R. Johnson, who generated procedural language for his game Ultima Ratio Regum, this work is fundamentally different because the language is grounded, i.e. the agents try to learn the meaning of language as it applies to the "physical" world they can "see". The talk can be viewed here: it is the fifth talk in the session on PCG and AI for Game Design.

Dan Ashlocke and Vanessa Volz have started a special taskforce on player evaluation. They are bringing researchers together to work on standardized metrics for evaluating player experience and game content. I have joined their mailing list, as I want to investigate how we can evaluate textual game content, such as narratives and dialogue. If you want to contribute as well, contact one of the organisers.

Christoffer Holmgård of modl.ai gave a talk about the research gap between academia and industry in the video games domain.

Keynote by Emily Short (natural language and games)

The absolute highlight for me was the keynote by Emily Short, titled “Natural language and games”. Short works as Chief Product Officer at Spirit AI (London), which develops tools for natural language understanding and natural language generation, specifically for games. Their main product is the Character Engine, an authoring tool for game development. It can be used to generate dialogue, or to support a human game writer. It can also be used to build games with natural language understanding (think of the text input of games like Zork).

When creating dialogue, the system keeps track of various aspects of the conversation: where we are in the global narrative, what the emotions of the speakers are, and what we know about the world in which the dialogue takes place. The Character Engine uses a list of classifiers for detecting various kinds of speech acts and conversational intents. It can be expanded with additional classifers to tailor to specific projects or domains.

Short talked about an experiment Spirit AI did to test their bot with real humans. The experiment involved two sets of participants: the first was a group of 1800 beta testers, and the second was a group of trade show visitors at GDC. Both groups interacted very differently with the bot.

The beta testers used shorter phases and ‘played’ multiple times. Surprisingly, they were also a lot more abusive towards the bot. The trade show visitors used longer and more varied sentences, and interacted with the bot for shorter periods of time. Their language also contained more typos. Because of these results, Short advises playtesting NLU games with a large number of people, so the bot can be tested on a large variety of users and inputs.

The keynote went beyond a regular “We developed thing X, this is how it works” talk. Towards the end, Short discussed the difficulties in applying natural language processing and generation for games. Games differ in a few aspects from the usual applications of NLG:

Real world information might not be applicable in the game world, and thus strategies for finding answers to out-of-domain questions might not work.
the player/user/reader is role-playing, which means that they have a different type of interaction than regular chatbots have with their users, such as a person has with a customer support chatbot.
There’s a larger variety in interaction styles: the NPC chat bot has to be able to mix functional questions (“How are you doing? Is grass green?”) with narrative moments to bring the story forward (the bot/NPC wants to introduce a certain quest line) and act in a believable way according to the NPCs goals and emotions (the bot is playing an alien from planet Y that doesn’t trust the player yet). All these types of interaction depend on different dialogue parameters and different NLG techniques.

In other words: problems that might be trivial to solve for a customer service chatbot could be complex when developing an NPC bot.

Short also mentioned an unsolved problem in sentiment analysis research, one that I have encountered in my own projects as well. If we want to develop sentiment analysis tools for games, we need a classifier that is tailored to the gaming context. However, a proper model for this context is hard to create, as most sentiment analysis research is focused on (and thus biased towards) product reviews. Most of the datasets that are used to train and evaluate sentiment analysis tools are crowd sourced product review datasets, for example from Netflix, Amazon, etc.

We really need open-source language models for sentiment analysis that can generalize to different domains, including game-playing and fictional game worlds. I'm currently working with Lorenzo Gatti to solve this problem for Dutch. We are trying to improve Pattern.nl, an NLP library for Dutch that also contains sentiment analysis functionality.

Keynote by James Dean (esports and WEAVR)

The conference ended with a keynote by James Dean, the founder of the UK division of the Esports League (ESL). I wasn’t familiar with the professional esports world, and it was nice to get an overview of the activities of the ESL. Esports are fundamentally different from traditional sports, notably in terms of monetization: only 15% of esports funding comes from advertising. The ESL is involved in a research and development project called "WEAVR" together with the University of York. The project tries to find new ways of engaging the audience during sports matches, as methods from traditional analog sports are not really suitable for esports. The project tries to incorporate VR, personalized information, adaptivity, and augmented reality.

Dean urged academia to put more effort into connecting with industry. People working in industry generally don't read scientific articles, so relying on publications to bring your research findings to people outside academia is not enough. York University used an industry outreach project to connect with ESL, which led to the collaboration.

Conclusion

COG was an enjoyable conference, where I encountered a lot of the usual suspects from FDG and ICCC and met loads of interesting new people. Especially if you’re looking to discuss your technical ideas with people from both industry and academia, this is a good conference to visit.

My estimate is that it will be a good fit for my future research. I haven’t done much research with digital games yet, as games are only a possible application of my work in natural language generation. Next year I plan to evaluate Churnalist in a games context, either with users, writers, or both. Another idea I have is to write a small game and use it to test the basic assumptions on which I build NLG tools. Both lines of research would fit nicely in COG’s program, so I’m sure I will visit this conference again.

The organisation has recorded all talks. Once they are uploaded, I will update this blogpost with links to the talks. You can find the proceedings here.

If you want to talk about any of the stuff above, send me a message on Twitter!

Thanks to Ruud de Jong for proof-reading this post. :)

Twitter guide for researchers

2019-09-23T00:00:00+00:00

In this blogpost I want to explain how I use Twitter, and how social media is helping me with my research.

This blogpost is meant for researchers that either already have an account at Twitter or want to start using it. Throughout this post, I do assume you are already familiar with how Twitter works. If you're not, read the documentation in the Twitter help center or just create an account, dive in and learn it by doing. ;)

How I use Twitter

I mostly use Twitter for these three things:

getting news about my research field when I am not at a conference
networking with other people: researchers, artists, industry people and companies.
communicating with others when I’m attending a conference

I use Tweetdeck combined with Twitter lists to manage streams for different accounts and topics.

“A list is a curated group of Twitter accounts. You can create your own lists or subscribe to lists created by others. Viewing a list timeline will show you a stream of Tweets from only the accounts on that list.” Twitter help: lists

“TweetDeck offers a more convenient Twitter experience by letting you view multiple timelines in one easy interface.” Twitter help: Tweetdeck

Using Tweetdeck and lists has the added bonus that I can separate my work-Twitter from my personal Twitter. I only read my work-related streams during working hours and read my personal timeline in my free time. :)

Here’s an example of my Tweetdeck streams, one for my research topics, one for my university and one for my department. On the right, there is a column with tweets with the hashtag #phdlife.

Getting news about my research field

I have two main lists for work: one with members of my research department and one with people who regularly post things that are relevant to my research.

Building the first list is fairly straightforward: you ask colleagues whether they’re on Twitter, add them to the list, and then take a look at their followers/following lists. You should be able to find most of your colleagues by walking their social network graph.

The list for your research topics is harder to build. The key is to build this list organically. It doesn’t have to be perfect the first time.

You can grow it over time by combining these strategies:

Try to find the relevant hashtags for your research field(s). In my case, that meant scrolling through the search results for #NLP, #NLG, #NLProc, #PCG, #procgen, etc. Some hashtags, such as #NLP, are mainly used for marketing, which means they can be ignored. On the other hand, #NLProc is a hashtag that is popular with the natural language processing academic community. Twitter accounts that use #NLproc might tweet stuff that I find interesting can be added to the Twitter list.
Look at the Twitter accounts that mention (or follow) the conferences you plan to attend. So far, this is my best source of list members. The majority of the members in my “research topics” lists are people that have attended CLIN (NLP), iNLG (NLG), ICCC (computational creativity), FDG (games) and COG (games). I noticed that important conference dates are shared earlier on Twitter than via the regular SIG mailinglists, which I find very handy.
Whenever you really enjoy the work (talk, publications, code, datasets) of a particular researcher, do a quick check to see if they are on Twitter. If they tweet about their research, add them to your list. Don’t add people to the list just because they work in your field and have a Twitter account. Some people just like sharing pictures of their dog or vacations. That’s a fine use of Twitter, but their messages shouldn’t end up in your carefully curated work stream.

Every now and again, go over your list and see if you should change anything.

I tend to read the tweets in this list whenever I’m at work and want to know what is happening in my research field. Sometimes I read the tweet stream daily, sometimes only once a week. Don’t worry about missing a few tweets. As your list grows, it becomes harder to keep up with the entire tweet history. Consider the twitter list a way to get quick updates on the “general feeling” of the field.

Networking with other researchers

Every now and then I come across work (a paper, a game, a blogpost) that I really like, or that has a special connection with my own work. At moments like that I like see if the author is on Twitter. If yes, I go over to their profile, introduce myself and remark on the overlap between our work. I can ask questions about papers I’ve read, ask for preprints of new work, ask about conferences they’re attending, or propose a collaboration. Twitter is perfect for meeting new people this way. It’s basically a never-ending, informal, digital conference.

It depends on your field how many of your peers are on Twitter as well. Don’t get discouraged if you can’t find the right people right away. If you start tweeting about your own work and interests, the right people might find you instead!

Bonus: moral support

Twitter is also a great place to discuss academic life in general. Use these hashtags to talk to other academics: #askacademia, #academia, #phdlife and #phdchat.

Using Twitter during conferences

At the very least, I follow the conference hashtag(s) (there’s often more than one!) in a special column in Tweetdeck. I create this stream by making a special column for the query "conferencename OR @officialconferenceaccount OR hashtag1 OR hashtag2". The advantage of Tweetdeck is that the column refreshes itself in realtime, so you get an ongoing livestream of messages.

In the special conference stream, I can see what’s going on in the “hallway track”, what’s happening in the different parallel sessions, and whether there are people interested in grabbing a drink and talking about specific topics. During the conference, the organisers sometimes post important schedule updates and announcements at the official Twitter account, if they have a good communications/social media chair. Conference participants can use the conference hashtag to ask questions and organise meet-ups. Discussions on Twitter often range from the contents of talks to directions and the best place to get coffee in town.

Live-tweeting talks

Sometimes I like to live-tweet during a conference. Live-tweeting can come in many different forms. Most often it means writing a small summary of a talk, sharing a picture of particular good slides, or sharing a link to papers or resources that the presenter mentions.

When there is already sufficient coverage of the talk itself, I prefer to focus on the meta-matters. In this sense, live-tweeting can also be: mentioning similar or related work; contrasting the research method to other approaches, recording particularly good oneliners or jokes, making pictures of interesting visuals, video-recording a live-demo, tweeting a question or commenting on the state of the field.

I’ve made quite a few friends by live-tweeting, and can recommend it wholeheartedly. It can take some effort, as you have to listen to the talks and jot down the interesting points (either as notes or directly as tweets) and curate your tweets and photos at the same time. However, I find that this heightened concentration helps me digest the contents of the talks better. Plus the tweets can serve as the basis for my elaborate conference reports.

"Ok, that's all very well, but what should I post on Twitter? How do I get people to notice me?"

It might be clear from this guide that I mostly use Twitter as a source of information. It is perfectly fine to just lurk on Twitter and not post many tweets yourself. However, if you would like to build a research-themed account on Twitter, here are some ideas for what you could post:

conference Call For Papers deadlines
conference dates & locations
announcements whenever you publish a new article or paper, preferably with some keywords and a link to the full text (if possible) and maybe a short summary.
links to published opensource software, printables, datasets and other research outputs that might be useful for other people
information about events that you will be attending
information about events that you would like to attend (this lets other people know that livetweeting is appreciated)
links to newspaper articles and popular science articles about your research field
comments on newspaper articles and popular science articles about your research field
links to other people's blogposts, papers, tools, etc. that you appreciate
relevant job openings at your department
relevant job openings at other universities
searching for research participants
asking and answering questions about your research field
academic memes
pictures of your lab, your office, demo sessions, conference visits, your colleagues, how you work, etc.

And, paraphrasing the words of blogger Julia Evans:

anything you wish you knew an hour/a day/a week/a month/a year ago.

About popularity

Some people like to measure the success of their Twitter usage by looking at Twitter's popularity metrics, such as the number of people that follow you on Twitter. Personally, I don't care much about the number of followers I have. I have been using Twitter since my bachelor studies and the number of followers has grown organically over years -- although I have noticed a sharp increase after every live-tweeted conference. I advice you to not care too much about social media metrics. It's most important to (1) have fun (2) make Twitter work FOR you.

All metrics in academia should be taken with a grain of salt, for the reason mentioned in this tweet:

Hi! I'm a researcher! It's taken me a month to write down my research plans for the coming two years. That plan is 1936 words long.

Apparently, my writing speed is 64 words/day.

Thankfully, we all know that "words written" is a meaningless measure of scientific productivity. https://t.co/3xIx5AfWw0
— Judith van Stegeren (@jd7h) October 16, 2019

I hope this guide gives you ideas about how you can wield Twitter as a powerful tool (sorry, I've been watching too much Critical Role) in your research practice. If you have any questions or tips, leave a comment below or talk to me on Twitter!

Thanks to Christoph Salge and Khiet Truong for giving feedback on this guide.

City guide Nijmegen

2019-10-20T00:00:00+00:00

A list of my favorite spots in Nijmegen, where I lived for seven years.

Coffee

Coffeelovers Mariënburg

This coffee bar is situated inside the city library. It has a long table, where you can drink a cup of coffee after perusing books, or you can entertain yourself with people-watching. When I lived in Nijmegen, I went here every saturday afternoon, after visiting the organic farmers market.

www.coffeelovers.nl

Bairro Alto and Downtown

Great spot for a good cappuccino or a chai latte. Bairro Alto has a cute hipster vibe because of the Etsy-sourced interior and dishes. Downtown is a bit more modern, as it caters to flocks of freelancers that bring their Apple computers to work in this clean, white, light-filled coffee bar.

www.bairroalto.nl and Downtown's facebook page

De Blonde Pater

Cute lunchroom and coffee bar right in the best part of the city center. It's always busy between 12am-2pm, because it's one of the best places to sit outside. The cafe is right next to the Lange Hezelstraat, a street filled with small boutiques.

www.blondepater.nl

Drinks

Café Samson

Craft beer, waiters with serious beer expertise and lots of free peanuts.

Café Jos

Good craft beer pub, just outside of the city centre.

Brouwerij de Hemel

City brewery situated in the 17th century "Commanderie van Sint Jan" building. Drink local beers, eat some snacks and soak up some sunshine on the big terrace. De Hemel also sells all kinds of products that are made with their own beer and spirits.

Dinner

Blixem

You can get a decent dinner here without going bankrupt.

Popocatepetl

Maybe I am biased, but I just love Mexican food. I particularly like the chicken fajitas. This restaurant is my favorite when I want to take people out to dinner.

Wally

Here you can eat the tasty, slightly haute-cuisine burgers, freshly cut fries and good ginger ale.

De Nieuwe Winkel

One of the best vegetarian restaurants in the Netherlands! Having dinner here was quite an experience. You can also eat non-vegetarian food here. Nice detail: in addition to a wine arrangement, they have a great alcohol-free-drinks arrangement.

Vesters

Stylish dining in a relaxed atmosphere. The menu is fixed and changes with the seasons: small menu, great quality food.

Take-away

Toon's Toon’s groente en fruit hoek (Houtstraat 33, Nijmegen)

This Turkish supermarket is located in the Houtstraat, opposite Café Samson. Ask for a "Broodje Alex". You'll get a quarter of Turkish bread, filled with all kinds of tasty salads for € 2.50.

Munt

A salad bar in a side street of the Molenstraat. You pay around a tenner for an epic portion of healthy greens.

WokForGo

Good Daging Smoor for cheap.

Shopping

Het Goed

This is a large secondhand store with a good range of clothing and retro furniture.

De Stikke Hezelstraat and Lange Hezelstraat

This street has a lot of nice boutique shops. It's particularly good for slow food lovers: go to Bakker Arend for fresh bread, Simon Levelt for coffee and tea, De Wit for cheese and butchery Sjek Floor for organic meat and charcuterie.

Klerengoed

This secondhand store is just outside the city center. The store has quite a chaotic vibe because of the mess, but do not underestimate what you can find here. They specialize in alternative women's clothing and also have a lot of second hand vinyl.

De Vinotheek

Tasty wines for every budget and a chat with the knowledgeable storekeeper Maarten.

Van Nature

Store with all sorts of green, organic and otherwise environment-friendly products, often directly from the producer. A good place to get presents, for yourself or for someone else.

De Waaghals

The record store of Nijmegen. Enter at your own peril. Whenever I go there, I tend to leave with a bag full of secondhand vinyl. They also have a very nice collection of second-hand vinyl in the basement.

Harry Potter and the Technical Twitter Meme

2019-11-22T00:00:00+00:00

NLP on social media messages

Today, Twitter user @iamdevloper started a new meme asking for Harry Potter titles with technical references in them. People started posting so many replies that a friend asked me whether I could find which ones where the best. This post is a write-up of my method. Click here to go straight to the results.

Tech Hogwarts.

Complete the title:

Harry Potter and the ________
— I Am Devloper (@iamdevloper) November 20, 2019

Method

I used the Twitter API (Python3 and python-twitter) to scrape 3203 tweets that are a reply to @iamdevloper. I filter specifically on replies to @iamdevloper newer than the original meme. I don't want to see retweets in my results.

These elements make up the raw Twitter search query:

@iamdevloper 
filter:replies
since_id:1197281108213293056
exclude:retweets 
-filter:nativeretweets

I filtered these posts to just include direct replies to the original meme. This left me with a dataset of 2733 tweets.

target = api.GetStatus(1197281108213293056) # the original meme
target = target.AsDict()
replies = [t for t in tweets 
            if "in_reply_to_status_id" in t.keys() and 
                t["in_reply_to_status_id"] == target["id"]]

We came up with two ways to determine "the best" Twitter posts in the series.

Rank 1: reach

We can sorted tweets by reach, i.e. the number of users that were (potentially) reached by a certain post. Each tweet is reached by the followers of the author, and the followers of every person that retweets it. I wasn't able to get the number of followers for every user that retweeted a tweet, so I decided to multiply the number of retweets by the median number of followers of all users in my dataset.

followers = [tweet["user"]["followers_count"] for tweet in tweets_ if "followers_count" in tweet["user"].keys()] 
followers += (len(tweets) - len(followers)) * [0] # users with 0 followers are ignored by the list comprehension
average_followers = sum(followers)/len(followers)
followers.sort()
median_followers = followers[math.floor(len(followers)/2)] #192 is median of followers_count

reach(tweet, median_followers):
    return tweet["user"]["followers_count"] + 
            tweet["retweet_count"] * median_followers

reach_statistics = [ (tweet, reach(tweet, 193)) for tweet in replies]
reach_statistics.sort(key=lambda x:x[1], reverse=True)
for i in range(10):
    print(reach_statistics[i][0]['full_text'], reach_statistics[i][1])

@iamdevloper Harry Potter and the Useless Use of Cat 185845
@iamdevloper Harry Potter and the Bashed Bin of Unix 131775
@iamdevloper Config of WebPack 74163
@iamdevloper haslayout 67400
@iamdevloper Harry Potter and the #HollowsHunter 41347
@iamdevloper Harry Potter and the Prisoner of Kanban 40799
@iamdevloper Harry Potter and the Image Pull Policy 35719
@iamdevloper Harry Potter and the Room of Unclear Requirements 28759
@iamdevloper Race Condition Harry Potter and the 26434
@iamdevloper hollow of vim. 25424

Rank 2: popularity

We can also sort tweets by popularity, i.e. calculate a popularity metric based on number of likes and number of retweets. I decided to count favorites with weight 1, and retweets with weight 3.

popularity(tweet, favorite_weight, retweet_weight):
    return tweet["favorite_count"] * favorite_weight + 
            tweet["retweet_count"] * retweet_weight

popularity_statistics = [ (tweet, popularity(tweet, 1, 3)) for tweet in replies]
popularity_statistics.sort(key=lambda x:x[1], reverse=True)
for i in range(10):
    print(popularity_statistics[i][0]['full_text'], popularity_statistics[i][1])

@iamdevloper Harry Potter and the Prisoner of Kanban 2233
@iamdevloper Harry Potter and the Room of Unclear Requirements 2030
@iamdevloper Race Condition Harry Potter and the 1936
@iamdevloper Harry Potter and the Prisoner of Vim 1417
@iamdevloper Chamber of NullPointerExceptions 854
@iamdevloper Harry potter and the unexpected ":" on line 43. 724
@iamdevloper szynszyliszys Harry Potter and the [object Object] 532
@iamdevloper Prisoner of jQuery 404
@iamdevloper Harry Potter and the ;SELECT * FROM users; 362
@iamdevloper HaaS (Hogwarts as a Service) 347

Prett(y|ier) printing

I liked the results of the popularity metric best. I sorted the tweets on their popularity and formatted the results in something a bit more readable.

First, I removed all usernames at the beginning of the tweet with a regular expression. Then I checked whether "Harry Potter" is already part of the tweet. If it's not, I prepend the tweet with "Harry Potter and the ". Unfortunately, this does not work perfectly, as some people inserted ampersands, abbreviations and other deviations from @iamdevlopers fill-in-the-blank template. There's a lot more I could do to make it prettier -- however, I'm just interested in getting the results in a readable format /fast/.

for i in range(50):
    booktitle = popularity_statistics[i][0]['full_text']
    # delete usernames at the start of the tweet
    booktitle, sub = re.subn(r"^(@[A-Za-z0-9_]+ )+\s*",r"",booktitle)
    if "Harry Potter" not in booktitle:
        print("Harry Potter and the", booktitle)
    else:
        print(booktitle)

Bonus: puns

So, which Harry Potter title is most suitable for making puns? Let's count how many tweets refer to each book in their text:

books = [
    "Harry Potter and the Philosopher's Stone", 
    'Harry Potter and the Chamber of Secrets', 
    'Harry Potter and the Prisoner of Azkaban', 
    'Harry Potter and the Goblet of Fire', 
    'Harry Potter and the Order of the Phoenix', 
    'Harry Potter and the Half-Blood Prince', 
    'Harry Potter and the Deathly Hallows']
keywords = [
    ['sorcerer','philosopher', 'stone'], 
    ['chamber', 'secret'], 
    ['prisoner', 'azkaban'], 
    ['goblet', 'fire'], 
    ['order', 'phoenix'], 
    ['blood', 'prince', 'half'], 
    ['deathly', 'hallows']]

bookcounts = {}
for book in books:
  bookcounts[book] = 0

for t in tweets_:
  for index, book in enumerate(books):
    for keyword in keywords[index]:
      if keyword in t["full_text"].lower():
        bookcounts[book] += 1
        break # count each book only once

for book in books:
  print(book, bookcounts[book])

Harry Potter and the Philosopher's Stone 83
Harry Potter and the Chamber of Secrets 178
Harry Potter and the Prisoner of Azkaban 131
Harry Potter and the Goblet of Fire 85
Harry Potter and the Order of the Phoenix 131
Harry Potter and the Half-Blood Prince 103
Harry Potter and the Deathly Hallows 94

Most tweets don't refer to a specific book at all, but Chamber of Secrets has the most puns.

So, which things are described as DEATHLY or SECRET by the replying punsters?

for t in tweets_:
  if "deathly" in t["full_text"].lower():
    print(re.search("deathly.*",t["full_text"].lower()).group())

    [ some omitted grep, tr, sort and uniq magic ]

      2 deathly deploy
      2 deathly exception
      2 deathly hallow
      2 deathly haskell
      2 deathly nullpointerexception
      3 deathly bug
      3 deathly ehlo
      3 deathly malloc
      4 deathly hello world

for t in tweets_:
  if "of secrets" in t["full_text"].lower():
    print(re.search("[^ ]* of secrets",t["full_text"].lower()).group())

    [ some omitted grep, tr, sort and uniq magic ]

      2 docker of secrets
      2 documentation of secrets
      2 repository of secrets
      2 vault of secrets
      3 container of secrets

Results: top 50 Tech Hogwarts titles

Harry Potter and the Prisoner of Kanban
Harry Potter and the Room of Unclear Requirements
Race Condition Harry Potter and the
Harry Potter and the Prisoner of Vim
Harry Potter and the Chamber of NullPointerExceptions
Harry potter and the unexpected ":" on line 43.
Harry Potter and the [object Object]
Harry Potter and the Prisoner of jQuery
Harry Potter and the ;SELECT * FROM users;
Harry Potter and the HaaS (Hogwarts as a Service)
Harry Potter and the Stack Trace of Secrets
Harry Potter and the Static site generator
Harry Potter and the Hotfix on Production
Harry Potter and the ________ // TODO
Harry Potter and the Order of the Arguments
Harry Potter and the Unexpected Identifier
Harry potter and the prisoner of the legacy system
Harry Potter and the Container of Secrets
Harry Potter and the Bubble-Sorting Hat
Harry Potter and the Unsecured AWS Container
Harry Potter and the harry potter and the harry potter and the 
    harry potter and the harry potter and the harry potter and 
    the harry potter and the harry potter and the harry potter 
    and the CTRL-C [as Voldermort: a while(true) loop]
Harry Potter and the Problem That Couldn't Possibly Be DNS.
Harry Potter and the Conflicts of Git
Harry Potter and the Passwords in Plain Text
Harry Potter and the Code that Worked on His Machine
Harry Potter and the Kernel of Panic
Harry Potter and the marketing manager who always blames site 
    speed for poor sales performance.
Harry Potter and the Full-Stack Prince
Harry Potter and the Infinite Loop
Harry Potter and the Chamber of Application Secrets
Harry Potter and the Order of the Phoenix Project
Harry Potter and the Half-Blood Print Statement
Harry Potter and the Deathly Malloc
Harry Potter and the Config of WebPack
Harry Potter and the [Object object]
Harry Potter and the Prisoner of Friday Deployments.
Harry Potter and the Goblet of Java
Harry Potter and the Chamber of Shared Secrets
Harry Potter and the Blue Screen of Deathly Hallows
Harry Potter and the Order of the Nginx
Harry Potter and the Missing Node Module
Harry Potter and the Useless Use of Cat
Harry Potter and the De-referenced Null Pointer
Harry Potter and the conflict of kubernetes.
Harry Potter and the facepalm of vulnerabilities.
Harry Potter and the Deathly Requirements.
Harry Potter and the Arrow of Lambda
Harry Potter and the Source on Your Phone.
Harry Potter and the Order of the Callbacks
Harry Potter and the Accidently Delete Master Branch
Harry Potter anD 7#!../'*zz)n3,+1j¥%£&lt;22e.-3;_)#;dj82;I✊$1@;
    Segmentation fault (core dumped)
Harry Potter and Order of Flex Items
Harry Potter and the Chamber of Segfaults.
Harry Potter and the Image Pull Policy
Harry Potter and the CHMOD of secrets.
Harry Potter and prisoner of ASCII-ban.

If you want even more, here's the (uncleaned) top 1000.

Beware of unescaped, unsafe, untrusted user input. ;)

Overview 2019

2020-01-07T00:00:00+00:00

Following some of my favorite bloggers, I've written an overview of 2019. It was a very good year: I spent a lot of time traveling, meeting new people and learning new stuff. I've noticed I'm slowly getting better in the things that were new in the first year of my PhD, such as writing scientific papers and formulating research questions.

Research

At the very beginning of 2019, I gave a talk at CLIN (a meeting of computational linguistics researchers that focus on the Dutch language) about an unfinished prototype of headline generator Churnalist. Here I also met Thomas Winters from KU Leuven, who ended up being my ICCC travel buddy later this year as we stayed at the same hotel.

I spend most of the winter months working on Churnalist and condensing everything I learned in my first full paper. The paper was accepted at ICCC 2019, so I got to travel to the US and meet the wonderful members of the computational creativity field. I wrote up my experiences in this blogpost.

To take a break from Churnalist, I took the month April off from my main research project so I could do a fun side project. I started analysing last year's NaNoGenMo projects because I was curious to see what approaches and data people use if they only have 30 days to create a story generator. By looking at other people's Github repositories (so many different languages!) I learned a lot about programming, procedural generation and low-cost story generation.

My notes about these projects turned into a second full paper, which was accepted at the StoryNLP workshop at ACL 2019 in Florence, Italy. Visiting ACL was an interesting experience because it was a HUGE conference, and only a tiny minority of the attendees were interested in creative text generation. I felt a bit lost in the beginning, but I had a great time in the end. I enjoyed the beautiful city of Florence (art! history! tiny cobblestone streets! icecream! truffle pasta!), met Dong Nguyen (who also did a PhD with my supervisor Mariët Theune) and found fellow story generation enthusiasts at the StoryNLP workshop.

In August, I visited COG 2019 to present a demo of my (now working!) prototype of Churnalist. People seemed to like my ICCC conference report, so I wrote another conference report about COG. A few weeks later, I published the code for Churnalist on Github.

Community

I was able to meet some of my favorite Twitter people In Real Life at ICCC, ACL and COG. It was wonderful to meet so many of you!

It feels great to be part of a research community where research, tech and art are so intertwined. Especially in the procedural content generation (PCG) and computational creativity (CC) communities, people tend to wear many hats: game developers also publish about their projects in scientific journals, and some researchers also produce games. A large part of the community is active on Twitter, or writes blogposts, tutorials, or open source code, or is involved in organising creative events like game jams. All this sharing makes it easy to find resources and learn new things!

Creativity

I had a lot of fun this year with all kinds of creative projects!

In March, I opened my t-shirt webshop with Spreadshirt, which means that people can now order my t-shirt designs online. I've created 5 new designs in 2019 and uploaded a bunch of my old designs (that were previously only for sale in the Netherlands). Spreadshirt prints t-shirts on demand and ships to all European countries, so it's now much easier to buy one of my t-shirts! I also like that everything on Spreadshirt is customizable, so if you want a mug or a hat or a sweater with one of my designs, you can!

I learned how to do 3D modeling in Blender by following this course from BlenderGuru.

In September, I started streaming on Twitch. On Thursday evenings I play old and new games, drink tea and chat with the stream audience. Since my graduation I haven't spent as much time playing video games as I would've liked, and thanks to my streaming schedule (and the contagious enthusiasm of my audience), I have finally played some games from my backlog & some nostalgic "oldies". So far I've played: Old Man's Journey, The Lion King (1994), House Flipper, Untitled Goose Game, Populous: The Beginning and Far From Noise. Besides these "Let's play" streams, I also streamed the design process of my Death Jazz t-shirt.

To set up a stream with decent quality, I had to learn how to work with broadcasting software OBS Studio, fiddle around with my audio settings, find suitable public domain music, read up on copyright laws and build some streaming layouts. I also tried out different open source video editing tools and had to learn how Twitch works. I hadn't realised how much time and effort is involved in running and producing and presenting a stream. It makes me appreciate well-produced live-shows like Critical Role and UnDeadWood even more.

I wrote 8 blogposts in 2019, which is a lot more than the 2 blogposts I wrote in 2018. Besides the conference reports, people seem to like the Twitter guide for researchers and the list of video game datasets.

I started an open source project in Python/Django called Josephine (named after Jo March from Little Women) for keeping track of the books I read. It was inspired by Hans de Zwart's booklist which is a bit like GoodReads, but completely self-hosted. Unfortunately, Hans' code is not open-source, so I decided to build my own version. If you like reading books and programming, feel free to contribute on Github.

The future

I'm half-way through my PhD! I've updated the chart I made earlier this year, which lists all my PhD projects and their research topics:

I think I have a clear idea of the general direction of my research, so it's time to start planning The Thesis (dun dun dunnnnn). I think I'm going to follow Austin Kleon's example by making a paper prototype. Then I can just flick through it and see where the "plot holes" are -- and start fixing them.

I'm currently working on evaluating Churnalist by asking people to judge Churnalist's outputs. That means learning new research methodology (setting up human subject research was not part of my computer science curriculum), working with the ethical committee of my department and learning how crowdsourcing platforms work. It's not going as fast as I had originally planned, but it's all very interesting (I've had fun collecting materials for the questionaire) and my colleagues are helping me fill the gaps in my knowledge. I'm trying to the take the best practices from various ICCC papers, and Ehud Reiter's blog also helps.

My next prototype will be a system that can generate fictional radio chatter. The Data2Game partners are working on a science fiction game and they were looking for procedurally generated messages for the comms station. Enter text generation! Currently there is a student working on this -- she's using NASA radio transcripts as source material -- and I will continue this project in the rest of 2020.

Another student is working on NLP (sentiment analysis) on Skyrim dialogues. Since I've been working on sentiment analysis on the side for some time, I hope we can turn her research project into something that is useful for game developers or games researchers.

I will continue streaming on Twitch, designing t-shirts and learning new things and then tweeting about them. I wish you all a creative 2020!

What NLP researchers can learn from NaNoGenMo

2020-01-24T00:00:00+00:00

Today was the Alice & Eve 2020 symposium at the University of Twente. I sent in an abstract for the poster session about last year's paper about story generation, in which I discuss 60 projects from NaNoGenMo 2018.

The audience of Alice & Eve's poster session was "people with a technical background", so a relatively general audience. I tried to write the abstract in such a way that anyone can understand it. I'm pretty happy with how it turned out, so now you can read it here. :)

NaNoGenMo is an online programming challenge where participants try to create a novel generator, i.e. software that can autonomously compose a piece of fiction, in 30 days. At the end of the challenge, participants have to share their code and an output text of at least 50,000 words.

Although some generators trivially write the word ‘woof’ 50,000 times or modify one of the classics of world literature, a minority of participants creates a generator that outputs something resembling logical original prose. Given that coherent text generation is an open problem in the Natural Language Processing research field, this is no mean feat.

We analyzed the code and output of 60 projects of NaNoGenMo 2018 to see how participants generated a coherent 50,000 word story, hoping that this would provide the NLP community with useful design patterns. We found that participants used four different approaches for coherent text generation: hardcoding a high-level text structure in the generator, simulation and emergent narrative, framing the output in a clever way, and faking coherence.

Story generators can be used to write new works of fiction, but also to create readable, informative texts that might normally not be accessible for a general audience. Think of texts that describe complicated expert knowledge or quantative data, like medical information for patients in a hospital, governmental policy documents, scientific findings or financial reports of companies. Design patterns like those found in NaNoGenMo projects can help us build better story generators, so that we can automatically create coherent, clear and engaging narratives.

If this abstract has made you curious about my findings, or story generation in general, you can find the open-access paper here. And below is the accompanying poster that I showed during the poster sessions at ACL 2019 and Alice & Eve 2020:

Course formats for research seminars

2020-01-28T00:00:00+00:00

During my master studies in computer science at Radboud University, I followed three research seminar courses: on software science, on mathematical foundations of computer science and on privacy. Normally, students would only do the one seminar course that had to do with their specialisation (i.e. software science, formal methods or computer security), but I chose to do all three as part of my free elective courses.

I like the format of seminar courses since students generally get a lot of freedom and responsibility during those courses. I also find that they are excellent practice for being a PhD student: I got to know the research staff better, practiced various research skills and got a more realistic view of what it means to do research (as opposed to doing scoped, guaranteed feasible homework).

All three seminars had different formats, and now that I am a PhD student (and occassionally help out with teaching at the University of Twente), it is helpful to have an overview of the potential formats for a seminar course, either for me or as inspiration for my colleagues.

I have tried to describe a lecture plan for each format. All courses are semester courses, i.e. two quarters of about 8 weeks of teaching.

If you get (too) many students for a course, you can do two things: (1) set a maximum number of students that can enroll (not always possible because of institutional policies) or (2) adjust the group size, and then adjust the workload by changing the lecture length and paper length where neccesary.

If you used one of these formats, I'd love to hear your experiences. If you have ideas for improvements, let me know in the comments. I hope these formats can be of some benefit to university teaching staff out there. :)

Option 1: part capita selecta, part research internship

Based on the course MFOCS research seminar (2014), half-year course, 6 EC

This course was given by a group of ~5 lecturers with the same research field. In the first half of the course, each lecturer would give a lecture about their specialisation, and describe it in the context of the larger research field. This is the capita selecta part of the course. In addition to these lectures, students would form groups of two and do a research project with one of the lecturers during the course. The research project was new, based on some ideas by the lecturer and not well-defined when we started. This is risky but rewarding, and can give students a realistic view on what it entails to do research. Good prep for students thinking of doing a PhD. For me and my project partner lead to our first published research paper.

Student groups would present twice: two weeks after starting (15 minute presentations), and near the end of the course (30+ minute presentations).

Lecture format (example):
1 introduction lecture
  lecturers present their topics
  deadline: students choose groups and supervisor (1 group per lecturer)
2 lecturer 1
3 lecturer 2
4 lecturer 3
5 lecturer 4
6 lecturer 5
7
8

1 student groups present their research plans (short presentations)
2 
3 
4 
5 student groups present their findings (long presentations)
6 student groups present their findings (long presentations)
7
8 deadline research project report

Option 2: lectures for and by students

Based on the course Privacy seminar (2015), half-year course, 6 EC

This course was given by two lecturers, who both had a different perspective on the course topic (privacy): one computer scientist, and one lawyer. They both gave a lecture at the start of the course, in which they gave an overview of the course topic from their respective specialisations. They took care to point the students towards useful resources (field-specific terminology, literature) needed for the project work of this course.

At the end of the first lecture, students make groups of 2 and choose a topic from a list provided by the lecturers. Based on this topic, they prepare a full lecture (2x 45 minutes) in which they teach their fellow students about this subtopic, as if they were a lecturer themselves. Each group also prepares a 10-page research report in which they summarize their findings and describe at least two different technologies that try to solve a particular problem in the area of their subtopic. This research report is not about doing original research, but writing a small survey of a particular area and summarising the most important problems and developments.

Tips for the student lectures

The student lectures can start directly after the first two introductory lectures given by the teachers. As a teacher, you can decide to give your students a bit more time to prepare before the first student lecture. You could skip a few lectures, or invite industry speakers to talk about the course topic. Do a "peer-review" in class after each student lecture. Ask students in the classroom to give compliments and points of improvements for the lecturing students.

Tips for the report

The report is due at the end of the course, but students would get a change to meet with the teachers to get feedback on the 'skeleton' (structure) of their report before handing it in. Additional meetings with one of teachers were possible in case of questions/problems, on an on-demand basis.

Lecture format (example):
1 introduction lecture
  lecturers present their specialisation
  deadline: students choose groups and topic
2 introduction lecture, lecturers present their specialisation
3 (guest lecture)
4 (guest lecture)
5
6
7 student lecture 1
8 student lecture 2

1 student lecture 3
2 student lecture 4
3 student lecture 5 
4 student lecture 6
5 student lecture 7 + deadline: research report skeleton
6 student lecture 8 
7
8 deadline: research project report

Option 3: focus on explaining and reviewing

Based on the course Software Science research seminar, 6 EC, half-year course

This course focuses on reading (influential) research papers, explaining them to others, and writing constructive reviews. Students write two long-form reviews of two influential research papers: one paper that is a 'classic' in a particular field, and one more modern paper (for example, a publication that has recently won a best paper award at a relevant conference). The lecturer provides a list of these publications at the beginning of the course, and students can choose which papers they would like to read. The lecturer then makes a one-to-one mapping from students to papers.

Student reviews consist of: a description of the problem that the author try to solve, how this problem is situated in the research field, the solutions that the authors propose, what kind of evidence they give for their solution, and an analysis of the writing style and argumentation. Reviews should be 4-6 pages. Besides the long-form reviews, students also read one of the two presented papers of that week, and write a short review (summary + analysis of at most 1 page) for that paper. In the second part of the course, students were asked to write a constructive "review" one of the student reviews from the first half of the course.

During the lectures of this course, students take turns in explaining "their" paper to their fellow students in a half-lecture of 45 minutes. Students in the classroom are encouraged to ask questions and give feedback.

Lecture format (example):
0 lecturer provides publication list, students send in their preferences
1 student presentation classical paper 1 + 2
2 student presentation classical paper 3 + 4
3 student presentation classical paper 5 + 6
4 student presentation classical paper 7 + 8
5 student presentation classical paper 9 + 10
6 student presentation classical paper 11 + 12
7
8 deadlines: first long-form review, 
  various short reviews of classical papers

1 student presentation new paper 1 + 2
2 student presentation new paper 3 + 4
3 student presentation new paper 5 + 6
4 student presentation new paper 7 + 8
5 student presentation new paper 9 + 10
6 student presentation new paper 11 + 12
7
8 deadlines: second long-form review, 
  various short reviews of new papers and/or review of student review

Interview: NaNoGenMo and coherence

2020-02-09T00:00:00+00:00

A few months ago I was interviewed by Wired about NaNoGenMo 2019, an online programming challenge where participants try to build a novel-generator in 30 days. The interview was part of the background research for this Wired article.

I figured some people might be interested in this interview too, so I decided to put the questions and their answers up on my blog. I've rewritten some answers of my answers to clarify them -- the reporter who interviewed me had read my NaNoGenMo paper, but perhaps you haven't. :)

Why the focus on coherence in your text generation research?

I focused on generating coherent texts, because that is an open problem within natural language generation (NLG) research.

It is easy to let a computer generate loads of gibberish, but quite hard to generate a story that looks "natural", as if it was written by a human. The longer the generated text, the harder it is to get a coherent whole. With current research, we can generate coherent sentences, and if we put in some effort, coherent paragraphs.

(Note that this was written before the release of GPT-2 and GPT-3. However, even these models don't solve the coherence problem completely: GPT-2 and GPT-3 work primarily for the English language, and their language models are hard to copy since you would need a bigger dataset and more computing power than most people have access to.)

Examples of things that can go wrong during text generation:

the text generator loses track of the topic it's discussing
the generator gets stuck in a loop and generates the same phrase over and over again
the generators forgets to mention an important piece of information, confusing the reader All these things can completely derail the flow of text.

Some people solve this coherence problem by giving the text generator software a text plan: a structure, a storyline, or a checklist of information that should be mentioned. However, someone (or something) needs to write that text plan, so with a text structure you don't solve the problem, but you move it elsewhere.

I try to solve this problem differently. My preferred solution is creating a text generator that can make readers believe they are reading coherent texts -- even if the texts are not coherent at all! In other words, I'm more interested in perceived coherence than actual coherence.

I got the idea by looking at the text generators of NaNoGenMo 2018. We humans are pattern matchers, and we are biased towards finding meaningful patterns even in random data. Some NaNoGenMo participants used this to their advance by tricking the reader and playing with their expectations.

Of course, "tricking the reader" is only a good strategy when you're generating creative texts, or fiction, not for informative texts. My research is about generating texts for in video games (mostly RPGs). The goals for those kind of texts are fundamentally different from goals of newspaper articles or weather reports. Factualness is less important than entertainment value, aesthetics, and fit with the game narrative.

From your paper, it sounds as though the novel length and form really pushes the boundaries on coherence. I wonder if you found most people were aiming for coherence, or for other attributes?

Given the projects I've seen, I think coherence was not an important requirement to the majority of the participants. I think most people are just trying to have fun and create something cool in one month. ;)

Whether coherence is a goal depends on the output format chosen by the participants. The NaNoGenMo challenge is defined as 'create a text generator that can output a text of 50,000 words', so the output format can be anything. The novel format is not enforced by the organisers of the challenge in any way, which is part of the fun. Some participants end up generating 50,000 words of word art, poems or "lists". For word art, the visual aspect is more important than coherent. Poems and lists are microtexts, so they should be coherent at the word-level and paragraph-level, which is less challenging than generating one coherent 50,000 word text.

It also depends heavily on the objective of the person participating in NaNoGenMo: some people (like me) use it to learn about new techniques, others are trying to create procedural art. Some participants just like to be surprised by the output of their generator, and some people use text generation to circumvent the effort of writing 50,000 words by hand. These are all really different goals, and require different levels of coherence.

Do you have a sense of whether some of the newer methods, such as large language models, might make coherence easier to obtain? Or are other methods, such as hard-coding narrative structure, still the best bet?

Using a large language model can definitely save you some time when developing a generator, as basic properties of the language are already embedded in the model. This means the programmer of the generator can focus on writing a specification of the /type/ of output (what is a good dialogue, poem, story, ...), instead of also having to define of the language itself (what is proper grammar, what is a sentence, what are valid words, where do we put an exclamation mark).

However, large language models are not the end of all our problems. ;)

Two problems that I regularly run into in my own research: New methods generally work only with large amounts of data. For example, when we want to train a machine learning algorithm to do something for us, we need a large dataset to train it on.

there are limited resources (datasets, opensource libraries) available for languages other than English, Spanish, Chinese, ...
there are limited resources available for specific application domains. For example: if we want to generate text that should convey some kind of emotion or opinion, we need information about how this is expressed in text. However, most of the datasets available with this type of text contain product reviews. Product review texts do contain emotions and opinions, but a different dataset, such as one containing texts about emotional memories or personal experiences, might be more suitable for our goal. Ehud Reiter has written a good blog post about a similar problem: https://ehudreiter.com/2019/08/01/do-we-encourage-inappropriate-data-sets/

New NLG techniques are a big improvement, but even when you use them, coherence is still an open problem. From what I've seen, even new approaches, like those using deep learning, often display the same problems with coherence that I've mentioned above, especially in longer texts.

For coherence of fictitious texts: how you use the technologies is more important than which technologies you use. Humans WANT to make sense of the stuff they read. We are natural pattern-searchers. We can use that when we generate texts, especially novels and other works or fiction. If you can use people's goodwill towards a automatically generated text, towards a computer, that's more important than presenting factual statements in the right order (which is essential when you're generating informative, non-fiction text!). Think about the ELIZA-effect described by Douglas Hofstadter: people will attribute more intelligence and purpose to a computer program than is warranted by its code. If you present people with computer-generated gibberish and tell them the "author" was drunk when they wrote it, or that the author has gone mad, people start actively looking for meaning in the text, and don't mind so much the grammar mistakes or unfinished sentences.

Have you tried your hand at NaNoGenMo? If so, were you aiming for coherence, the semblance of it, or something else?

I'm planning to participate this year for the first time! Together with my colleague Lorenzo Gatti I'm going to do a science-fiction themed project, as next year I'll be working on generating text for a new science fiction game. NaNoGenMo is a perfect playground for trying out new techniques, so I'm going to play around with GPT-2. :)

A comparison of GPT-2 and BERT

2020-08-04T00:00:00+00:00

GPT-2 and BERT are two methods for creating language models, based on neural networks and deep learning. GPT-2 and BERT are fairly young, but they are 'state-of-the-art', which means they beat almost every other method in the natural language processing field.

GPT-2 and BERT are extra useable because they come with a set of pre-trained language models, which anyone can download and use. Pre-trained models have as main advantage that user don't have to train a language model from scratch, which is computationally expensive and requires a huge dataset. Instead, you can take a smaller dataset and "fine-tune" the large, pre-trained model to your specific dataset with a bit of additional training, which is much cheaper.

For my next research project, I want to use one of the "state of the art" deep learning approaches to text generation, so I read up on GPT-2 and BERT and created this overview. I hope it is helpful to other people on the internet!

Note that I just compared GPT-2 and BERT for the two languages that are relevant to my research (i.e. Dutch and English). Especially BERT has spawned a long list of alternative, spin-off models. Before you decide on a language model for your particular problem, read up on the strengths and weaknesses of a language model for specific tasks!

Quick decision table

I would use the following approach for specific task/language combinations, based on what I read about GPT-2 and BERT so far:

Task	English	Dutch
Natural language processing (NLP)	BERT	BERT (one of the BERT models for Dutch)
Natural language generation (NLG)	GPT-2	It depends: If input and output are not sensitive data: Combine GPT-2 (English) with GoogleTranslate API (Dutch to English and vice versa) If you only need short texts (15-token sentences) and you have bi-directional prompts: BERT (one of the BERT models for Dutch)

A very quick introduction to GPT-2 and BERT

GPT-2

Useful general information about this approach can be found in OpenAI's README and the model card for GPT-2
GPT-2 is only partially open source. The training dataset (called 'WebText') is proprietary and not published (although there are on-going discussions about this on GitHub). The opensource community is trying to create an opensource version of the training dataset, such as this initiative of Brown University researchers and this project.
All sizes pre-trained models are freely available. The biggest model was at first kept closed source for fear of abuse.
Models can be finetuned with smaller datasets for particular generation purposes.
It is only avaible for English. As far as I know, no model is available for Dutch, or other languages. Some people are trying to train a model for other languages than English, but lack of similar-sized heterogeneous training sets remains a practical problem:
Is widely being used for experimental natural language generation, ranging from hobby projects to exploratory research. Particularly cool examples: Procedural Magic cards, AI dungeon game
Is supported by Transformers, the biggest opensource library for AI/ML/NLP in Python.
There is lots of information available on this model: example code, colabs, tutorials, spin-off opensource projects, etc.
It is easy to use, even for students. I have seen multiple successful study projects of my students (for the courses "NLP", "advanced NLP", bachelor theses, etc) in the past two years. It's great for letting students experience the entire NLP research pipeline in a short time span: GPT-2 allows them to create a text generator, generate some outputs and evaluate them with humans in a couple of weeks.
Outputs of smaller models tend to converge on repetitive language, loops, etc during generation. If output quality is important, you need to either use a bigger language model (larger computation cost), or take some time to do automated output cleaning, or manual inspection and selection of artifacts (i.e. human-computer co-creation).
Can be used in combination with "field-specific tokens" to control the flow/coherence/structure of output text. This is best explained as using xml-tags in your training data and output prompt, to guide (force) the generator to generate a specific data structure. See for example the method used in this paper (page 3).
The next version, GPT-3, has a similar architecture as GPT-2 but is even larger in size.
GPT-3 is commercially available (not open source!) via an API. The authors have written a paper and published some research data, but training data and models are not available.
Demo API access is available for researchers ("private beta"), but there's a long waiting list as of June 2020. More media coverage about GPT-3: The Next Web, New York Times and The Verge.

BERT

BERT stands for "Bidirectional Encoder Representations from Transformers", and "is a new method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks."
There is an academic paper available on arxiv, in which the creators explain the theoretical details. The same paper was later published in NAACL conference of ACL.
BERT was created by Google researchers, and it seems to be fully open source.
Fun fact: the name BERT fits into the long-standing NLP tradition of naming systems after Muppets.
All different sizes pre-trained models are freely available via Github. Sizes range from Tiny to Large.
There are a number of smaller BERT models (tiny, mini, small, etc), which are intended for environments with restricted computational resources. Paper related to the smaller models: Well-Read Students Learn Better: On the Importance of Pre-training Compact Models.
Just as with GPT-2, models can be finetuned with smaller datasets for particular purposes.
Some of the models are integrated in HuggingFace's Python library Transformers.
Researchers and users are encouraged to share new models and performance comparisons online. BERT has a very active open source community, which seems to be mostly driven by the Transformers library of HuggingFace. BERT has various spin-off models, such as RoBERTa (from Facebook) and DistilBert (by HuggingFace). Some examples can be found in the 'supported models' list of HuggingFace: https://github.com/huggingface/transformers#model-architectures
Contrary to GPT-2, BERT has models for various languages: both monolingual models (English, Dutch, etc) and multilingual models. Multilingual models do not seem to perform as well as monolingual ones, possibly due to undertraining on the various included languages. This is a nice comparison paper for monolingual vs multilingual BERT-models, focused on Nordic languages: Is Multilingual BERT Fluent in Language Generation? by Rönnqvist et al. (2019).
Performs well on NLP tasks, even compared to GPT-2.
BERT was originally not meant for language generation, but Wang and Cho (2019) published a series of 'hacks' that uses BERT's training method (cloze test + follow-up sentence test) for generation. Relevant paper: BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model by Wang and Cho (2019).
Because of the deep bi-directionality of the neural architecture, BERT seems to perform best for text generation if there is a prompt prepended AND appended to each output sequence. If I recall correctly, GPT-2 is also bidirectional, but only shallowly, which is why it can perform better than BERT on 'standard' left-to-right text generation.
There are at least two BERT models for Dutch: BERTje (De Vries et al.) and RobBERT (Delobelle et al.), and there might be more.

It seems that if you want normal left-to-right generation in English, GPT-2 is still the best way to go. BERT's main strength is NLP tasks, and the variety of languages for which a pre-trained model is available.

If you've used this overview to help you choose a language model, let me know in the comments. I'd love to hear about your NLP/NLG projects.

My data-driven investment toolkit

2020-08-18T00:00:00+00:00

This is the first post in a series about data-driven investing.

In this post I describe a software toolkit called hetty, named after Hetty Green, an American investor and contemporary of other famous investors like Andrew Carnegie, Henry Ford, John R. Rockefeller and J.P. Morgan. Because I mostly follow a value investing philosophy, hetty consists of a set of scripts for gathering, analyzing and monitoring fundamental financial data. Although I will not publicly share the scripts, I hope the description of the problems I tried to solve might give you some inspiration for implementing your own investing toolkits.

This post will not contain information about specific companies, or recommendations for securities. I'm not a licensed investment banker, which means I'm not qualified to give Serious Advice about these matters. I also believe it's not necessary to recommend any specific stocks. As Warren Buffett pointed out in his essay about coin-flipping orangutans: the most succesful investors in the world pick vastly different stocks -- there are many valid strategies for investing, and there is not one strategy that fits all investors. Instead, do your own research and invest in what you know.

Stock prices

The first script I wrote was the one that I still use most often: it scrapes the current price of a specific stock, currency, fund or commodity. I say 'current price', but generally the price is delayed by 15 minutes. Of course I can get this information from my broker, with the added benefit of realtime price updates. However, for a value investor, knowing the "realtime" price for a stock is not essential at all. If a stock is a good deal at €5,30, it's probably also a good deal at €5,36. I'm not a high-frequency trader. If I need to buy a stock now for my investment strategy to work, it's probably not a very good strategy to begin with.

Getting stock prices with a script has two big advantages.

The first advantage is that I can get prices in bulk. In the web interface provided by my broker, I can get one stock price per search. Getting a list of current prices, for say, 10, 100 or 1000 stocks would take too much time (and clicks and typing). When you have a script, getting prices for 600 companies is as much hassle as getting the price for one company.

The second big advantage is that I can spend less time on my broker's platform. The broker's business model is to make money off transaction costs. In other words, it's in the broker's best interest if I buy and and sell as much stocks as possible using their platform -- that's how they make money. Consequently, their platform is engineered to make buying and selling as tempting as possible. Flashing stock prices and bold headlines give users a sense of urgency, and pretty colours and big buttons give the impression of a video game or slot machine. Value investor and writer Guy Spiers called Bloomberg Terminals "the informational equivalent of smoking crack cocaine" -- and that's also how internet brokers design their web interfaces. Banks and their UI designers have made it almost trivial to spend large sums of money in the stock market. Good for brokers, but generally not for investors.

Alerts

Once I could collect stock prices, I built a simple alerting system that sends me an email whenever a stock has reached a certain price.

I do a lot of research to find companies that I would like to invest in. However, sometimes the stock price is on the high side. In that case, I set an alert for a lower stock price, so I get an email whenever the stock price has reached a point where buying that company is a good deal according to my investing strategy.

If I already own a stock, I can use alerts to notify me of a pre-determined sell-condition. For example, if the company is an asset play, I can set the net asset value per share as the target share price.

I imagine many investors will find this useful functionality. You can set a target price for your favorite stocks and walk away, instead of staring at your favorite stock quote website for hours on end. I was really surprised that my broker does not offer anything like this -- but maybe bankers reserve this kind of functionality for their premium clients?

Monthly overview

Generally, I buy stock once a month, for a fixed amount of money. (This is called Dollar Cost Averaging, even though I trade in euros.) I do this for three reasons:

Investing is a form of saving (for me), so setting a fixed amount each month means I won't invest too little.
Investing is a hobby, so setting a fixed amount means I won't spend more money than I planned.
I want to mitigate timing risk: if I would invest all my money in one go, I would risk losing a lot of money if that moment turns out to be a peak (high point, local optimum) in market history.

Because I buy stocks once a month, I figured it would be useful to get a monthly report on companies and stock prices to help me decide which company I'm going to buy. I like a contrarian investment style, i.e. I like to buy those stocks about which the general public is feeling very pessimistic. These stocks are generally sold for huge discounts to what the underlying company is worth ("fair value").

In March 2020, Morningstar.nl posted an article in which they discuss how the corona-virus market crash in March 2020 influenced stock prices across different sectors. If you can't read Dutch, GoogleTranslate can translate the article for you.

One of the graphs from the article shows the price-to-fair-value ratio over time:

You can clearly see a huge dip around March 2020, when almost all companies were selling at a discount.

The other graph shows which sectors were selling at a discount:

Even without a market crash in recent history, there are always a few sectors that are very popular and a few that most people try to avoid. I figured that this information could be used to inspire my stock choices each month.

Unfortunately, Morning Star's Fair Value calculations are only available for paying customers, and even then the metric is not fully transparent. However, I can still use the data visualisation method without using the metric, if I use an alternative metric like price-to-earnings ratio (P/E ratio) instead. I would never make investment decisions based on price-to-earnings ratios alone, but they can be used to gain insight into the popularity of specific sectors.

I wrote a script that, given a list of companies, collects the P/E ratio and sector information, and compiles an overview of the stock market per sector. For each sector, the overview contains the average and median P/E ratio, and a list of companies with a P/E below the average. Generally, I tend to pick one undervalued sector, and then research the companies within those sector to see if I can find an undervalued but high-quality company.

This is a snippet of my monthly overview email for May 2020, which shows the statistics for the Automotive sector. The crazy high average P/E is, of course, because of outlier Tesla (TSLA), which currently has a P/E of 950.

Automotive
---------------------------
Company name                         ticker     quote       P/E
---------------------------------------------------------------
Fiat Chrysler Automobiles NV         FCA:IM      8.02      3.20
Peugeot SA                            UG:FP     13.08      3.65
General Motors Co                     GM:US     22.29      4.62
Volkswagen AG                       VOW3:GR    127.98      5.93
CIE Automotive SA                    CIE:SM     15.99      7.20
Bayerische Motoren Werke AG          BMW:GR     54.00      7.30
Pirelli & C SpA                     PIRC:IM      3.55      8.10
Cie Generale des Etablissement        ML:FP     89.10      9.19
Ford Motor Co                          F:US      5.09      9.72

Sector average P/E Ratio                                 323.69

Other stuff

Besides the stuff described above, I've written a framework for trading bots and a framework for backtesting, so that I can try out new investing strategies without actually buying stocks. I've also created a dashboard in Prometheus and Grafana that shows a collection of macro-economic indicators, which I use to gauge investor/consumer/industry sentiment before buying. I'm planning on writing posts about those in the future.

If you want to discuss any of this, please let me know! I don't know many people interested in investing and stock picking in real life, so I'd love to hear from you.

Financial data sources for individual investors

2020-09-20T00:00:00+00:00

This is the second post in a series about data-driven investing.

In the first post I described Hetty, a collection of tools for monitoring stocks and doing financial data analysis. Naturally, Hetty needs a steady inflow of financial information, and I prefer not to input that information by hand, so that's where online data sources come in.

In this post I want to discuss the sources for financial data that I reviewed as potential input sources for Hetty. I found many questions on Quora and Medium posts about this, but most of the "review articles" that discuss data brokers seem to be written by developer evangelists (ie. marketing people) of the data brokers themselves. Hmm.

Data wishlist

low-cost or free. If I'm investing $300 each month, I don't want my financial data to cost a significant percentage of that amount.
API with good documentation
data in a non-proprietary format such as JSON, CSV
allow enough requests to do analyses on a stock universe of 500-1500 stocks
financial information for European companies, not just US companies
high data accuracy. After all, I might base investment decisions on it.

Spoiler: I did not find a service that has all these properties.

Quandl

Quandl is a marketplace for data. Third parties can use it to sell subscriptions to financial datasets. Quandl seems to target institutional investors, but they offer a set of 'core financial data' for individual investors.

The platform was clearly built with automation in mind. Upon registration, you immediately receive an API key for interacting with the platform. Quandl also provides sample data for most of their datasets, which is a handy feature for developers.

However, Quandl is not for me. Even very basic information can only be found in 'premium' datasets, which cost about $30 dollar per month per dataset. This seems a bit steep since you can get the same data for free from Yahoo Finance. For example, a dataset with End of Day stock prices for US equities was $39,99 a month for personal use, or $299 anually. Since I'd need to combine various datasets to get information for all the (European and American) companies I invest in, this would quickly become too expensive. It's also a shame that you need to sign up for a Quandl account to just see the pricing for each dataset.

Alpha Vantage

Alpha Vantage is financial data broker startup. They cater to individual investors that need financial data in open formats such as JSON and CSV. Their website specifically mentions the cryptocurrency and foreign exchange (FX) crowd, two areas of investment where many people do algorithmic trading. The standard API is free, and limited to 5 requests per minute and 500 requests per day. This would not be enough for Hetty's large-scale stock screening, which I run once a month. If you want more requests per minute an no daily limit, you need to buy a subscription to their premium API, which starts at $30 per month.

Unfortunately, I got the feeling that Alpha Vantage's back-end is not completely ready yet. At first, the website kept going offline, so I couldn't get an API key. When I finally got some data out of the API, the stock prices contained many outliers: weird gaps and spikes in the data that did not reflect the real-world stock price movement.

If Alpha Vantage's technology becomes a bit more stable, I might revisit it later to see if it fits my need, especially since Yahoo Finance is scheduled to disappear in the future. In terms of product offering, Alpha Vantage came closest to being a solution for my problem -- only I don't trust the data quality yet. If I am going to pay for a subscription to financial data, I want to be 100% sure of the data quality.

Atom.finance

Atom.finance is another financial data startup, which offers a platform for tracking your investment portfolio, a stock screener, a social network for investors, etc. Unfortunately, they currently do not offer any information about securities traded in European markets, and they also don't have an API.

I registered to view the platform, but I quickly found out that Atom was not what I was looking for. Their user management back-end still needs a bit of work. After I deleted my account, I kept receiving marketing emails with an unsubscribe link to a non-existing user account. Their customer support fixed it for me manually after I sent them an email.

Bloomberg

Bloomberg is a big name in the financial world since they also make Bloomberg Terminals, which are used by professional traders for getting financial information. They offer a clear, minimalist website with financial news and general information about equities. The website even has up-to-date information for many European companies.

I like that they report earnings for each company in a consistent format, unlike the Yahoo website, where the reported financials can be very different across companies. Unfortunately I ran into problems when I wanted to collect company information in bulk. The Bloomberg website has a very strict anti-bot-policy, which makes it impossible to scrape, and access to the Bloomberg API probably does not fit my budget.

Yahoo Finance

Yahoo Finance is a great source of information for me. I've read multiple reports that the service will disappear in the future, but so far, it doesn't look like that will happen any time soon.

Apparently, Yahoo Finance used to have an API, but it's gone now (or only available for premium members). However it doesn't matter that the API is gone, as I can grab all the data I need from the JSON data embedded in each webpage! The Yahoo Finance website has no strict anti-scraping measures, unlike Bloomberg, so I can easily scrape and compare data for 1000 companies in one go. As an added bonus, Yahoo allows users to download historical stock prices for each company. Just go to a specific equity, go to 'historical prices', and click 'Download'.

As for accuracy, stock prices, market information and general company information are high quality for every listed equity. If you also want recent fundamental data (like balance sheets and statements of income), the data quality differs per company. The data for companies traded on a US market is generally high quality, but the information might be incomplete, inaccurate or missing for companies outside the US.

Conclusion

So far, Yahoo Finance is my best source of information outside of official company filings. I only use it for up-to-date stock prices, as fundamental information is not accurate for European companies.

I have not found a service yet that has all the properties I mentioned in my data wishlist above. I'd love to know about any alternatives to the websites I've discussed in this post, so if you know another data source that I should investigate, let me know!

Computer security resources for self-study and career planning

2020-09-27T00:00:00+00:00

This is a list of self-study resources for people that want to get into the computer security field. I started this list in december 2015, just before I started my job as a security specialist at the National Cyber Security Centre of the Netherlands. It contains resources for learning new techniques and tools, self-study, career planning and personal development. All of these were useful to me personally, and every link is hand-picked. ;)

As you can see, computer security is a huge field. Please don't feel overwhelmed. If you don't know where to start, check out the resources listed under 'career'.

Most resources are in English and some of them are in Dutch.

I hope this list is helpful. If you know any good additions, feel free to let me know via Twitter, email or in real life.

Assembly

Microcorruption, Matasano [online challenges]
TIS-100, Zachtronics [game]
Assembly 101, FriedSpace [website]

Capture The Flag

PicoCTF, Carnegie Mellon University [online challenges]
Microcorruption, Matasano [online challenges]
OverTheWire, [online challenges]
Certified Secure, Certified Secure [online challenges] (Dutch)

Capture The Flag (Beginners)

PicoCTF, Carnegie Mellon University [online challenges]
OverTheWire, [online challenges]
Certified Secure, Certified Secure [online challenges] (Dutch)

Career

How To Become an Infosec Expert, Part I, Alex Kirk, Talos Intel [book list]
Don't Call Yourself A Programmer, And Other Career Advice, Patrick McKenzie, Kalzumeus [article]
CTF Field Guide: Career Cheatsheet, Trail of Bits [article]
How to Milk a Computer Science Education for Offensive Security Skills, Raphael Mudge [article]
Google careers: Guide to technical development, Google [article]
Starting an InfoSec Career - chapters 1-7, Lesley Carhart [article]

Cryptography

Cryptopals, Matasano [programming exercises]
The Code Book, Simon Singh [book, non-fiction]
Nice introduction to secret language and cryptography for a general audience.
Cryptography I, Stanford University [online course]

Cybercrime

Fraudehelpdesk, [organisation] (Dutch)

Cybersecurity In The Nederlands

Cybersecuritybeeld NL 2020, Nationaal Cyber Security Centrum [publication] (Dutch)

Dark Web

The Dark Net, Jamie Bartlett [book, non-fiction]

Ethical Hacking

Helpende hackers, Chris van 't Hof [book, non-fiction] (Dutch)
Coordinated vulnerability disclosure (CVD): the guide, Nationaal Cyber Security Centrum [publication]
A guide for organizations and ethical hackers for the responsible reporting and handling of vulnerabilities in information systems and (software) products.

Fraud

Fraudehelpdesk, [organisation] (Dutch)
Catch me if you can, Steven Spielberg [movie]

Incident Response

A Soft-ish Introduction to Malware Analysis for Incident Responders, Greg Carson [article]

Information Security

Information security for journalists, The Centre for Investigative Journalism [publication]
Security Engineering, Ross Anderson [book, non-fiction]

Internet Freedom

Bits of freedom, [organisation] (Dutch)
Little brother, Cory Doctorow [book, novel]

Linux

OverTheWire, [online challenges]
How Linux Works: What every superuser should know, Brian Ward [book, non-fiction, technical]

Lockpicking

Toool: The Open Organisation Of Lockpickers, [organisation]

Malware Analysis

How to Get Started With Malware Analysis, Lenny Zeltser, SANS [article]
Practical malware analysis: The Hands-On Guide to Dissecting Malicious Software, Michael Sikorski, Andrew Honig [book, non-fiction, technical]
A Soft-ish Introduction to Malware Analysis for Incident Responders, Greg Carson [article]
REMnux, Lenny Zeltser & David Westcott [operating system/toolkit]

Penetration Testing

How to Milk a Computer Science Education for Offensive Security Skills, Raphael Mudge [article]
SEC560: Network Penetration Testing and Ethical Hacking, SANS [training course]
Helpende hackers, Chris van 't Hof [book, non-fiction] (Dutch)

Physical Security

Toool: The Open Organisation Of Lockpickers, [organisation]

Privacy

Information security for journalists, The Centre for Investigative Journalism [publication]
Bits of freedom, [organisation] (Dutch)
Little brother, Cory Doctorow [book, novel]
Je hebt wél iets te verbergen, Maurits Martijn & Dimitri Tokmetzis [book, non-fiction] (Dutch)

Programming

TIS-100, Zachtronics [game]
Cryptopals, Matasano [programming exercises]

Ransomware

Fraudehelpdesk, [organisation] (Dutch)
No More Ransom, Europol, Kaspersky, Politie NL, e.a. [website]
Initiative of Europol, Kaspersky Lab and the Dutch police to collect decryption keys for ransomware and put them online. Victims of ransomware can check this website to see if it is possible to undo the encryption.

Responsible Disclosure

Helpende hackers, Chris van 't Hof [book, non-fiction] (Dutch)
Coordinated vulnerability disclosure (CVD): the guide, Nationaal Cyber Security Centrum [publication]
A guide for organizations and ethical hackers for the responsible reporting and handling of vulnerabilities in information systems and (software) products.

Reverse Engineering

REMnux, Lenny Zeltser & David Westcott [operating system/toolkit]

Social Engineering

Catch me if you can, Steven Spielberg [movie]

Wargames

PicoCTF, Carnegie Mellon University [online challenges]
Microcorruption, Matasano [online challenges]
OverTheWire, [online challenges]
Certified Secure, Certified Secure [online challenges] (Dutch)

This list was last updated on 2020-09-27

Talk transcript: Fantastic strings and where to find them

2020-10-19T00:00:00+00:00

This is a textual version of the talk I gave at the Intelligent Narrative Technologies workshop at AIIDE 2020, for the research paper titled Fantastic Strings and Where to Find Them. The talk was transcribed using Whisper.cpp and co-edited with the open beta of ChatGPT to improve the flow of the text.

Intro

Hi everyone, thank you for tuning in to my talk. My name is Judith van Stegeren and I collected three new datasets with video game text.

But beyond just discussing the three datasets, I also want to address the underlying problem. Why is it so challenging for researchers to create new video game text corpora? And what are some possible approaches that you could take if you want to create a new dataset?

I am a PhD student at the University of Twente and my research focuses on language generation for video games. There are many different types of texts in video games, but the ones that I am most interested in are in-game dialogue, game narratives, and flavor text for RPG games. Flavor text is the decorative text that is not essential for gameplay or completing the game, but adds more depth to the game world.

Video game datasets

As I began my research, I was curious to learn what other datasets were being used by researchers in the field. Personally, I read a lot of papers from the procedural content generation and natural language processing fields. And I found that most researchers tend to work with texts from other domains, even if they list video games as their intended application domain. For example, researchers working on story generation use story corpora, and those working on dialogue generation use dialogue corpora. However, these corpora are typically sourced from domains outside of video games.

One exception is Skyrim. Many papers use data from Skyrim because it comes with its own modding software, which is relatively user-friendly. This makes me think that we should encourage game developers to publish official modding software, as it can be very helpful for researchers.

Game writing is different from other writing

Additionally, I started reading about how game writers work in the industry, as I believe this could provide useful inspiration for creating my own text generators. I came across a great book by Chris Bateman in which the authors argue that game writing is fundamentally different from other types of writing and fiction writing.

Firstly, there is the interactive aspect of games to consider. But it's also because of the game development process. Game writers have to be able to adapt to the sometimes unpredictable development process, as games can change stories, genres, or styles within a few months. Game writers need to be flexible enough to handle this. Learning all of this made me realize that if I want to analyze and generate new video game texts, I want to work with source data from the video game domain.

Creating new datasets

However, I learned that there are not many available datasets with video game texts. So I had to create my own. But that turned out to be quite difficult. Ideally, I think the best data can be found directly in game files. But extracting those strings from an installed game is not straightforward. The strings might be spread across many different files, and the game engine reassembles them into what you see on screen during gameplay. Extracting this data is not easy, even for researchers with a computer science degree. You need to understand how the game works at a low level to access this data.

I talked to a game developer friend of mine and he gave me some ideas. I also read about how others have solved this problem in scientific papers.

I eventually found three possible approaches that I wanted to investigate further. These are ordered from easiest to most complex.

The first option is to build a web scraper and search the internet for fan-made websites with in-game texts. There are many places on the internet where you can find transcriptions of quest texts, dialogue, cutscenes, and other in-game texts.
The second option is to use official modding software, if it is available for the game. Not all games come with their own software, but those that do are generally user-friendly and well-documented. Since the publisher released the software, they often have a vested interest in its success, making it a good option to consider.
The final option is to search online for fan-made tooling for modification or extraction of game assets on websites like Nexus Mods. This could also be a viable option.

Pros and cons

I tried out all these approaches in practice and they led to three different datasets. However, each has its own pros and cons, so depending on your needs and technical proficiency, as well as whether the publisher has released anything, you can choose one of these methods.

By the way, unfortunately, none of these methods are scalable, which is a disadvantage. You can build one extraction tool for one game, but it is unlikely that you can use it for another game, with the exception of games from the same publisher.

The first option, using wikis, is the easiest to use. However, as with Wikipedia, the data is crowd-sourced and may require extra cleaning, have typos, or be incomplete. The data available for different games is also subject to the fan community, so it can vary.

The second method is my favorite, but only a minority of games come with publisher-sanctioned modding software.

The third method, using fan-made tools, is for adventurers only. It requires installing a lot of undocumented or old tools and seeing if they can do what you need. However, the hard work paid off in the end, as it led to some high-quality data for my final dataset.

Dataset 1: Elder Scrolls

For my first dataset, I scraped all of the in-game books from the RPGs in the Elder Scrolls series from a fan wiki. For the second dataset, I used the official Torchlight 2 modding software, called Guts, to extract all the strings from that game. And for the third dataset, I used a collection of open-source fan-made tools to extract all the branching dialogue from Star Wars Knights of the Old Republic 1.

So, what does in-game books from the Elder Scrolls mean? The Elder Scrolls is a series of action RPGs, and the developers use in-game books to convey more information about the in-game world to the player. The player character can walk around the world and find books in shops and houses, and can read every book they encounter.

The advantage of this dataset is that the books contain relatively long texts. Most game text is short to save the player time, but this dataset is quite large. In a previous project, I used it to learn the semantics of non-English words and create a sentiment analyzer based on those semantics. I believe it can be useful for many different NLP applications focusing on video games. It's a great starting dataset because of its size.

Dataset 2: Torchlight 2

The second dataset is very different from the first one in that it contains a wider variety of texts. The first dataset only contained fictional book texts, while this dataset is smaller but has a greater variety of texts. It contains the starting, intermediate, and completion dialogue for every quest in Torchlight 2. In addition to quest dialogue, it also contains GUI texts such as one-line quest objectives and toolbox tips. It also contains long-form summaries of the main quest and the entire main quest narrative of the game. I really like the variety of this dataset.

Dataset 3: Star Wars

The third dataset contains branching dialogue, which is a feature that Star Wars Knights of the Old Republic 1 is known for. The game was published by BioWare, who are famous for their games with high-quality writing and voice acting.

Through branching dialogue, the player influence how their character reacts to situations and dialogue from other characters in the game. Not every choice is significant, but some can change the direction of the story. In the Star Wars games, there is a big focus on good versus evil, Jedi versus Sith alignment, and branching dialogue is one way they implement this.

Branching dialogue is not something you find in every RPG, but I am happy to have collected this dataset as it contains over 30,000 lines of dialogue, including information about the conversation and speakers, and all branching options. This dataset can be used to research how branching dialogue in games works and to potentially generate new branching dialogue. I hope it is helpful for researchers interested in this topic.

The third dataset again contains only one type of text, but it is a large dataset with a wide variety of dialogue that is heavy on emotion and personality. Unlike the books in the Elder Scrolls dataset, which contain neutral "facts", this dataset has a diverse cast of characters. Researchers interested in analyzing personality, linguistic style, and emotions in narratives may be able to use this dataset.

Wrap-up

I will now take questions in the workshop Zoom session, but if you watch this talk later or have any questions, feel free to send me a message and I'd be happy to respond. You can find some information about the dataset in the GitHub repository.

For those who are not technically savvy, don't worry, I made sure that all the data is easy to use by formatting it into a CSV file. It can be read with Excel or your favorite programming language. I want to make it as usable as possible.

Thanks for listening and I'll take some questions now.

Networking tips for your first conference

2021-04-15T00:00:00+00:00

I wrote this list of tips for networking just after I visited my first conference, where networking was supereasy! And it was FUN! Although Troopers 2017 was an industry conference, the tips should work fine for academic conferences too.

I talked to so many interesting people, people working in infosec from all over the globe. I even managed to help some of them, by answering questions on social media, introducing people to each other, and pointing people to good self-education resources on the internet. I love to help people, so that made the networking activity feel very natural for me. It was not like the picture I had in my head at all (people in suits loudly talking about themselves while scattering business cards at random). I also loved to exchange horror stories about employers or infosec-gone-wrong. And the most fun thing is to form a spontaneous group of conference-friends and explore the local cuisine.

All the tips in no particular order:

Take advantage of the social activities that are offered at the conference. They are meant to facilitate fun and relaxed social interaction. So register for the social dinner, participate in a Capture The Flag competition, visit the craft booths (lock picking, soldering, ...), fiddle with the conference badge, hang around the coffee machines, and talk to random people. Academic conferences often have a social dinner, and game conferences might have a meet-and-greet with local game developers, or demo sessions where you can try out new games!
Find people with the same job as you and exchange experiences and horror-stories. Exchanging stories is also a great way to scout for potential future employers. At the same time, you get to know which companies you should avoid.
Use the information you can find online. Follow the live tweets (or posts on other social media) via the conference hashtag and see what other visitors are up to. Do they tweet about particularly good talks? Have they tried activities you haven't tried yet? Do they tweet about locations (either at the conference or in town) you haven't visited yet? Do they tweet requests, questions or invitations for additional activities? Maybe you can help other people too, by sharing your own experiences, or answering their questions. This post has more tips about using Twitter during conferences.
At the end of every day, write down the names of the people you've met, what they do, where they come from, and how you can help each other. You meet so many people at a conference that you really won't be able to remember this after one or two days.
Feel free to exchange business cards, but only if at least 1 party can do something for the other.
Speakers and crew members are not demi-gods. If you have questions, or you think you can help them or tell them something that they might find interesting, don't be shy and talk to them! However, if you run into them after their talk or during their 'off hours' during the conference, be respectful of their time and energy. They are normal people who also need coffee, sleep, down time, etc.
Registration desk people and other crew members know a lot about the conference. Remember that they are often locals! Make use of this.
Fun conversations arise most easily during a random meeting in the hallway, ie. at the coffee machine, at a pinball machine, in the cloakroom row, at a dinner, etc. The "hall-way track" is often my favorite conference track!
If you're attending a conference with an international audience: PLEASE, do not, by default, converse in your own language with colleagues or fellow countrymen. You immediately exclude a large part of conference attendees, and you prevent new people from entering the conversation. It is rude. It is also to the detriment of yourself: you might deprive yourself of new acquaintances, friendships or interesting information.
Take some time to recharge in between conference days or in-between talks. Take a walk, sit outside in the sun, find a quiet space at the conference venue to relax for a while, or enjoy some alone time in your hotelroom. My favorite coping methods for conference fatigue are (1) having dinner in a nice restaurant in the delightful company of my e-reader, or watching a movie in my pyjamas.
Dare to show yourself; being on a business trip doesn't mean you can't be yourself. Be representative AND authentic. Yes, that's perfectly possible. One of my favorite memories from Brussels is a CTF-team member, a serious infosec consultant in daily life, showing off his crazy coloured, unmatched socks. I think one was orange with tiny avocados and one had a bicycle print. "Look!" he declared, "I always wear these to clients. Compliant, yet defiant!" Great socks, great quote.
If the conference is at a different location than where you're staying, talk to the conference attendees stay in the same hotel. Maybe you can walk to the venue together, share a taxi, or meet up for breakfast in the hotel.
Ask questions to start conversations. It's easy to find a suitable question for anyone. New people: is this your first conference? Locals: do you have any tips for good restaurants or places to visit in town? People you met earlier during the conference: Hello again, how is your day so far? What was your favorite talk so far? Connecting with people is not that hard. Be considerate and friendly, and people will remember your face.
If you're shy: lots of people that visit a conference for work visit the conference alone. Chances are that other person near the coffee machine feels as disoriented, alone or shy as you.
Be attentive in your follow-up to people. Do what you promise. If someone you meet asks for a brochure, a link to a website, a tip, an introduction, write the request down and contact them as soon as possible. I try to handle all these requests either the same day, or I deal with this todo-list the first workday after I return from the trip.

I hope you liked this list. I certainly would have liked to read it in 2016! I remember eagerly scanning the internet before my trip, searching for blogposts that could tell me how conferences "work" and what you're supposed to do when you're there. Feel free to share your favorite conference experience or coping method in the comments.

Talk transcript: Fine-tuning GPT-2 on World of Warcraft quests

2021-08-03T00:00:00+00:00

This is a textual version of the talk I gave at Foundations of Digital Games 2021, for the research paper titled Fine-tuning GPT-2 on annotated RPG quests for NPC dialogue generation. The talk was transcribed using Whisper.cpp and co-edited with the open beta of ChatGPT to improve the flow of the text.

Intro

Good morning everyone. My name is Judith van Stegeren and I'm here to present a fun project that I worked on together with my co-author Jakub.

The project is a quest-giver dialog generator using GPT-2. We searched for a suitable data set to use and looked at multiple online RPGs. In the end, we chose to use a World of Warcraft data because it had the right amount of data points and context to make the project interesting.

Can GPT-2 support game writers?

This project wasn't initially focused on World of Warcraft. We simply wanted to see if GPT-2 could be used to support game writers.

This project is part of my PhD research, which has three main themes.

First, there is a lack of example texts from video games when using machine learning for video game text. So, finding new video game corpora and exploring how they can be used is something I'm interested in.
Second, procedural content generation is a thriving field, but there aren't many researchers focusing on generating textual game assets. Combining natural language processing or generation with procedural content generation is an area that needs more research in my opinion.
Third, there is a gap between academic research and industry practice. Even when natural language generation researchers find a new method, it may not be practical for the game industry. I believe we should try to address this by keeping industry requirements in mind when conducting research like this.

There are many Transformer-based large scale language models, but we chose to use GPT-2 for our research because it is the most accessible for non-machine learning experts and non-computer scientists. This is mainly due to the availability of tutorials and Google Colabs written about it, as well as the popularity of GPT-2. It is also supported by a PyTorch wrapper called GPT-2-simple by Max Woolf, which makes it easy to use. Additionally, GPT-2 is already being used in AI Dungeon, which shows its potential in a gaming context.

Finding a suitable dataset

Our original idea was to create an NPC dialogue generator, but we hadn't decided on a game yet. To fine-tune a large scale language model on game text, we needed examples. Jakub looked at Destiny fan wikis, the Eve Online quest database, and the WoWHead quest database for World of Warcraft. We ultimately chose to use World of Warcraft because it had more data points and richer content than the other options. The quests were longer and had more text variation, providing us with more information to work with.

GPT-2 is a powerful language model that can learn latent NLP tasks from examples. This means that even if we don't explicitly code a task for it, GPT-2 can learn the task by looking at the examples. The original transformer paper and the GPT-2 paper used machine translation examples to demonstrate this.

For example, if we give GPT-2 a lot of examples that include the word "translate" and strings in the source and target languages, GPT-2 will learn that this is the correct output format. This means that if we give it a sentence in the source language and the target language, it will automatically generate a translation that fulfills the latent task it learned. We want to use this same approach for generating quest dialog.

We wanted to use a data set that not only included raw text strings from the game, but also had metadata about those strings. This would allow us to use the metadata to structure the task for GPT-2. For example, we could say "give us quest dialogue given this quest title and quest objective." We could also train GPT-2 to generate the quest title and objective given a piece of quest dialogue.

Generating World of Warcraft quests

We used the World of Warcraft quest database to create data points with the structure of "start of text," "title," "objective," "NPC dialogue," and "end of text." This information is shown to the player when they start a quest or subquest. By feeding GPT-2 enough of these data points, it will become a NPC quest dialogue generator.

We generated a bunch of quests using GPT-2 and mixed them with original World of Warcraft quests written by humans. We asked 32 people to rate the quests on several aspects, which is a common evaluation method in computational creativity. On average, the generated quests were rated worse than the handwritten quests, but in two out of five categories, the differences were not significant. We also found that in some cases, the generated quests had a wider range of content quality than the handwritten quests. This means that in some cases, a generated quest might be rated higher in quality than a human-written one. This suggests that GPT-2 can be successfully used for human-computer co-creation.

Overall, these results were not surprising, but they were encouraging. This was only a pilot project, and we found that picking the right data set and looking at the limitations of this approach are more interesting than the approach itself, as many people have used similar methods for fine-tuning GPT-2.

What can we learn from this?

The new contribution of this work is not the approach itself, but rather exploring how we can generalize a large scale language model trained on game data to other games and video game genres. For example, we trained GPT-2 on World of Warcraft quests and used it to create more World of Warcraft quests or dialogue. But what would happen if we used the same language model to create content for a Star Wars game, My Little Pony game, or a Dungeons & Dragons game, which are similar but not the same as World of Warcraft? This is an interesting area for future research.

It is important to consider other models besides GPT-2. While GPT-2 was initially chosen for its accessibility to non-experts, it has limitations, such as its large size and limited language support. I would be interested in seeing similar research projects using models like BERT, which supports multiple languages and is being developed to be smaller and more efficient. The Google BERT team is also researching this direction, whereas OpenAI is focusing on making larger and larger language models, such as GPT-3. Both approaches have merit and are worth researching.

Finally I would like to experiment with incorporating more information about the player or player character into each data point. This could be useful for generating more personalized or adaptive content. This could potentially be used for creating adaptive games that can change and respond to the player's actions or choices. It would be interesting to explore this further and see how it can influence the generated content.

Wrap-up

In conclusion, this project shows that it is possible to use GPT-2 for human-computer co-creation. It is a promising first try and can be expanded in many ways. Making machine learning tools more accessible to game developers, as well as providing high-quality video game text corpora with metadata, would be beneficial for research in both procedural content generation and natural language generation. Additionally, exploring how we can generalize and transfer large language models to be reused across multiple games, genres, and domains is a valuable direction for future research.

Thank you very much and I'll be happy to answer any questions you have.

How I value mergers and acquisition deals

2022-11-06T00:00:00+00:00

Hi everyone! In this post I'll show you how I valuate mergers and acquisitions and other special situation investments. I started doing this after reading You can be a stock market genius by Joel Greenblatt. Despite the clickbait title, this is a really good book about special situation investing.

This post is a write-up about the merger between US supermarkets Kroger ($KR) and Albertsons Companies ($ACI). I describe how I found about a new merger, what sources I used to find more information, and how I did computations to assess whether it might be a good investment. It involves quite some numbers, but don't worry, if you can multiply and divide, you have all the math you need to understand this post. Note that this is only a rough back-of-the-envelope calculation. The numbers are all estimates, but it gives me a good enough indication of the upside and downside of an investment.

And finally: this is not investment advice, or an endorsement of the companies I mention here. Educate yourself, do your own research, and have fun.

My first experience with risk arbitrage

My first experience with investing in a merger was the Fiat -- Peugeot merger deal in 2019/2020. I had just started stockpicking and I was looking into companies with low Price-to-Earnings (PE) ratios. The car industry is highly cyclical, which means that the stock prices go up and down with the economic cycle, and Fiat was trading for 2-5 times their yearly earnings at the time. After doing some research, which means reading a few press releases and doing basic math, I decided to buy shares in Fiat. According to my computations, I would spend a few euros per Fiat share, and I would receive that same amount per share (plus I would get keep the share I bought!) from Fiat once the deal was completed, some 12 months into the future. This hypothesis turned out to be right, which led to my first doubled investment (+100%).

Finding out about the Kroger -- Albertsons merger

Today I looked into the Kroger -- Albertsons merger, which I encountered by accident. I was browsing the website of financial magazine Barron's, searching for articles with 'takeover' or 'acquisition' in the title. Since I own some Kroger shares, my interest was piqued.

Barron's: Kroger buying Albertsons merger stock price

I am no longer a Barron's subscriber so I couldn't read the article itself, but I know that companies are legally obliged to make all information related to mergers and acquisitions public on their Investor Relations website. You can find these (gasp!) by googling the company name + "investor relations", or search the corporate website of a public company.

I googled around for more news articles about the deal, and found another article on Barron's website:

Barron's: Judge pauses Albertsons 4 billion special dividend payout

Collecting facts

News websites

Once I'm interested in a specific deal, I start collecting facts. I always by looking up a few news articles on Yahoo Finance, Barron's, the Wallstreet Journal website, MarketWatch, etc. This is much more efficient than immediately diving into the formal SEC forms and other official documents.

From the Barron's articles I could gather the following:

Kroger will pay 34.10 USD in cash for Albertsons shares, which are now trading for about 21 USD.
A US judge temporarily stopped the payment of Albertsons "special dividend"

Investor's Relations websites

More information about the merger is really easy to find on Kroger's investor relations page:

Here we can collect more facts:

Albertsons is planning on paying a special dividend of 4 billion USD to shareholders, which was planned for November 4, 2022.
The transaction is expected to close in early 2024. Let's presume that's March 2024 at the latest. Then the runtime for this deal is March 2024 minus November 2022 = 17 Months
34.10 (cash offer) / 21 (current stock price of Albertsons) = 1.62, or 62% profit in 17 months. Not bad.
a 4 billion dividend divided by 11 billion market cap worth of shares means a 36% dividend: 4 billion / 11 billion = 1.36. The market cap of 11 billion I found on the Yahoo Finance homepage for Albertsons.

I wanted to learn more about the special dividend. It was paused by the judge, can I still participate? Was it cancelled? Would someone need to own the shares on a specific date to qualify for the special dividend?

On the Albertsons IR website, we can find this:

"Subject to the outcome of a store divestiture process, the cash component of the $34.10 per share consideration may be reduced by the per share value of a newly created standalone public company ("SpinCo") that Albertsons Cos. is prepared to spin off at closing in conjunction with the regulatory clearance process described further in the Transaction Details below. As part of the transaction, Albertsons Cos. will pay a special cash dividend of up to $4 billion to its shareholders. The cash component of the $34.10 per share consideration will be reduced by the per share amount of the special cash dividend, which is expected to be approximately $6.85 per share. This cash dividend will be payable on November 7, 2022, to shareholders of record as of the close of business on October 24, 2022."

From this piece of text I could gather more facts:

It's November 6, so I am not eligible for the special dividend. I should have owned shares on October 24 at the latest.
the special dividend should be about 6.85 USD.
the special dividend will be subtracted from the cash offer for Albertsons shares, so that's 34.10 USD - 6.85 USD = 27.25 USD per share.
the merger deal is subject to anti-trust regulations. I think this means that Albertsons and Kroger are not allowed to merge if that leads to a monopoly type situation where competitors no longer stand a fair change. Kroger and Albertsons are planning to spin off (divest) some of their stores into a spin-off company, which will continue as an independent public company. "SpinCo" is the working title for this spin-off. Albertsons shareholders will receive shares of this spin-off company, and the value of those shares will be subtracted from the 34.10 as well.

Since the current stock price of Albertsons is only 21 USD, does the market think that the spin-off company will be worth 34.10 cash - 6.85 special dividend = 6.25 USD per share?

The spin-off

Let's find more information about this spin-off. There are two other interesting bits of text on the IR website:

The per share cash purchase price payable to Albertsons Cos. shareholders in the merger would be reduced by an amount equal to (i) three times four-wall adjusted EBITDA for the stores contributed to SpinCo divided by the number of Albertsons Cos. common shares (including common shares issuable upon conversion of Albertsons Cos.' preferred stock) outstanding as of the record date for the spin-off plus (ii) the per share amount of a special pre-closing cash dividend of up to $4 billion payable to Albertsons Cos. shareholders, which is expected to be approximately $6.85 per share. This cash dividend is expected to be payable on November 7, 2022, to shareholders of record as of the close of business on October 24, 2022.

and

"In connection with obtaining the requisite regulatory clearance necessary to consummate the transaction, Kroger and Albertsons Cos. expect to make store divestitures. As described in the merger agreement and subject to the outcome of the divestiture process, Albertsons Cos. is prepared to establish an Albertsons Cos. subsidiary (SpinCo). SpinCo would be spun-off to Albertsons Cos. shareholders immediately prior to merger closing and operate as a standalone public company. Kroger and Albertsons Cos. have agreed to work together to determine which stores would comprise SpinCo, as well as the pro forma capitalization of SpinCo. The establishment of SpinCo, which is estimated to comprise between 100 and 375 stores, would create a new, agile competitor with quality stores, experienced management, operational flexibility, a strong balance sheet, and focused allocation of capital and resources to provide customers with continued value and quality service and associates with ongoing compelling career opportunities."

I tried to figure out what the spin-off could be worth by combining numbers from these paragraphs.

Facts:

the spin-off will be comprised of 100 to 375 stores.
the spin-off will be valued at 3x adjusted EBITDA (Earnings Before Interest Taxes, Depreciation and Amortization) of those stores.

We don't know which stores will be transferred to the spin-off company, but they will be Albertsons stores. So what is the EBITDA of an average Albertsons store?

I took a look at the last annual report (or "form 10-K") filed by Albertsons to get more information about this. You can find these reports by going to EDGAR, the official database of the US Securities and Exchange commission.

EDGAR: Albertsons Companies, Inc

"As of February 26, 2022, we operated 2,276 stores across 34 states and the District of Columbia under 24 banners including Albertsons, Safeway, Vons, Pavilions, Randalls, Tom Thumb, Carrs, Jewel-Osco, Acme, Shaw's, Star Market, United Supermarkets, Market Street, Haggen, Kings Food Markets and Balducci's Food Lovers Market. Additionally, as of February 26, 2022, we operated 1,722 pharmacies, 1,317 in-store branded coffee shops, 402 adjacent fuel centers, 22 dedicated distribution centers, 20 manufacturing facilities and various digital platforms."

"Adjusted EBITDA was $4,398.4 million, or 6.1% of Net sales and other revenue, during fiscal 2021 compared to $4,524.0 million, or 6.5% of Net sales and other revenue, during fiscal 2020."

Facts:

Albertsons has 2,276 stores, with a combined adjusted EBITDA of 4398 million USD in 2021.

Some estimates:

4.3 billion EBITDA divided by 2276 stores means 1.8 million USD on average per store.
in case of 100 stores in the new spin-off: 3x 100 stores * 1889279 = 566 million
in case of 375 stores in the new spin-off: 3x 375 stores * 1889279 = 2.1 billion

We need to divide the spin-off value by the number of Albertsons Cos. common shares (including common shares issuable upon conversion of Albertsons Cos.' preferred stock) outstanding as of the record date for the spin-off.

For a current count of the outstanding shares, I'm looking at the last quarterly report (or "form 10-Q") of Albertsons, also in EDGAR.

"The Company calculates basic and diluted net income per Class A common share using the two-class method. [...] Diluted net income per Class A common share is computed based on the weighted average number of shares of Class A common stock outstanding during each period, plus potential Class A common shares considered outstanding during the period, as long as the inclusion of such awards is not antidilutive. Potential Class A common shares consist of unvested restricted stock units ("RSUs"), restricted common stock ("RSAs") and Convertible Preferred Stock, using the more dilutive of either the two-class method or as-converted stock method. Performance-based RSUs are considered dilutive when the related performance criterion has been met."

From the nearby table, I gather that the number of diluted Class A common shares was 576.3 million on September 10, 2022.

More estimates:

In case of 100 stores in the spin-off, the spin-off will be valued at 566 million EBITDA / 576.3 million shares = 0.98 USD per share
In case of 375 stores in the spin-off, the spin-off will be valued at 2100 million EBITDA / 576.3 million shares = 3.64 USD per share
Interestingly, Albertsons is currently valued at about 7 times it's EBITDA per share: 4398 EBITDA / 576.3 shares = 7.63 EBITDA per share. The current stock price is 21 USD, so the price-to-EBITDA ratio is about 3, just like the future spin-off!

Conclusion

Conclusion: if I want to participate in the spin-off, I can buy shares in Albertsons tomorrow for 21.50 USD per share. If everything goes according to plan, in spring 2024 I will get:

0 USD special dividend
1 share of the spin-off (estimated valued at 1.00 to 3.64 USD)
27.25 USD minus the value of the spin-off share

So is this riskless investment? Free money?

Certainly not. There are many reasons a deal can fall through, such as shareholders (of either company), activist investors, a third-party, legal disputes and anti-trust laws. You only need to read the articles about Elon Musk's takeover of Twitter to know that mergers and acquisitions can be rollercoasters.

Investing in mergers and acquisitions is sometimes called risk arbitrage. As investor, you take the risk that the deal doesn't close, and in return, you get the computed amount as reward if everything goes as planned.

So what can go wrong? What is the downside?

If the deal falls through for whatever reason, an acquired company's stock price might sink to the level is was trading on the day the deal was announced. In Albertson's case this was October 14, 2022. However, at that time the stock was trading for 24 dollar, which is higher than the current amount. The stock price dropped dramatically on the "ex-dividend date", the deadline for being eligible for the special dividend.

It could also be that the share of the spin-off is worthless, or worth much less than the assigned value of 3x EBITDA. This can happen if Albertsons and Kroger decide to spin-off the worst performing supermarkets. If I want to be conservative, I can value the spin-off share at 0 USD. Then the worst case up-side of buying a 21.50 USD share is 34.10 - 6.85 special dividend - 3.64 spin-off share price = 23.61. This translates into a profit of 23.61 / 21.50 = 1.098, or about 10 percent, over 17 months. It's not too bad, but it's also not great.

I think the upside on this deal is not enough for me to participate, but I enjoyed the necessary detectivework nevertheless. If you've made it until here, well done. I think this is my longest blogpost yet. I hope you found this write-up interesting!

A workflow for turning talks into blogposts with ML

2022-12-11T00:00:00+00:00

This blogpost was co-created with ChatGPT.

In this blogpost, I will describe the workflow I used to efficiently turn my talk videos into blogposts with machine learning. The heavy lifting is done by two relatively new machine learning tools: whisper.cpp and ChatGPT.

I use whisper.cpp, created by Georgi Gerganov, for audio transcripton. Whisper.cpp is a lightning fast open-source implementation of OpenAI's automatic speech recognition (ASR) model Whisper. I love that it runs offline, on my 5 year old computer, without any third party dependencies. It transcribed 15-30 minute talks in mere minutes. The audio transcripts are edited by hand and then fed ChatGPT for further processing.

ChatGPT is a large language model by OpenAI -- as it will repeatedly tell you if you feed it the wrong prompt. It can be used for a variety of natural language processing tasks, such as text completion, translation, and summarization. I use ChatGPT to help me rewrite my transcripts into more fluent and natural-sounding texts.

Here's the workflow:

Clone the whisper.cpp git repository and follow the excellent installation instructions in the readme.
Optional: download a specific GGML model for whisper. However, you don't NEED a fancy model. The base model that is downloaded as part of the whisper.cpp demo worked fine for transcribing talks. The base model supports multiple languages. (Edit: for transcribing English talks I now use the medium-sized model for English: ggml-medium.en.bin)
Prep the audio of the talk you want to transcribe. My old talks are on YouTube, so I used youtube-dlp to download the video and extract the audio as an mp3-file. Example: youtube-dlp -x https://www.youtube.com/watch?v=dQw4w9WgXcQ --audio-format mp3
Whisper.cpp only works on 16-bit wav files, so I then convert the mp3 to wav with ffmpeg. Example: ffmpeg -i filename.mp3 -ar 16000 -ac 1 -c:a pcm_s16le filename.wav
I tell whisper.cpp to transcribe the file, following the demo in the project's README. Example: ./main -m models/ggml-base.bin -otxt filename.wav
Now we start prepping the transcript to make it ingestible by ChatGPT. Start by adding newlines to the transcript and break up the text in clear paragraphs.
Edit the text to remove any irrelevant paragraphs.
Remove incorrect words or substitute them for placeholders such as [name]. Things like names, urls and hyperspecific jargon can be tricky to transcribe for Whisper.
Use ChatGPT to help you rewrite the transcript into a more fluent and natural-sounding text. A useful prompt for this step is: "You are a professional editor. Rewrite this transcript into a more fluent text. Break up long sentences without changing the meaning or leaving out information. Make the flow of the text more natural." I started out with adding the prompt to every interaction with ChatGPT. However, the bot is optimized for dialogue and it pretty good in remembering the conversation so far, so retrospect this was not really necessary.
Feed one or two paragraphs at a time to the language model. If you provide it with too much text (e.g. three or more paragraphs), it may start to leave out important information.
Concatenate all rewritten paragraphs.
Factcheck and spellcheck the text. If you skip this step, I will cry and/or lose my hope for the future of humanity.
Finish up the blogpost by adding links, references, and images.

Writing a blogpost this way can still be a lot of work, but it definitely beats transcribing 30 minutes of audio by hand. As added bonus, I now have text-searchable (and search engine discoverable!) textual versions of my past talks.

Stock market games: Neopets

2023-01-10T00:00:00+00:00

This post is part of my Stock Market Video Games series! In these posts, I'll do a deepdive in the stock markets that we can find in video games! I want to find out how these fictional stock markets behave, and how they differ from the real world. Are they random or realistic simulations? And how can we use that knowledge to get excess returns? After all, we should only invest in what we understand.

As far as stock market video games go, the first virtual stock market I encountered was in online game environment Neopets, around the early 2000s.

Neopets is a website where you take care of a virtual pets, play games and interact with the world of Neopia and other Neopets players. On the website, you can play online flash games in exchange for virtual money, or NP (neo points). I would spend NP on buying food or items for my virtual pets. I would save my money in the Bank of Neopia, which would give me a whopping 12% interest (!). However, after playing for a while, I found out that "investing" my NP in the Neopets Stock Market would yield even more than that.

How it works for players

NeoPets has a large community, so we can find quite some guides online that explain how their stock market works. Most of the information in this post is taken from this excellent guide by Jellyneo.net.

Stocks are shares in 40+ fictional Neopets-themed companies. Some companies can be found as location in the Neopets game environment.
There is no "fundamental" information, apart from the ticker, a flavor icon, and stock price all companies are indistinguishable
Stock prices update every 30 minutes
The stock market is always open
You can only buy stocks that have a stock price of 15 NP or above, although there is a special in-game reward that allows you to buy stocks with a minimum price of 10 NP.
Stock prices are integers >= 6. The highest price ever recorded according to NeoStock.info is VPTS, with a price of 1770 NP.
A stock value will never go below 6 NP, but there is (in theory) unlimited upside.
Players can buy at most 1,000 stocks per day
Players can sell unlimited stocks per day for a fee of 20 NP.
The stock market has a daily reset that determines opening prices and the 1,000 shares cap for buying shares

Realism

If we look at the list above, the NeoPets stock market does not seem very realistic. The market is always open, there are hard caps on buying activity, and stock prices change only 48 times per day. Stock prices seem to be random, and companies are interchangeable apart from ticker and price. Realistic bits are that players pay transaction costs for selling stocks, and some companies have disappeared over the years.

According to Jellyneo.net's guide, stock prices USED to be linked to in-game economics. However, this led to stock price manipulation by groups of players, so now prices are random:

"Other than the current prices, the different companies are functionally indistinguishable. Years ago, the Stock Market was actually influenced by how well certain Neopian businesses did; for example, Hubert's Hot Dog stocks went up and down in value depending on how many items the shop of the same name sold. However, this led to pockets of users grouping together and buying out entire shops to inflate the value of their stocks, so The Neopets Team put a stop to that. These days, stock price updates are random, though [they] do follow some patterns [...]."

Fan-website Neostocks.info hosts statistics about the the NeoPets stock market in general, and specific stocks. In earlier years, the website neodaq.com (now down) had advanced tools for monitoring the Neopets stock market, including game timers, technical charts, and elaborate Excel sheets for tracking player portfolios. I can imagine that charting and technical trading was a real thing for hard-core players back when the stock prices were still linked to in-game user behavior.

Using archive.org, I looked up the list of tickers on neodaq.com, which listed 52 companies in 2003. Neostocks.info lists 43 companies, so some companies have disappeared or "gone bankrupt".

A funny detail is that Neopets has a stockbroker character called Nigel the Chia, who will occassionally give you tips about which stocks to buy. However, his tips are truly random, so following Nigel's tips is generally detrimental to your return. I suspect these events are hard-coded sentences with random tickers, because I've read that some of the stocks he recommends no longer exist. Whenever you want to sell stocks, Nigel the stockbroker earns a trading fee of 20 NP. That's a nice touch of realism: it's a bad idea to blindly follow stock tips, and your stock broker will always profit from increased trading activity.

How it works under the hood

Some NeoPets player communities have collected 6 months of stock price data and done some analysis to find patterns, such as price change distributions. According to Jellyneo's guide, the stock market prices can be modelled as a Markov chain. Only the current stock price determines the minimum and maximum price change, and all stocks follow the same probability distributions.

I think this visualization by Reddit user not-the-artist says it all:

If you want to know more about the distributions, read Detailed Stock Behavior in Jellyneo's guide, or the Reddit post that inspired it.

How to get rich*

*in the NeoPets stock market

Only buy at stock prices of 15 NP
Only sell at a profit
Have patience

Profit guaranteed :)

Of course, this is what made the Neopets stock market so enticing for me, as a ten year old: "Unlike the real world, you are guaranteed to make a profit on all of your stocks eventually; it's just a matter of waiting."

I remember I would buy stocks only when they were at 15 NP. I would spread my expendable NP income over multiple cheaply priced stocks and... just wait for the price to go up. Since there were no company fundamentals, no analist coverage, or investor relations updates to follow, I only had to wait (days or weeks) before the price would have randomly ambled to 30, and I would sell at a 100% or more return. The only thing I had to do was check the stock prices once every day.

This concludes my research into NeoPets Stock Market! If you would like more background about NeoPets, I can recommend this micro-documentary about NeoPets by People Make Games:

I hope you enjoyed this blogpost. If you know of any other video games with stock markets that I should play and/or look into, drop me a line below in the comments, via email or Twitter!

Job hunting tips for people in tech

2023-06-01T00:00:00+00:00

A friend of mine is currently looking for a new job. She was asking around for a few good tips on job hunting. I spent quite some time looking for a job right after finishing my PhD in 2022, so I had ample to share. After penning down these tips on irc, we decided that storing them in a more persistent, public place might be useful for others as well.

Please keep in mind that everything below is based on my personal experience and might be overfitted to an extraverted tech-worker in the Netherlands.

Tip 1: Take your time.

The process of job hunting can be exhausting. Reflecting on what you want, exploring your options, searching vacancies, and applying for jobs takes time and energy. Finding a good job is a marathon, not a short distance race. Pace yourself and treat yourself kindly.

Tip 2: Define your nightmare scenario

Create a "dealbreaker" list that highlights things you absolutely cannot tolerate in a job. Take the time to visualize the worst job in the world. This list will help you determine which opportunities are not worth your time and energy. And even if you receive a high salary offer from a place that ticks a few of the items on your dealbreaker list, this will help you with saying no to the temptation. You can also use it to evaluate your new job, after you've been there for a few months: did any of these dealbreaker properties pop up in after a while?

Here is my dealbreaker list from 2021 as an example:

My nightmare job

Lengthy daily commute by car or train

Working over 40 hours per week

Frequent overtime, on-call, night, or weekend shifts

Lack of personal development opportunities or training

Limited vacation days

Micromanagement, e.g. fine-grained time tracking

Difficult to take leave

Lack of supportive or friendly colleagues

No job security or permanent contract

Working in a restructuring organization

A bullshit job

Being underpaid compared to peers with similar backgrounds or experience

Working for a financially unstable organization

Being the only woman in the team

Bro-grammer or macho culture

A highly competitive work environment

Strict 9-to-5 (or longer) presence requirement at my desk

Overly strict punctuality rules

Limited influence over my workprocess.

No autonomy

Narrowly defined role

Lack of acceptance for alternative clothing, progressive values, etc

An organization reluctant to use open-source software like Linux or Git, preferring corporate-style software

This list turned out to be very important to me. A financial company had offered me a job with really high compensation, but just before I signed the contract, the HR manager came back to tell me they could not support remote working of any kind. This meant I would have to commute for 3-4 hours every day -- something that violated multiple points on my dealbreaker list. ;) I took a job closer to home, which was the best decision I made that year.

Tip 3: Reflect on your dream job

Make a second list, this time with all the properties of "the perfect job". As I've said before, this is a great starting point for hunting jobs online. Think about everything that you would like or that makes you happy: travel distance, salary, day-to-day activities, work culture, colleagues, working in a specific sector, working with specific technologies.

Here is my dream job description from 2021 as an example:

My dream job

Close to or in my hometown, or completely remote

In finance or related to investing

High probability of fixed contract

Salary of at least X

Involves at least some research

Requires software engineering and data wrangling

My PhD research is relevant or my research skills can be applied

Environment that allows me to learn a bit more about running a business

Small or medium sized company

Not a non-profit or government org

Allows me to talk, blog, publish etc. about my work

Tip 4: Ask other people for advice

Once you know what you are looking for in a job, it's time to talk to other people about this. Use your IRL social network (friends, family, peers, colleagues) and online social networks (e.g. Mastodon, LinkedIn) to find people that are already working in your dream job. In other words, look for people who you'd want to trade places with.

Send these people a message such as this:

"Hello, nice to meet you! I'm [your name here]. I'm currently doing X, but I would like to do Y in the future. I see that you are doing Z. Could we have a 30-minute virtual coffee chat and discuss this further? I'm curious about your advice / I would like to ask you some questions."

Here are some examples for Z:

"You work at a company I like!"

"You work in a sector I'm interested in!"

"You did a cool relevant hobby project!"

"You gave an interesting talk!"

When I was preparing for a job in industry in the final year of my PhD, I cold-emailed a bunch of people like this. 80% of people said yes (and 20% of people never responded). In many of the conversations, I was offered a follow-up conversation with a hiring manager job, even when that was not my intention when I requested the conversation.

Why does this approach work? If you are sending messages to the right people, and you are respectful and show up prepared, people don't mind investing a bit of time in you. Most people will say "yes" because everyone likes talking to an enthusiastic person!

Besides, most successful company are always looking for good employees, and these are particularly hard to find. If you're proactively reaching out to other people, this shows you are a person with soft skills. If your contact liked talking to you, they might think "hey-- I bet this person would be a fun colleague!" Your contact will tell their manager/HR department/friend at another place about the positive impression you made. Suddenly the barrier to getting hired is much lower than when you would've applied somewhere through a job ad.

Tip 5: The internet is a gold mine.

Reading can give you an information advantage. Being more prepared than the average applicant will immediately pay off. Patrick McKenzie wrote two very useful blog posts about careers and salary negotiations. If you ask me, this is non-optional reading.

Dutch entrepreneur Bert Hubert also gave an interesting talk titled "Career Development & Money: How to be happy at work for the next 40 years (from a mostly Dutch/European IT perspective)".

Tip 6: Search for technologies, not roles

You can of course apply to jobs through official published vacancies on "careers" pages or vacancy sites like Indeed. However, finding suitable technical jobs can be difficult because role titles tend to change as frequently as JavaScript frameworks (pun by ChatGPT). Searching for specific technologies such as languages (Python), libraries (Pandas) or frameworks worked much better for me than searching for specific roles like data scientist or machine learning engineer.

On that subject: if you're in data science, be aware that the title "data scientist" is very problematic. At one company, it means "person who uses Excel or Tableau fulltime", while at another company it means "person who manages AWS instances and builds custom ML pipelines using 200 different bleeding-edge technologies". If you're planning on applying to a "data science" role, be sure to come to the interview with an extensive list of questions to see which variety they actually mean...

Tip 7: Give context for your skills

Many tech CVs have to pass Human Resources before they are sent to CTOs and technology managers. We techies LOVE putting large lists of programming languages, tools and obscure libraries in our CVs, to show off our projects and how competent we are. However, these lists are often difficult to interpret for people without a tech background!

This slide from a talk by Vincent D. Warmerdam illustrates the problem nicely:

A recruiter advised me to include a 'tech translation page' in my CV. This page should be a prose description of how and when I would use the things I listed as my technical proficiencies.

Time for a concrete example! Instead of writing

Operating systems: Windows, Linux (Arch Linux, Debian/Ubuntu, Kali), iOS, Android

you can write

I’m familiar with Windows, multiple flavors of Linux, iOS and Android. I use Linux for development and other technical tasks. I run Arch Linux on my personal laptops and manage multiple VPSes (virtual private servers) that typically run Debian. In the past I’ve also used Ubuntu, Linux Mint, and Kali Linux for projects in the information security domain.

A longer example can be found on page 3 of an old version of my CV.

In the days before ChatGPT, this appendix also signaled to hiring managers that you could actually write a piece of text. Now that we have large language models, every text might be written by a robot, so I wonder how long we will keep using written applications...

I hope these tips were useful! Good luck with your job hunt.

How to install Herrie on Arch Linux

2024-09-02T00:00:00+00:00

Herrie is the best music player there is! That is, if you're like me and you like minimal terminal applications.

Whenever I get a new laptop and a fresh Arch Linux installation I immediately miss it -- once every 6 years, which is exactly long enough to forget how to install it. ;) It used to be in the AUR, but not anymore. So here's a cheatsheet for installing Herrie on Arch Linux in 2024, even though the application was last actively developed in the 00s. I think Herrie's longevity is a testament to the creator's superior programming skills. Thanks Ed! <3

# install dependencies
sudo pacman -S gettext dbus dbus-glib glib2 base-devel glib2-devel \
alsa-lib libao libcurl-gnutls libid3tag libmad libmodplug libsndfile \
libvorbis ncurses pulseaudio

# get the latest version of the repository
# NOT the latest official release herrie 2.2
git clone git@github.com:EdSchouten/herrie.git

# get libxspf from the AUR: 
wget https://aur.archlinux.org/cgit/aur.git/snapshot/libxspf.tar.gz
tar xzfv libxspf.tar.gz
cd libxspf
makepkg -sri
cd ..

Sometimes the compiler is complaining about missing somefile_c.h files, even when you're sure you've installed the right package. The compiler might be expecting the files in /usr/include but sometimes they're in /usr/include/some_lib. You can create a symlink to fix that.

For example:

sudo ln -s /usr/include/libmodplug/modplug.h /usr/include/

Install herrie. Herrie defaults to using ALSA so if you use pulseaudio, don't forget to explicitly tell ./configure. Read the README for more info on the ./configure flags.

cd herrie/herrie
./configure pulse
make
sudo make install

Lessons from the first year of my PhD

2025-06-16T00:00:00+00:00

I finished my PhD in computer science more than three years ago. Now that I have a little bit of mental distance from my PhD time, I'm going through my research diary to see if there's stuff in there that can be useful for other people.

Apparently I'm incapable of writing short-form texts, so here is part 1. ;)

General life advice

When you're stuck, ask for help.

Research

Take inspiration from chemists/laboratory workers and keep a research log. It will occasionally save your life (and sanity). My log from year 1 is 139 pages long. That's about half the length of my PhD thesis.

Your research log doesn't have to be fancy. My research log was just a very long Libreoffice document. Each entry consists of a date and whatever I wanted to write about that day! It's filled with braindumps, todo lists, project ideas, observations, philosophical ramblings, doubts, meeting notes, and questions.

I recommend that you keep your research log 100% private so there's no pressure to sound smart or sophisticated whatsoever. Express yourself freely while writing in this doc. It will help you think.

It's a perfectly valid strategy to enter a new research field by reading interesting-looking papers and painstakingly looking up every unknown abbreviation, word, concept, and technique. Here's a list from my first week at work, way back in 2018:

Definitions to look up after reading survey paper 1:

computational linguistics

nlproc

canned strings

English glosses

lexical items

ontological status of categories

surface linguistic forms

discourse history

morphologic

orthographic

bidirectional grammars

linearising words

forest (machine learning)

Every research project starts with not knowing what you're going to do (exactly). This should not worry you or cause anxiety: it is an integral part of the process.

Start writing your thesis from day 1, even if it's just writing down the main premise of a paper in two sentences.

Research can be an intangible activity, so take measures to make it more tangible! This helps to combat PhD progress anxiety. I did this by writing my research log, evaluating my overall progress and happiness via a monthly evaluation questionnaire (blogpost forthcoming), and making daily tallies of distraction-free pomodoro sessions.

When I started my PhD, it took me an entire workday to work through a single 8-page conference paper. This will get better over time.

Never forget you're studying a subfield of a subfield of a subfield. You can't know everything. See also Matt Might's Illustrated Guide to a PhD.

If you're feeling insecure about your research work: talk about your insecurities with friends, family and colleagues - but, whatever happens, continue working. Don't let your insecurities hold you back. In the words of Matt Might: be tenacious.

When doing a PhD you will invariably bump into your own weaknesses. In the end you're the project manager and the only project contributor.

When designing experiments with human participants: it's a huge advantage if the researcher does not have to be in the room while the data gathering/experiment takes place. This way you can do longitudinal experiments without sacrificing all your research time because you have to babysit (sometimes literally) your participants.

There is a large difference in quality between PhD courses. Ask your colleagues which courses are worth your time and consider ignoring/replacing the rest.

Half of the experiments in Human Media Interaction don't involve any automated system at all. Instead these experiments are "wizarded": a PhD student sits behind a curtain and acts like an automated system, so that data gathering can take place. And yes, sometimes it's hard not to laugh when you're being the Wizard.

Applied research

If you're creating software/code artifacts during your PhD, it's worthwhile to invest in your software engineering skills. Scripting (=run once code) is not the same as software engineering! Find people that are working as a software engineer in industry and humbly ask them for feedback on your code. Thanks to the advice of my friends, I got started with virtual environments, pinning my dependencies, linters, some typechecking (but still not enough), more complex version control, and writing basic tests.

Not all learning needs to immediately pay off. You should have some time for exploring new technologies or doing sideprojects. For example, I started learning Flask, Django, test-driven development, typed Python, and Javascript in my first year. I also built a first (tiny) js game, a static site generator, a few webscrapers, a text generation demo based on grammars (so cute!), and did more devops on my VPS because I managed the website for our research consortium.

Try out demos or beta products of companies operating within your area of expertise. This gives you a sense of what industry is working on and what they're considering worthwhile problems. Example: in January 2018 I tried out "Microsoft Cognitive Services" which had a picture description API at the time. This was before LLMs with vision capabilities so this was a big deal! Another colleague (specialized in dialogue systems) was doing side-projects on a secondhand Amazon Echo device.

If you have to create artifacts (games, stories, illustrations, large documents that are not scientific papers or your PhD thesis, ...) for your research project, and they're not essential to the research itself, try to work from things that already exist as much as possible.

For demo sessions and demo markets at conferences: pre-recorded videos are the way to go. Live demos are cool, but the chance that your multi-component text generation pipeline suddenly doesn't work is not worth it. Better not waste the precious time and attention from audience in the room, or the people who wandered past your corner of the demo floor.

Demos don't have to be representative or recent, as long as they're entertaining, they bring your point across and spark conversations with new and relevant people.

Communication

It is completely unnecessary in academia to explain why you're asking a question.

Ask open questions, and try not to fill in too much when asking questions. I noticed people ask closed questions for various reasons. Some people want to look smart. Other people mistakingly think communication is more "efficient" if they can guess upfront what your answer will be. Example: if I ask someone "Did you use technique X?", they will say "No, I used technique Y", and spend most of their time explaining why they're not using X -- which is not what I wanted to know. It's better to ask "What is the underlying technique?".

Networking

Find your community within academia. Find a group of people that shares your research interests, with a positive, non-toxic, supportive culture. No, just having shared research interests is not enough.

Find your peers! Your direct colleagues at your university might not be working on the same topic as you, but some people in the world probably are. It often takes a few months before you visit your first academic conference and meet these international colleagues. Luckily, you can speed up this process a bit. Every now and then, spend some time on the internet and use your best OSINT skills to find pockets of people working on similar stuff as you. Look into social media platforms, "old-skool" mailinglists, irc channels, forums, research magazines, or special interest "workgroups". For example, my first conference was CLIN28 (Computational Linguistics in the Netherlands) and I discovered many relevant people and cool projects by trawling Twitter for historical mentions of "CLIN" or the conference website.

Great ways to find conferences to attend or follow: ask your supervisor; visit the conferences that have published your favorite paper in their "conference proceedings"; stalk your academic friends on social networks; subscribe to relevant newsletters and alerts; any websites/groups/other conferences mentioning a conference you've already visited.

I hope this was useful. See also my blogpost about using ~~Twitter~~ microblogging as a researcher, and tips for attending (industry) conferences.

A pragmatic method for picking PhD research projects

2025-06-17T00:00:00+00:00

This blogpost applies (as all my PhD related writings) to a PhD in computer science in the Netherlands. It will be different for other research fields and other countries.

In The Netherlands, most PhD students get a contract as university employees for 4 years. After the first 12-18 months there is an evaluation moment. By then, you are expected to have either your first conference paper under your belt (co-written with your supervisor), or at least have a paper manuscript ready for publication.

In my experience you can defend your PhD in computer science if you managed to get 4 conference papers published and gather these in a PhD thesis with an acceptable "narrative arc": a logical string of research questions within a specific narrow research field.

That means that if you manage to do one research project per year and publish it in a conference, you're on track with your PhD.

Too many ideas, too little time

Of course, as a PhD student you're always pressed for time.

Apart from reading papers, writing code, conducting experiments, doing evaluations, and writing papers, you're also spending time on teaching, mandatory coursework, meetings, helping your colleagues, loafing around the coffee machine, traveling, and exchanging ideas with highly-educated strangers. You might do an internship at a company, or perform extra academic service activities to prepare for your next job interview: (sub)reviewing papers, organizing workshops, and so on.

So your research must be reasonably efficient.

To make matters worse, you probably have more research ideas than you can execute on during your PhD time.

You're especially liable to Research Idea Overabundance if:

you have many different interests and see connections everywhere
you are surrounded by interesting people
you work on multidisciplinary problems
you like to read widely

Choose anyway

The list below describes a very pragmatic approach for selecting research ideas.

You can use this list of statements to find the most promising research ideas, in such a way that will probably allow you to finish your PhD within the fixed term of 4 years.

The checklist is meant for selecting research projects that take 3-6 months, and have a reasonable chance of succeeding and getting published.

If you do one such project every 6 months, you can do 7 full research projects in 3.5 years and then still have take 6 months to finish up your thesis and find your next job. Not all your projects might end up in your PhD thesis, and that's fine. The focus of your thesis will probably shift organically over time, as you collaborate with different colleagues and develop your own research vision.

I would argue that it's even possible to do a 6-month research project in the first half year of your PhD. In fact, I highly recommend it. Just treat it as a more advanced master's thesis. It probably won't be your best work, because when you start, you're still feeling out the nuances of your research field. But you will immediately get your feet wet, and you'll probably learn a lot faster than when you're spending your first 6 months just reading survey papers and worrying about your qualifier exams.

My experience

I used this approach to pick research projects during my PhD, and I was finished within 4 years. I would strive to do two 6-month projects per year, and have time left for submitting papers, waiting for reviews, and dealing with unforeseen setbacks.

In the end I finished with 10 publications -- written between 2018 and 2021 with various wonderful co-authors -- of which several were suitable for inclusion in the narrative arc of my PhD thesis.

Checklist for selecting a 6-month research project

Something similar has been done before, but not exactly the same. List similar papers.
You can find peer-reviewed literature on this topic. List papers.
It's clear which subfield of your work domain this research falls into. List the most important keywords you will use to describe the research project.
You can come up with a good validation method for your research. List papers that use same methodology.
All necessary data is available or can easily be created. List links to datasets.
All necessary code is available or can easily be created. List links to git repositories, models, Python packages etc.
All necessary hardware, infrastructure and other resources are available or can easily be obtained. List resources and how/where you will get them.
It's clear what problems you will encounter during your research. List of problems and how you will tackle them.
There is someone who can help you if you get stuck with your research. List names, contact details and their expertise.
The research is interesting from an academic perspective. Do ~~ancient men with beards~~ established people in your field find this interesting or remarkable? Describe your reasoning.
Bonus points: the research is interesting from a commercial/economic perspective. Do entrepreneurs, companies, multinationals benefit from it? Describe your reasoning.
Bonus points: the research is interesting from a societal perspective. Does your mother, grandmother, or neighbor benefit from it? People from other countries? Specific groups? Extend this question according to your personal values. Describe your reasoning.
There is a journal or conference where the paper that describes this research will fit in. List names of publications. You get bonus points if your paper will cite other papers from this publication.
The research can be completed within an estimated 6 months. Incorporate time for unforeseen problems: plan for 4 months and the work will probably be done in 6 months. Add a rough weekly planning.
The research project cannot be broken up into sub-projects worthy of a full conference paper. If your research projects can be divided up even further: that's great! Write down your vision for the larger project, and go through this checklist for each subproject. No cheating.
John Regehr wrote: "Every good research idea is a departure from the accepted wisdom, but it's important to depart at the right level." So... is your departure from accepted wisdom at the right level? Find the middle ground between "Everyone is wrong! I know better!" and "This small insignificant change has no impact on the world whatsoever." Describe your reasoning.
The paper fits into your envisioned narrative arc of your PhD thesis. Describe how the main research question of the project relates to your other papers or research topics.

I hope this list helps you with finishing your PhD on time. Feel free to also send this to your master students when they're struggling to pick a worthwhile topic for their master thesis.

Niels van Spaendonck, kunstenaar

2025-06-29T00:00:00+00:00

This blogpost is a write-up of some internet research I did on Niels van Spaendonck after buying her etching "Ampullen". The original write-up is in Dutch, but if you're curious and can't understand Dutch, just use your browser-native translation functionality or throw this post in GoogleTranslate. Machine translation is pretty decent nowadays. :)

Ik heb onlangs de ets/aquatint "Ampullen" van Niels van Spaendonck gekocht. Hij hing in een chaotische kringloopwinkel en viel bijna niet op tussen alle zooi, maar de zachte structuur van de ets fascineerde me. Ik kon bij thuiskomst erg weinig over deze kunstenaar vinden, dus ik heb wat online onderzoek gedaan en hieronder verzameld wat ik kon vinden.

Over Niels van Spaendonck

Allereerst: Niels van Spaendonck is een vrouw. Niels is een afkorting van Reinilda.

Reinilda Carola Anna Maria Dolk (‘Niels’) geb. Leiden 27 augustus 1910. Woonde en werkte in Leiden, Rijswijk, Den Haag; thans in Tilburg. Huwde 31-7-1939 te Tilburg met E. A. J. A. M. van Spaendonck. Leerlinge van de Akademie v. B.K. in Den Haag, etsles van R. van Rossem en lessen van W. van Leusden; later avondcursussen op de Bredase Akademie ‘St. Joost’. Schildert portretten, tekent (pen) en etst naar vrije verbeelding. Verkreeg een prijs voor portretschilderen onder de leden van ‘Kunstliefde’ te Utrecht, waar zij lid van is.

Bron: Lexicon Nederlandse beeldende kunstenaars 1750-1950 van Pieter Scheen (1969), gepubliceerd op Ensie.nl.

De achternaam van Niels van Spaendonck is nogal verwarrend, en dat komt omdat ze trouwde met een familielid! Haar moeder was Aloysia ("Louise", "Louisa" of "Wies") van Spaendonck en haar vader's achternaam was Dolk.

Aloysia Christina Vincentius Maria van Spaendonck (‘Louise’) geb. Tilburg 15 juli 1888, overl. Tilburg 14 december 1967. Huwde 27-11-1908 aldaar met H. C. M. Dolk, van wie 14-3-1925 in Den Haag gescheiden. Moeder van de schilderes R. C. A. M. Dolk. Woonde en werkte in Tilburg, Leiden, Rijswijk (Z.H.), Den Haag en Tilburg. Leerlinge van Ch. L. P. Zilcken. Schilderde en aquarelleerde stillevens, bloemen en portretten. Exposeerde in Den Haag. Leiden en Tilburg.

Bron: Lexicon Nederlandse beeldende kunstenaars 1750-1950 van Pieter Scheen (1969), gepubliceerd op Ensie.nl.

Niels huwde op 31 juli 1939 met Eduardus Andreas Josephus Antonius Maria van Spaendonck uit Tilburg. Dat was een achterneef uit Tilburg met dezelfde achternaam als haar moeder.

Stamboom Niels van Spaendonck:

Reinilda Carola Anna Maria Dolk was een kind van Henricus Carolus Marie Dolk en Aloysia Christina Vincentius Maria van Spaendonck
Aloysia Christina Vincentius Maria van Spaendonck was een kind van Carolus Gerardus Joseph Maria van Spaendonck en Anna Maria Francisca Houben
Carolus Gerardus Joseph Maria was een kind van Andreas Josephus van Spaendonck en Maria Rosalia Janssens

Stamboom Eduard van Spaendonck (echtgenoot van Niels):

Mr. Eduardus Andreas Josephus Antonius Maria van Spaendonck was een kind van Eduardus Antonius Maria van Spaendonck en Ludovica Antonia Maria Mutsaers
Eduardus Antonius Maria was een kind van Andreas Josephus van Spaendonck en Maria Rosalia Janssens

Bovenstaande gegevens komen uit Stamboom Driessen op Genealogieonline.nl. Online stambonen zijn niet altijd de meest betrouwbare informatiebron, maar de gegevens komen overeen met wat ik elders heb gevonden.

Eduardus van Spaendonck werkte als notaris, en commissaris bij de Tilburgsche Waterleiding Maatschappij en de Tilburgsche Spaarbank. Zijn aktetas is Brabants materieel erfgoed! Meer informatie over zijn leven vind je bij Brabants Erfgoed of Stadsmuseum Tilburg.

Niels van Spaendonck overleed op 27-10-2002. Ze ligt begraven in Tilburg, samen met haar echtgenoot. Bron: Begraafplaats Heike Tilburg

Lijst van kunstwerken

Het oudste werk dat ik online heb kunnen vinden is uit 1972, toen was Niels van Spaendonck ongeveer 62 jaar oud. Hier is de lijst van alle werken die online staan in chronologische volgorde:

Man en vrouw in tuin (1972) Bron: kunstveiling.nl

Wind (1972) Bron: Marktplaats.nl

Klanken (1973) Bron: kunstveiling.nl

Dansende vrouw (1974) Bron: gecachete advertentie op Marktplaats.nl

Tussen takjes (1975) Bron: Museum Van Bommel Van Dam

Oude tekens (1975) Bron: Museum Van Bommel Van Dam

Fantasiefiguren (?) (1975) Bron: Kunstveiling.nl

De koning danst (1977) Bron: Stichting Heden

Meisje (1977) Bron: Invaluable

Winterlandschap (1978) Bron: Museum Van Bommel Van Dam

Boompjes (1978) Bron: kunstveiling.nl

In het bos (1979) Bron: Stichting Kunstcollectie Essent + Enexis

Wording (1979) Bron: kunstveiling.nl

Ampullen (1980) Bron: mijn eigen collectie en kunstveiling.be

De geest uit de fles (1980) Bron: Stichting Kunstcollectie Essent + Enexis

Muzische sfeer (1980) Bron: artpeers.de

Stilleven (1983) Bron: Museum Van Bommel Van Dam

Stilleven in de schemering (1983) Bron: Museum Van Bommel Van Dam

Hemelse boodschap (jaar onbekend) Bron: gecachete advertentie op Marktplaats.nl

Familie

De familie Van Spaendonck is een familie uit de gegoede burgerij uit Tilburg die ook veel kunstenaars heeft voortgebracht. De beroemdste kunstenaar uit deze grote familie is Gerardus van Spaendonck, die aan het hof van koning Louis XVI van Frankrijk werkte als bloemschilder en miniatuurschilder. Een van zijn bloemstillevens hangt in het Rijksmuseum. Het Parool schreef in 2018 een artikel over deze aanwinst. Gerard kreeg geen kinderen, dus Niels van Spaendonck is geen directe afstammeling van hem.

In 2010 was er een gezamelijke tentoonstelling van actieve kunstenaars uit de familie Van Spaendonck in Tilburg. In de Trouw van 3 mei 2010 staat een leuk interview met drie van de exposerende familieleden: Els van Spaendonck, Marie-Colette van Spaendonck en Joris van Spaendonck.

[...] Ruim twintig Van Spaendoncken exposeren in het Duvelhok, een oud fabriekspand in het centrum van Tilburg. Ze stammen uit een familie van welgestelde Tilburgers, die ooit haar kapitaal vergaarde in de wollenstoffenindustrie Daarnaast is de familie al eeuwenlang artistiek. In het Duvelhok hangen kunstwerken van de laatste generatie. De diversiteit is groot: er hangt werk van bejaarde Van Spaendoncken naast tekeningen die een 17-jarige Van Spaendonck maakte op school. Opvallende metalen sculpturen, stillevens van boeken en schilderijen, van mensfiguren met enorme open monden delen de ruimte. Een enorm bloedend hart, uitgevoerd in wol, hangt in een hoek. [...] De familie verwijst graag naar de afstamming van Gerard en Cornelis van Spaendonck: gerenommeerde kunstenaars uit de achttiende eeuw. Gerard schopte het tot hofschilder van de Franse koning Lodewijk XVI. Zijn werk hangt in het Louvre in Parijs. „Ze hadden beiden geen kinderen”, zegt Joris. „Dat heeft een oom van me ooit verteld.”Toch zit kunst ook deze tak van de familie in het bloed, zegt Marie-Colette. Zijzelf groeide op zonder haar kunstminnende Van Spaendonck-vader, die omkwam in de oorlog. Het was niet haar opvoeding die haar aanzette tot beeldhouwen en later tot fotograferen. Haar moeder had er niets mee. „Het is een drang, ik moet me bezighouden met wat mooi is, dingen betekenis geven”, zegt ze. Alle Van Spaendoncken delen dat gevoel voor esthetiek, denkt Marie-Colette, ook de niet-kunstenaars. „Het huis van mijn grootmoeder was prachtig ingericht. Alle dingen waren in balans.”

Tentoonstellingen

Artindex.nl heeft een uitgebreide lijst van publicaties en tentoonstellingen van Niels van Spaendonck. Hieronder noem ik de dingen die ik d.m.v. het internet kon verifieren, op chronologische volgorde:

Uit de Leidse Courant van maandag 23 juni 1969:

Galerie de Sfinx, O.Z. Vooburgwal 241, Amsterdam, exposeert tot zondag 6 juli (1969) schilderijen en grafiek van Niels van Spaendonck

Uit th berichten 34, 12e jaargang, 15 mei 1970:

Agenda: Tentoonstellingen: Galerie Willy Schoots: Niels van Spaendonck, grafiek en tekeningen

Volgens het RKD:

Niels van Spaendonck exposeert in de Schouwburg - Galerie grafiek; Schouwburg Tilburg; 1972-04-14 tot 1972-05-04

Uit het Limburgs dagblad 16-01-1973:

Bonnefantenmuseum: Schilderijen, etsen van Willem van Leusden, etsen van Niels van Spaendonck, tm 28 januari van 10 tot 12 en van 14 tot 17 uur, zaterdags en zondags van 14 tot 17 uur.

Van het Utrechts Archief, in 777-1: Schilder- en tekenkundig genootschap Kunstliefde te Utrecht:

"2. Lijst van gehouden tentoonstellingen over 1807-1977:
1971: "Vrouwen van Kunstliefde", werk van Ton Diekstra, A.L. Boellaard van Heeckeren, Emmy Eerdmans, Antoinette Gispen, Charlotte Groen, Jo de Recht, N. van Spaendonck-Dolk, Eka Thoden van Velzen en Plony van Zuylen, 27 november-18 december.
1974: Grafiek van Niels van Spaendonck, en beeldhouwwerk van Joep Faddegon, 12 januari-3 februari.

Uit Leidse courant 13 mei 1977:

Exposities: De Plataan (stationsstraat zoetermeer) Niels van Spaendonck tm 28 mei werkdagen 13.30-17, vrij 19-21.

Van de website van Eka Thoden:

Utrecht, “Kunstliefde”, duo-tentoonstelling met Niels van Spaendonck, 1978

Uit het cultureel supplement van NRC handelsblad van 19 april 1985:

Putten Gem[eente] Huis: Niels van Spaendonck e.a., tot 5 mei.

Veilingen

In de context van veilingen kwam ik Niels van Spaendonck maar twee keer tegen:

Catalogus van Veilinghuis Peerdeman voor zondag 9 maart 2014:

Niels van Spaendonck (1910), zes kleurenetsen, allen gesigneerd en gedateerd '82, '83 en '84 - Niet verkocht

Catalogus van het Zeeuws Veilinghuis van mei 2025, en een kopie op Invaluable.com:

Isaac Israels (1865-1934) 'Self portrait in his studio', ca. 1931, signed lower left (in the painting on the easel), oil on panel. H. 26.5 cm. W. 35 cm. Provenance: Louise Dolk-van Spaendonck, a painter who was friends with Isaac Israels (Israels probably gave the painting to her); Reinilda van Spaendonck-Dolk, better known as Niels van Spaendonck (also a painter); thence by descent. See 'Lexicon Nederlandse beeldende kunstenaars 1750-1950' by Pieter Scheen, volume MZ, 1970, p. 373 for more information about Louise Dolk-van Spaendonck.

Collecties

Er bevinden zich 5 werken van Niels van Spaendonck in de collectie van Museum Van Bommel Van Dam: "tussen takjes", "stilleven", "stilleven in de schemering", "winterlandschap", "oude tekens". Bron: Museum Van Bommel Van Dam

Stichting kunstcollectie Essent-Enexis heeft "In het bos" en "De geest uit de fles". Bron: Stichting kunstcollectie Essent-Enexis

Stichting Heden, een kunstuitleen in Den Haag, heeft "De koning danst". Bron: Stichting Heden

Stichting Océ Kunstbezit heeft volgens ARTindex Lexicon Online ook werk van Niels van Spaendonck, maar ik kon niet vinden welke.

Eerste kennismaking met een kunstveiling

2025-06-30T00:00:00+00:00

There is also an English translation of this post here: My first experience with an art auction.

Na het lezen van Onschatbare waarde (2004) van Bert Bakker en Johan Bosch van Rosenthal was ik wel benieuwd naar hoe zo'n veiling in het echt verloopt. Het dichtstbijzijnde grote veilinghuis is Veilinghuis Derksen in Arnhem, dus ik had in mei 2025 de kijkdagen van hun eerstvolgende veiling in mijn agenda gezet.

Onschatbare waarde, Bert Bakker en Johan Bosch van Rosenthal, 2004

Kijkdag in Arnhem

Een week voor de veilingdagen bleek ik ook daadwerkelijk tijd te hebben om er langs te gaan. Mooi! Ik kon via de website alvast de hele catalogus van de veiling bekijken. Met een account kon ik alvast een shortlist maken van interessante objecten.

Ik zag veel leuks. Ik ben voornamelijk geinteresseerd in 2D kunst, dus er stonden voornamelijk schilderijen op mijn shortlist, van serieuze olieverf schilderijen tot moderne grafiek.

Gesigneerd G J Hoffmans, platbodem op volle zee met driekleur bij het roer, paneel 20e eeuw, 40x50 cm

Hendrik Valk (1897-1986), Naakt, zeefdruk gedateerd 1963, EA, 47/100, 25,5x18 cm

Keisuke Yamamoto (1961-), "Light Time Silence 24", litho genummerd 3/20, 47x87 cm

Op zondag, de laatste dag van de kijkdagen, belde ik last-minute naar Christiaan om te vragen of hij mee wilde. We gaan normaal gesproken samen graag kringlopen af. Hij had tijd, dus we konden samen in zijn auto carpoolen naar Arnhem. We hebben ons twee uur goed vermaakt met kijken naar meer dan 3000 kavels die stonden uitgestald in het gebouw van Derksen. Drie verdiepingen met schilderijen, keramiek, glaswerk, antieke meubels, prenten, aziatische kunst zoals Chinees porselein en Japanse netsuke, sieraden en horloges, oude Leica fotocamera's, en nog wat "rariteiten" die niet in een categorie vallen.

Ivoren netsuke van een aap, Japan, Meiji periode ca. 1900, 31 mm hoog

Christiaan verzamelt glas en houdt van fotografie, en ik verzamel grafiek. Daarnaast houden we allebei ontzettend van Japanse kunst. We hebben ons in Arnhem uitstekend vermaakt. ;)

Ik vroeg me vantevoren af hoe ze de honderden ingelijste werken gingen tonen die in de veilingcatalogus stonden -- die zou je niet allemaal tegelijk in een veilinghuis kunnen ophangen, daarvoor waren het er teveel. Beneden hingen de meeste olieverfschilderijen, en op de eerste verdieping stonden grote houten stellingkasten met daarin de minder dure grafiek netjes op een rij, zoals in een bibliotheek. Als je bepaalde prenten in het echt wilde zien, moest je het kavelnummer doorgeven aan de vriendelijke meneer achter de balie, en dan werden de werken voor je uit de stellingkast gehaald.

Christiaan en ik zagen allebei wel dingen die ons aanspraken, dus gewapend met een reeks notities gingen we weer terug naar huis.

Voorbereiding

In de hele week na de kijkdagen was er elke avond om 19:00 een live veiling via het internet. Derksen streamt alle veilingen via hun website, met een veilingzaal zonder publiek. Het bieden verloopt volledig via internet. Medewerkers van Derksen zitten wel in de zaal en representeren in real-time via hun laptop de biedingen van het online publiek.

Daarnaast werden de minder bijzondere kavels verkocht via een "timed online veiling", wat ongeveer vergelijkbaar is met een veiling op Ebay: de kavels hebben een duidelijke countdown timer, alle biedingen zijn openbaar, en je hebt lang de tijd (bijvoorbeeld een week) om een bod uit te brengen.

Ik wilde wel meebieden op een aantal werken die ik in het veilinghuis in het echt had gezien, dus ik heb twee avonden meegekeken met de live veilingen. Op maandag werden er alleen antieke meubels geveild, iets waar ik geen directe interesse in had, dus ik heb die veiling gekeken om eens te zien hoe het allemaal werkt.

Om te voorkomen dat ik teveel of te weinig zou gaan bieden, deed ik voor de werken waar ik Serieuze Interesse in had eerst wat vooronderzoek. Wat kost een vergelijkbaar werk van deze maker typisch gezien? Met of zonder lijst? In wat voor staat is het werk? Wat is de oplage? Met deze informatie bepaalde ik hoeveel ik zelf bereid was om er voor te betalen.

Ik maakte ook een Excelsheet met daarin alle biedstappen exclusief en inclusief de veilingkosten van 30%. Hiermee kon ik snel zien wat ik netto kwijt zou zijn bij een bepaald bod. Een bod van 100 euro betekent namelijk 130 euro afrekenen. Omdat het om een percentage gaat, kan het bij een hoger bedrag erg snel gaan met de bijkomende veilingkosten.

Ik zag tijdens de meubelveiling van maandag al dat het snel kon gaan. Er leek ongeveer ~1 seconde latency te zijn bij mijn videoverbinding met de veilingzaal, wat betekende dat je als bieder snel moet reageren. Anders zit er teveel tijd tussen het moment dat de veilingmeester in het echt afhamert, en het moment dat jij op je scherm ziet dat dit gebeurt, en kan het zomaar voorkomen dat je te laat bent met je bod. Met mijn voorbereiding zorgde ik ervoor dat ik een duidelijk plan had en tijdens de veiling zelf zo min mogelijk hoefde na te denken.

Deelname live en timed online veilingen

De schilderijen en grafische werken die live werden geveild werden uiteindelijk te duur voor mij en mijn aanvalsplan, waarna ik heel braaf stopte met bieden. Maar de hilarische glazen tijger die in de vitrine stond, naast de glaskunst die Christiaan uitvoerig had bekeken tijdens de kijkdag, bleef keurig binnen mijn budget. Deze ga ik komende woensdag ophalen in Arnhem.

Polychroom glazen beeld van een zittende tijger, 27,5 cm hoog

Oeps, naast grafiek en keramiek heb ik nu dus ook mijn eerste glazen object.

Bij de timed veilingen had ik ook verschillende ervaringen. Ik vond het fijn dat hier in ieder geval wat minder tijdsdruk achter zat. Sommige leuke werken stegen erg snel in prijs, dus daar haakte ik op een gegeven moment af. Andere werken spraken mij erg aan, terwijl er juist heel weinig interesse was van andere veilingdeelnemers. Uiteindelijk kocht ik nog 4 ingelijste werken voor 5, 5, 15 en 15 euro in de timed veilingen. Twee aquarellen van Henri Herman L. Tieland, een Chinees/Japanse inkttekening van pruimenbloesem en een kleurrijk werk van Dirkjan Ribbeling.

Sumi-e schilderij van pruimenbloesems

Conclusies

Ik weet niet of ik nog in de toekomst vaak wil deelnemen aan live veilingen. Ik vind het concept van geld uitgeven onder tijdsdruk heel vervelend. Daarnaast moet je soms best lang wachten op de kavelnummers die je interessant vind. Het is onmogelijk om exact in te schatten wanneer kavels uit jouw shortlist langskomen in de live veiling, en de eerste keer dat ik dit probeerde, was ik meteen te laat. Het verschilt namelijk heel sterk hoe lang de veilingmeester doet over 1 kavel. Bij sommige kavels is het al snel duidelijk dat er niemand interesse heeft, en is de veiling na 30 seconden voorbij. Bij andere kavels gaat het bieden lang door. Zo ging het bieden op een prachtige art nouveau kast door tot wel 1400 euro.

Het viel me ook op dat ik totaal niet in kon schatten wat de uiteindelijke veilingprijs ging worden voor de kavels. Een litho van Escher met serieuze waterschade werd verkocht voor 650 euro. Een rijkbewerkte Indonesische kris (een soort kort zwaard) met schede werd verkocht voor maar 45 euro.

Maurits Cornelis Escher (1898-1972), getiteld "Orde en Chaos", kompasroos, litho 1955, 40x38 cm (waterschade aan rand onderzijde en rechterzijkant)

Het zal iets te maken hebben met vraag en aanbod, afwezigheid/aanwezigheid van specialistische kennis onder de veilingdeelnemers, de stand van de economie, de gewildheid van sommige kunstenaars, tijdsperiodes of stijlen in de huidige kunstmarkt, en nog allerlei andere aspecten.

Deze "willekeur" is ook wat mij zo fascineert aan veilingen en (de economie van) de kunstwereld.

Ik heb het gelukkig heel makkelijk. Ik hoef alleen maar 3 vragen te beantwoorden: vind ik het leuk, past het qua grootte in mijn huis, en is het binnen budget?

Er is in ieder geval geen enkele correlatie tussen hoe mooi of interessant ik iets vind en hoe duur het is.

My first visit to an art auction

2025-06-30T00:00:00+00:00

This is an English translation of my Dutch post Eerste kennismaking met een kunstveiling.

After reading Onschatbare waarde ("Priceless Value") (2004) by Bert Bakker and Johan Bosch van Rosenthal, I wanted to experience an art and antiques auction in real life. The nearest major auction house is Veilinghuis Derksen in Arnhem, so last May I marked the viewing days of their next auction in my calendar.

Onschatbare waarde ("Priceless Value"), Bert Bakker and Johan Bosch van Rosenthal, 2004

Viewing day in Arnhem

A week before the auction days, it turned out I actually had some time to visit. Great! I could already view the entire auction catalog on the website. With an account, I could create a shortlist of interesting objects.

I saw many appealing items. I'm primarily interested in 2D art, so my shortlist mainly consisted of art on paper, from serious oil paintings to modern art prints.

Signed G J Hoffmans, flat-bottomed boat on open sea with tricolor at the helm, panel 20th century, 40x50 cm

Hendrik Valk (1897-1986), "Naakt" ("Nude"), screen print dated 1963, artist's proof, 47/100, 25.5x18 cm

Keisuke Yamamoto (1961-), "Light Time Silence 24", lithograph numbered 3/20, 47x87 cm

On Sunday, the last day of viewing, I called Christiaan last-minute to ask if he wanted to come with me. We enjoy going to thrift stores together. He had time for the viewing, so we could carpool in his car to Arnhem. We spent two entertaining hours looking at more than 3000 lots displayed in Derksen's building. Three floors with paintings, ceramics, glassware, antique furniture, prints, Asian art such as Chinese porcelain and Japanese netsuke, jewelry and watches, old Leica cameras, and some "oddities" that don't fit into a category.

Ivory netsuke of a monkey, Japan, Meiji period ca. 1900, 31 mm high

Christiaan collects glass and loves photography, and I collect graphics. We also both absolutely love Japanese art. We had an excellent time in Arnhem. ;)

I wondered beforehand how they would display the hundreds of framed works listed in the auction catalog -- it wasn't possible to hang them all in the auction house, there were just too many. Downstairs most oil paintings hung on display, and on the first floor stood large shelves with the less expensive framed prints neatly arranged in rows, as in a library. If you wanted to see certain prints, you only had to give the lot number to the friendly gentleman behind the counter, who would then retrieve the works from the shelves.

Both Christiaan and I saw lots of things that we liked, so after making some notes and taking photos, we headed back home.

Preparation

Throughout the week after the viewing days, there was a live auction every evening at 7:00 PM via the internet. Derksen livestreams all auctions through their website, from an auction room without in-person bidders. Bidding takes place entirely via the internet. Derksen employees do sit in the room and represent the online audience's bids in real-time via their laptops.

Additionally, the less special lots were sold via a "timed online auction," which is roughly comparable to an eBay auction: the lots have a clear countdown timer, all bids are public, and you have plenty of time (for example, a week) to place a bid.

I wanted to bid on several works I had seen in person at the auction house, so I watched the live auctions for two evenings. On Monday, only antique furniture was auctioned, something I had no direct interest in, so I watched that auction to learn the structure of an auction, and how bidding works on the online platform.

To prevent myself from bidding too much or too little, I did some preliminary research on the works I was Seriously Interested in. What does a comparable work by this maker typically cost? Is it framed or not? What condition is the work in? What is the edition size? With this information, I could determine how much I was willing to pay for it.

I also made an Excel sheet with all bidding steps excluding and including the 30% auction costs. This allowed me to quickly see what I would pay net for a certain bid. A bid of 100 euros means paying 130 euros to the auction house. Because it's a percentage, the net price can quickly increase at higher bid amounts.

I already saw during Monday's furniture auction that bidding happened really fast. There seemed to be about ~1 second latency in my video connection to the auction room, which meant you had to react quickly as a bidder. Otherwise, there's too much time between when the auctioneer actually hammers down and when you see this happen on your screen, and it could easily happen that you're too late with your bid. With all my preparation, I had a clear plan, so I needed to think as little as possible during the auction itself.

Participation in live and timed online auctions

The paintings and graphic works in the live auction ultimately went over my budget, after which I very sensibly stuck to my plan the campagne and stopped bidding. But the hilarious glass tiger that stood in the display case, next to the glass art that Christiaan had extensively examined during the viewing day, stayed neatly within my budget. I'll be picking it up in Arnhem next Wednesday.

Polychrome glass sculpture of a sitting tiger, 27.5 cm high

Oops, besides some prints and ceramics, I now also have my first glass object.

I also had mixed experiences with the timed auctions. I liked the lack of time pressure here. Some nicer prints rose very quickly in price, so I dropped out at some point. Other works really appealed to me, while there was very little interest from other bidders. Eventually, I bought 4 more framed works for 5, 5, 15, and 15 euros in the timed auctions. Two watercolors by Henri Herman L. Tieland, a Chinese/Japanese ink drawing of plum blossoms, and a colorful work by Dirkjan Ribbeling.

Sumi-e painting of plum blossoms

Conclusions

I don't know if I'll want to participate in live auctions often more in the future. I find the concept of spending money under time pressure very unpleasant. Live auction seem to involve a lot of waiting around until the lot number come up that you find interesting. It was impossible to estimate exactly when lots from my shortlist would come up in the live auction. The first time I tried to join the auction just in time for a specific lot number, I was immediately too late. The time the auctioneer spends on each lot varies significantly. With some lots, it quickly becomes clear that no one is interested, and the auction is over after 30 seconds. With other lots, the bidding goes on for a long time. For instance, the bidding on a beautiful art nouveau cabinet went up to 1400 euros.

I also noticed that I couldn't estimate at all what the final auction price would be for the lots. A lithograph by M.C. Escher with serious water damage was sold for 650 euros. An elaborately decorated Indonesian kris (a type of short sword) with sheath was sold for only 45 euros.

Maurits Cornelis Escher (1898-1972), titled "Orde en Chaos" ("Order and Chaos"), compass rose, lithograph 1955, 40x38 cm (water damage at bottom edge and right side)

It probably has to do with supply and demand, absence/presence of specialist knowledge among auction participants, the state of the economy, the desirability of certain artists, time periods or styles in the current art market, and all sorts of other aspects.

This "randomness" is also what fascinates me so much about auctions and (the economics of) the art world.

Fortunately, I am in a very straightforward situation. When buying art, I only need to answer 3 questions: do I like it, does it fit in my house, and is it within budget?

In any case, I've found that there is no correlation whatsoever between how beautiful or interesting I find something and how expensive it is. :)