Hacker Group or Grunge Band: An AI No-Code Case Study

The Setup

I had an epiphany yesterday while reading this article, which is that the names of Chinese hacker groups sound strangely like the names of 90’s grunge bands. For example, I could totally see “Back Door Diplomacy” playing at The Crocodile circa 1991. (Although to be fair, “Volt Typhoon” conjures more of an 80’s hair metal vibe. They might have opened for Poison.)

So I thought this sounded like a great opportunity for game, where given a name you have to guess whether it refers to a Chinese hacker group or a 90’s grunge band. This idea also makes me feel a little nostalgic, because this is the kind of game my brother and I would often write in BASIC on the IBM PC when we were about 9 and 7 years old. But in 2023 instead of writing this myself, I’m going to let ChatGPT do it for me.

And when I say “do it for me”, I mean, “really do it for me”. I don’t want to write a single line of source code. Much has been made about how AI is going to replace programmers as everyone will be able to generate software with prompts. Many of the people saying this are the programmers themselves. But I wonder how many of them have actually tried to generate a non-trivial and useful working program (not just a function) without getting their hands at least a little bit dirty fixing silly errors that tend to pop up in ChatGPT-generated code.

So come with me on this journey.

Command-Line

Like all good ChatGPT stories, this one starts with the prompt:

“Write a command-line program in Python which is a game to determine whether a name belongs to a Chinese hacker group or a 90s grunge band. Display a random name from a list of 5 Chinese hacker groups and 5 90s grunge bands and ask the user for their guess. Repeat this for 5 times and calculate and print a score.”

As requirements and user stories go, that’s (sadly) probably better than ones a lot of us are used to working with. It led ChatGPT to generate this initially-plausible-looking thing:

import random

def get_random_name():
    hacker_groups = [
        "APT1",
        "Darkhotel",
        "Comment Crew",
        "Deep Panda",
        "Cicada",
    ]
    grunge_bands = [
        "Nirvana",
        "Pearl Jam",
        "Soundgarden",
        "Alice in Chains",
        "Stone Temple Pilots",
    ]
    all_names = hacker_groups + grunge_bands
    random.shuffle(all_names)
    return all_names.pop()

def play_game():
    score = 0
    for _ in range(5):
        name = get_random_name()
        print("Guess whether '{}' is a Chinese hacker group or a 90s grunge band.".format(name))
        guess = input("Enter 'hacker' for Chinese hacker group or 'grunge' for 90s grunge band: ").lower()

        if (guess == "hacker" and name in hacker_groups) or (guess == "grunge" and name in grunge_bands):
            print("Correct!\n")
            score += 1
        else:
            print("Incorrect. The correct answer was '{}'.".format("hacker" if name in hacker_groups else "grunge"))
            print()

    print("Game over! Your score is {}/5.".format(score))

if __name__ == "__main__":
    play_game()

Of course, “plausible” isn’t the same thing as “correct”, and when I run this first attempt, I get:

Guess whether 'Pearl Jam' is a Chinese hacker group or a 90s grunge band.
Enter 'hacker' for Chinese hacker group or 'grunge' for 90s grunge band: hacker
Traceback (most recent call last):
  File "hackers_vs_grunge.py", line 39, in <module>
    play_game()
  File "hackers_vs_grunge.py", line 29, in play_game
    if (guess == "hacker" and name in hacker_groups) or (guess == "grunge" and name in grunge_bands):
NameError: name 'hacker_groups' is not defined

If you actually read the Python code, it’s obvious what’s going on, with play_game() trying to reference the name arrays that are locally defined in the other function. But today I'm pretending I can't figure that out for myself, so I just gripe back to ChatGPT about the error. And to its credit, it correctly identifies and corrects it.

Iterating

Now I have the basic requirements done in just two chat messages. Maybe it’s time to start preparing for a new career. But I’m still not entirely satisfied with this program, so I’m going to keep revising it.

First, “APT”-something is a pretty obvious hacker group name. I also happen to know that a lot of them go by “Unit”-something. I’d like to leave them out of the question list.

“Exclude any hacker groups starting with ‘APT’ or ‘Unit’“

Here’s where I got into a bit of a circular conversation with ChatGPT. Its first try at this requirement was to leave “APT1” in the code, but add a conditional to filter out the undesired names inline. Too complicated, and in effect, this left me with only four possible hacker group name prompts and violated the spirit of the original requirements. I should point out that if I wasn’t reading code at this point, this bug would have been very hard to detect via black-box testing.

“Why not just leave them out of the code altogether?”

Obligingly, ChatGPT now removed “APT1” but didn’t replace it with a new hacker group name.

“Now I only have 4 hacker groups, I want 5.”

I felt like ChatGPT was being a bit stand-offish with me at this point. Now it added a fifth item — “NewHackerGroup1”. I’m pretty sure there’s no actual Chinese hacker group given that name. So I asked it to generate a new real one.

    hacker_groups = [
        "Shell Crew",
        "Emissary Panda",
        "Energetic Bear",
        "Buckeye",
        "Gothic Panda",
    ]

Ok, progress.

The next thing I wanted to change was that the grudge bands were too obvious. Anyone who was even alive in the 90’s has heard of all of those bands. Probably has several megs of MP3s on their phone in fact.

“Chose different grunge bands that are more obscure”

    grunge_bands = [
        "Mudhoney",
        "Screaming Trees",
        "Mother Love Bone",
        "Tad",
        "L7",
    ]

Slight improvement, As someone who lived most of his life in Seattle, I still don’t think those bands are all that obscure, but let’s roll with it and play our game.

% python hackers_vs_grunge.py

Guess whether 'Buckeye' is a Chinese hacker group or a 90s grunge band.
Enter 'hacker' for Chinese hacker group or 'grunge' for 90s grunge band: hacker
Correct!

Guess whether 'Shell Crew' is a Chinese hacker group or a 90s grunge band.
Enter 'hacker' for Chinese hacker group or 'grunge' for 90s grunge band: grunge
Incorrect. The correct answer was 'hacker'.

Guess whether 'Mother Love Bone' is a Chinese hacker group or a 90s grunge band.
Enter 'hacker' for Chinese hacker group or 'grunge' for 90s grunge band: hacker
Incorrect. The correct answer was 'grunge'.

Guess whether 'Tad' is a Chinese hacker group or a 90s grunge band.
Enter 'hacker' for Chinese hacker group or 'grunge' for 90s grunge band: hacker
Incorrect. The correct answer was 'grunge'.

Guess whether 'Emissary Panda' is a Chinese hacker group or a 90s grunge band.
Enter 'hacker' for Chinese hacker group or 'grunge' for 90s grunge band: grunge
Incorrect. The correct answer was 'hacker'.

Game over! Your score is 1/5.

Nice, got this done. But if my brilliant insight and game are going to make me Internet-famous, I’m going to need this in the form of a web application.

Web Application

“Change this into an application using HTML and JavaScript”

ChatGPT dutifully generated HTML and a corresponding JS file with a fairly reasonable UI for this game:

The page didn’t work at all — when it started it didn’t post a question until you submitted a guess, and after the second guess the game just ended. Since the web page at least looks directionally correct, I still felt like it might be productive to start revising it by describing these bugs to ChatGPT.

ChatGPT got the first question to start showing up. But the game got stuck after the third question. It took four attempts at, “The game gets stuck after the third question”, but then I finally got to my working game.

Maybe not the best UI, though. Let’s fix that.

“Change it so that instead of typing "hacker" or "grunge" in a textbox, I click either a "hacker" button or a "grunge" button”

ChatGPT was able to pull this off, but left another bug behind where if you kept clicking buttons after the game was over the score kept incrementing. That one was quite easy for ChatGPT to fix. But even better was:

“How about just hide the buttons when the game ends, and have a new button that restarts the game”

On the first try, that didn’t do anything at all — make things worse, in fact — because it only regenerated the JS file and not the HTML where the “restart” button should have been defined. Another prompt, another fix.

Now with some testing I observed that the questions would potentially repeat inside the same game, which isn’t great. (The command-line application had this problem, too.) But again this one was fixable with a single chat message.

The UI is kind-of plain, so let’s see if ChatGPT can also serve as our UI designer.

“Give the web page some styling that evokes the grunge aesthetic”

I love the first part of ChatGPT’s explanation of what it claims to have done with this instruction, specifically, “To evoke the grunge aesthetic, we can use a combination of dark colors, distressed textures, and a vintage feel.” At the end of the day, I’m not sure it really accomplished this task well, but I’ll let you decide. Either way, it’s at least a somewhat less boring UI.

The Punchline

My mission was technically…a success:

By doing nothing but a series of prompt revisions with ChatGPT, I was able to generate a working, “useful” program without directly touching any source code.

I still feel like I could have cranked it out faster by hand myself, but then again, as an experienced programmer familiar with many different languages and types of applications, I’m not really the target persona for no-code tools.

On the other hand, even on a problem like this it is clearly powerful to me as a programmer to use the AI as a tool, with prompts like “Declare a const javascript array where the 20 members are all name of Chinese hacker groups and none of them start with "APT" or "Unit"“ for example, to significantly speed up my own work. In other words, pretty much the way that most engineers are using ChatGPT and AI coding assistants today.

In spite of