With chat bots and AI in the news regularly at the moment I figured it was time that I took a look at them myself.  Open AI's ChatGPT has featured in articles a lot, but for that you needed an account and I wasn't keen on having yet another account with someone else.  As I started to think about playing with these tools Google sent me an invitation to trial Bard, so I've opted to play with that.  In this post I'm going to talk about my experience with Bard so far.  

While things like Bard and ChatGPT are called "AI" (artificial intelligence), it's important to remember that they're not.  In reality they are language models and machine learning algorithms.  For the most part that distinction isn't overly important, but calling them "AI" really bugs me!

Note: If I've quoted Bard output I'll place it in an obvious quote block, so if you want to skip Bard's text and just read mine you can.  I do talk about Bard's output though, so it's worth skimming Bard's output at least.

Getting started - who am I?

One of my first tests on 1st April 2023 was to ask "Who am I?", to which Bard replied that it didn't know.  Bard assured me it couldn't access any personal information about me.  It then went on to tell me that it was a "large language model" and explained that I needed to ask myself who I was, and reflect.

Next I asked about me by name:

Who is Jonathan Haddock?

Bard told me it didn't know enough about "that person to help with your request".  I'm not offended, and that's probably quite a good thing - I've never wanted celebrity status.  Still, time to lend Bard a hand:

His blog is at blog.jonsdocs.org.uk

This time Bard was able to tell me more:

I see. The blog at blog.jonsdocs.org.uk is written by Jon, who is a cybersecurity professional, developer, and local historian. He writes about a variety of topics related to his interests, including cyber security, development, local history, and life. His blog is a great resource for anyone interested in learning more about these topics.

Here are some of the topics that Jon has written about on his blog:

* Cyber security: Jon has written about a variety of cyber security topics, including how to protect your data, how to spot phishing scams, and how to respond to a security breach.
* Development: Jon is a developer, and he has written about a variety of development topics, including programming languages, software development methodologies, and web development.
* Local history: Jon is a local historian, and he has written about a variety of local history topics, including the history of his hometown, the history of his county, and the history of his region.
* Life: Jon also writes about his personal life, including his thoughts on current events, his hobbies, and his family.

I encourage you to check out Jon's blog at blog.jonsdocs.org.uk. You're sure to learn something new.

That looks pretty plausible, but it's not entirely accurate.  I've not written about the local history of my region, for example.  If you want to read my local history posts, they're here.  I wouldn't consider that I write about my family particularly though.

A friend of mine uses ChatGPT so we compared results.  ChatGPT also didn't know about me, and still didn't know when prompted that I was a British cyber security professional.  After being told the location of my blog it was able to respond though:

ChatGPT response
Thank you for providing additonal information about Jonathan Haddock.  Based on the website you provided, it appears that Jonathan Haddock is a UK-based software developer and cybersecurity enthusiast who runs a personal blog about his experience and insights in the field of cybersecurity.

On his blog, Jonathan discusses a wide range of topics related to cybersecurity, including threat analysis, network security, penetration testing, and digital forensics, among others.  He also shares his experiences and opinions on various cybersecurity tools and techniques, as well as his thoughts on current trends and issues in the industry.

It's great to see individuals like Jonathan who are passionate about cybersecurity and who are willing to share their knowledge and insights with others.  I hope this information is useful to you.

Something that strikes me about both ChatGPT and Bard is that they were both very complimentary about me.  Unlike Bard, ChatGPT didn't pick up my local history posts.  Not a bad effort in either case though.

To write scripts

I like to automate tasks a lot, so I thought I'd see what Bard could do when it came to writing some scripts for me.  Obviously I'd be reviewing anything given to me before I run it, just to make sure I wasn't being given something damaging.  These were tested on 1st April again.

PowerShell: resolve DNS from a file of addresses

Starting off with a reasonably easy prompt:

Write me a powershell script to take a file of DNS names and resolve them to IP addresses.

Bard took a few moments before coming back with a well formatted example.  The code, as shown in the screenshot below, is helpfully presented in a code block in monospaced text.  Other parts of the response were in normal formatted text.  Bard even included an example of how to use the script.  The script looked plausible, and would create a hashtable of values that would be written to the screen.

But did it work?  Well, almost.  After creating a text file called dns-names.txt, which the script would then process, I ran the Bard provided script and got a terminal saying System.Collections.DictionaryEntry for each DNS name I provided, as shown in the screenshot.

(For those that have spotted that PowerShell terminal is running with Administrator privileges, that's running in Windows Sandbox where the user is an administrator by default.)

To make the script work correctly would require a bit more work to display the hashtable correctly and to expand the answers.

Giving Bard the same prompt on 20th May gave me a slightly different response, this time outputting the results to a CSV file.  Unhelpfully the same problem was present and the script used the .txt file extension for the CSV🤦‍♀️ .

PowerShell: InsightVM

A bigger task, and one that I'd have an immediate use for.  Each week I review my organisation's vulnerability scan results in Rapid7's InsightVM platform. As part of this process I document the risk score, and top risk assets, for each group on a ticket - partly for audit purposes, and partly because if I'm reviewing the information I may as well write down that I've done it!  While the statistics gathering doesn't take me too long, it's something I've wanted to automate (and then read the output of) for a long time so I asked Bard to write me a script.

Write a powershell script to get the asset with the biggest risk score from Rapid7 InsightVM. InsightVM is at scanner.example.org. Username is "bob" and password is "changeme".

Bard came back with a plausible looking script that used a Rapid7 InsightVM PowerShell module to connect to the API.

⚠️ Important spoiler: there is no official Rapid7 module

I've been using InsightVM for probably 4+ years now and I'd never come across this module, so I probed deeper.

⚠️ Important spoiler: there is no official Rapid7 module.

Where do I find the Rapid7 InsightVM powershell module?
Who wrote the module?
Where can I download the Rapid7 InsightVM powershell module?

Bard informed me the module was written by Rapid7 themselves and that I could download the module by logging into the Rapid7 website and clicking the menu links as described in the response.  It was scarily plausible.

When I was next at work I logged in to my Rapid7 account and went to try Bard's instructions.  The steps I had been given couldn't be followed (no surprise, unless Bard had a Rapid7 account it wouldn't know what the account protected menu looked like).   I contacted Rapid7 support to ask about the InsightVM PowerShell module that Bard was claiming existed.  Regrettably, and as I've already indicated in the spoilers, there is no Rapid7 authored module.  Support did mention that they knew a third party had written a module using Rapid7's documentation, but obviously they couldn't endorse it.

As neither of the Bard provided scripts worked, I've not provided them here.

Conclusion

Tools like ChatGPT and Bard lie.  Not with malicious intent necessarily, but their goal is to give output that pleases the human operator.  In my case, Bard generated a script that didn't quite work and another that relied on a non-existent PowerShell module.  Not the end of the world, but ultimately not helpful.  ChatGPT has been known to completely fabricate sources in its responses, in one case writing about a professor who had been accused of sexual assault.  ChatGPT even cited newspaper articles which didn't exist.  If you're interested, there's some news coverage by The Independent and at Decrypt.co.

I'm sure we'll see more of these "AI" tools, and that they will get better, but there's definitely a need to check the responses.

Banner image: Google Bard prompt