Great post! I did a similar experiment with ChatGPT, on exactly the same topic: Can you rely on ChatGPT advice fro weight loss? The short answer is no. The long answer is complicated. The user really need to know how to prompt ChatGPT properly to avoid wrong answers. If the users don't know some basics, and cannot formulate their prompts skillfully enough, ChatGPT advice for weight loss would actually lead to weight gain.

Please read my case study co-authored with ChatGPT here https://www.amazon.com/dp/B0BX75GRQJ

The last chapter shows how ChatGPT admits its mistakes and learns to retrieve reliable information guided by skillfully designed prompts.