I tested several tools and tried different prompting techniques. After months of experiments and dozens of articles written with the help of large language models (LLM), I’ve decided to stick with good-old writing by hand.
However, ChatGPT has changed my approach to writing for the better, even if it is not the tool I use most often. I don’t think the technology is ready to replace professional writers. But that can change very soon, which is why I’m also preparing for what is to come.
ChatGPT on autopilot
Seeing all the examples of ChatGPT writing full articles, I decided to give it a shot. I experimented with ChatGPT 3.5, GPT-4, Claude, and Llama 2.
My observation was that advanced LLMs can write a decent article on a very general topic with a simple prompt. I usually don’t like the tone and style, but the content is not bad. But as a professional writer, most of my work is producing original content that I haven’t found elsewhere. And the models can make false statements, so I would only advise using them if you can fully validate their responses.
Next, I tried using LLMs to write about recent events. I tried writing with Bing Chat and GPT-4 with web browsing enabled. I either queried for news or gave them links to news reports and asked the model to summarize them or write an article based on them.
I did not like the results. The articles they generate don’t follow a logical flow and I ended up restructuring the text.
The models sometimes repeat facts at different places or place them where they don’t make sense. And since I always had the suspicion that the LLM might be hallucinating, I would read the reference articles and verify the veracity of the model’s article. This ended up making the writing cycle longer and more frustrating than doing it without AI.
ChatGPT as a copilot
I gave up on using ChatGPT as an automated article writer and tried to see if I could use it as a copilot that helps me flesh out the outline of an article. I started by gathering and structuring my thoughts as short statements in Word. I then gave the entire outline to GPT-4 and prompted it to flesh it out along with additional instructions such as word count, style, tone, etc.
The results were much better. There were fewer hallucinations, especially when the word count was much closer to the actual length of the outline. The more room I gave the model to elaborate, the more the content became repetitive. And GPT-4 likes to summarize and repeat all its talking points at the end of the text it generates.
Another observation was that the longer the outline was, the less accurate the output became. For example, the model often missed the word count or simply removed important parts of the outline from the finished article. But the results were much better than previous experiments.
I started iterating on this process. First, I decided to break down the outline into smaller parts, each representing a section of the article. I gave each of them to the model as a separate prompt, along with additional instructions such as word count.
The results were much better, and I realized that with thorough editing, the copy was acceptable. I added more instructions in my prompts, such as “use active voice” and “write concise sentences.” I also changed my writing process to spend a lot more time on my outline. As I did research and interviews for my articles, I started putting thoughts, facts, and quotes into my outline document. Then I reviewed the outline, trimmed the excessive details, reorganized the facts, and trimmed some more.
I continued to optimize my prompts, adding instructions such as “keep the quotes as is” when there were quotes in my outline. The writing process became a matter of feeding the outline sections to the model, adding instructions, and editing the final copy. I was still doing a lot of editing, but the process was more pleasant and saved me some time.
You can see an example of the prompting here, and the final article here. I also used ForeFront to create a GPT-4 assistant, which made the prompting much easier.
Back to the basics
At this point, I realized that my outlines were so detailed that fleshing them out manually would be much faster than letting ChatGPT or Claude do the writing and then editing the results myself.
I eventually decided to return to manual writing. But I preserved the process of writing detailed outlines before writing the first draft, especially if I was doing research. It has made my writing more concise, my editing faster, and the entire process more pleasant. Now, I write and iterate on my outline several times before writing my first draft.
I also use it to figure out scientific topics during my research. For example, I’ve reviewed many research papers in the past months. When I find inconsistencies in the nomenclature between papers, GPT-4 can help clarify things. (Again, note that I only use GPT-4 as a source of knowledge for things that I already know a great deal about and I’m sure the model has been trained on thoroughly. For example, GPT-4 has great knowledge about the workings of transformer models—a topic that I’ve written about extensively. It is a good tool to help figure out the ambiguities about new papers on transformer architectures.)
Will LLMs replace professional writing?
Subscribe to continue reading
Become a paid subscriber to get access to the rest of this post and other exclusive content.