Does The New GPT-4 Deep Research Tool Work Well? Here’s What I Found Out
A deep dive on this LLM tool's data visualization research and analysis capabilities
As a data science professor, I have been using GPT-4 daily for more than 2 years to help me with my work.
And wow, it really is an outstanding and indispensable tool for many data science and writing tasks.
And, oh my god is it ever terrible at some other tasks — like performing any kind of deep research.
Until recently. Those of you who have a GPT-4o paid account may have noticed a new feature:
GPT-4o now has the ability to perform “deep research” on most any topic or domain. This tool is an attempt to fill the vast hole in GPT-4 LLM research capabilities that have been there since this new tool was rolled out a few years ago.
Almost unanimously it seems, users have been complaining about the complete and utter lack of ability of the tool at providing consistent and accurate research — on any topic.
The Deep Research tool attempts to fill this void.
So what can we use the GPT-4o Deep Research tool for? Any research where you are trying to find more information on a particular research topic — or even across domains where you want disparate knowledge to be brought together and synthesized.
So is this new Deep Research (DR) tool any good at doing this research? Has it filled this gaping chasm in writing and research capabilities?
Here’s what I found out.
GPT-4o Deep Research — A Practical Example
For me, the best way to understand the capabilities of a new tool is to take it for a test drive. Let me show you an example of how I use this tool in my day-to-day work.
Now, I teach a number of courses and workshops on data visualization and data storytelling. One of the author/researchers I have been focused on lately is Dr. W.E.B Du Bois.
Many of Dr. Du Bois’ data visualization projects (circa 1900) vividly documented the early Black American journey from slavery to emancipation.
There are lots of different sources of information on Dr. Du Bois. What I want is to find an interesting angle to synthesize his work.
Now before we start prompting, the big disclaimer here (from me and also from the tool itself) is that the research is only as good as the information that is out there. If you are researching a niche topic where there is very limited access to publicly available information, then this tool will not work miracles.
But if we are researching a topic that has publicly available information, then we can enable the GPT-4o Deep Research Tool and start asking questions. Let’s do this:
To start with, I am asking a fairly general question. As I have found with extensive prompting of this LLM tool over time, it is always best to ask for a list of choices.
In my prompt, I ask the GPT-4o Deep Research tool to give me 5 essay topics related to Dr. Du Bois’s work — the DR tool responds with a very useful query on context and audience:
Great! This tool is actually thinking. Next, I povide more details on the topic, to help “sharpen the knife” so to speak. How does DR respond? After a decent period of time (over 6 minutes):
The list of topics that the DR tools provide are actually pretty darn good. Without going in to too much detail, the topics suggested are:
From Numbers to Narrative: Du Bois’s Emotional Data Storytelling
Data Visualization as Activism: Du Bois’s 1900 Charts and Today’s Social Justice Graphics
Making Data Accessible: Clarity and Design from Du Bois to Modern Infographics
The Art in the Data: Du Bois’s Visual Innovations and Their Influence on Today’s Graphics
Humanizing Data: Du Bois’s “Data Portraits” and Ethical Storytelling in Visualization
Each topic has a 250 word write-up on what content could be discussed in a full essay, along with a list of references.
Let’s focus in on one of the topics and see how the DR tool performed:
A useful discussion to have around Dr. Dubois is how his innovative work has influenced today’s graphics. And from a data storytelling perspective, how Dr. Dubois blended art with information. Today’s modern data storytellers use creative graphics to communicate meaning emotionally.
Great! Now on to the last step — let’s ask the tool to write an informative essay where the goal is to create an article that synthesizes Dr. Du Bois’s work and that may be used as part of the curriculum for a workshop or course.:
The response from the DR tool is another set of questions:
Thanks! To make the article suitable for a Medium publication on data visualization, could you please confirm:
1. Should the tone be more academic, journalistic, or conversational/informal?
2. Do you want me to include direct references or quotes from Du Bois’s original work or scholars?
3. Should I recommend or include specific modern data visualizations (with links or examples) that echo Du Bois’s style?
4. Are images or charts (from Du Bois or modern creators) expected to be embedded or described in the article?
These are actually some very good questions. My response to these questions is to make the article more “informal” without direct quotes (as it is more informal) and that it should include specific modern examples and embedded charts and images from both sources.
The response from DR:
Now we just have to wait to see what the DR tool can do!
After 11 minutes, 35 sources, and 63 searches, the DR tool is finished! Here is a screenshot of the first section of the article it wrote:
OK, great, we have a reasonable title and subtitle. Upon reading the introduction — it is neat and concise and most importantly, accurate. My experience with GPT-4 writing tools up to this point is that they are consistently inaccurate. As an example, here is a highlighted source:
The link to this source works, and the reference link is working — and accurate! Going through the entire article — all of the references are from reasonable source and all of the links work.
GPT-4o Deep Research — The Good and The Bad
So what I do like with the GPT-4o DR tool is that the generated article in this example is:
Well written: this is not new to the DR tool. Most all modern LLMs can write well structured and grammatically accurate content.
Well researched, well referenced: The DR tool takes a significant amount of time (11 minutes in the above example) to be thorough.
Accurate: All of the references are from reputable resources and all the links work.
This is a very good article, and will work great as a reference article to be included as resource material for a introductory course or workshop.
However, there are still some things that are just not right:
The references are all in-line: this is not a common approach for an informal Medium-written article. I want these removed.
There is only 1 actual data visualization shown in the article. For an article that is showcasing “art” in storytelling — this is not adequate.
The writing style is, well, still too “AI-ish” for my liking. It’s too clean, it’s too vague, and it still lacks any sort of emotional appeal.
Unfortunately point 2 and 3, which are common themes with AI-generated content, are not removed with this tool. There is no doubt that they are reduced, but the DR tool still falls off the guardrails put in place.
So let me try one last prompt to “clean it up” a little:
The DR tool verifies my request by asking if I would like a more “personal” tone and if I want the actual images inline. After responding “yes” to both queries, the article is written.
The final result can be viewed and read: HERE.
In Summary…
The GPT-4o Deep Research tool is a massive improvement on previous writing tools available in GPT-4o.
Its ability to synthesize and summarize complex material into a distillable format is really good. I am already using it to help me develop academic curriculum and to supplement my workshops with new angles and examples.
However, the tool is nowhere near perfect. It still requires careful prompting to start it on the right track, and it still needs adjustment prompts to fill in the holes when it “gaps out” on the guardrails that have been put in place.
It’s definitely worth a test drive — let it do some research for you — and let me know how it goes!