Limitations
Authors: Gauthier Salavert, Chiara Semenzin (PhD)
Unali is not a diagnostic tool. It helps with, but does not fully automate, CAM therapy creation.
Unali is an early product using early technology, attempting to help with complex topics. Unali recommendations should be taken as a starting point.
Unali is only as good as the research and crowdsourced knowledge it uses. While we think researchers are a very careful and rigorous group on average, there is research with questionable methodology and even fraud. Unali does not yet know how to evaluate whether one paper is more trustworthy than another, except by using some imperfect heuristics like citation count, journal, certain methodological details (sample size, study type, etc.) all of which are aggregated into a score for convenience. We’re actively researching how best to help with quality evaluation but, above a certain threshold of relevance, Unali summarizes the findings of a bad study just like it summarizes the findings of a good study.
Similarly, when you are impressed by Unali results, it’s in large part because some researchers worked hard to do the actual research and present it.
Double-check Unali's work. Confirm that Unali’s summaries and extracted information are correct by clicking on the provided research link to review the abstract or full text of the paper. Search for both sides of your question to minimize confirmation bias.
The team building Unali
Unali is built by Second Anthem LTD, a team of dedicated data scientists and software engineers closely advised by health professionals, all of them distributed across Europe and the US. Our team brings experiences from research academia, tech, and the medical field.
Unali is self funded and does not have any ties with pharmaceutical companies nor the product manufacturers or therapy providers it recommends. Our team Is primarily motivated by bringing new ideas to the table to help people with chronic conditions or symptoms, making sure that artificial intelligence is used in a safe manner and productive manner through high quality, ethically sourced data.
How Unali works?
Unali is an early-stage product, with updates and improvements every week. As of May 2023, the therapy creation workflow is implemented as follows:
Unali retrieves relevant scientific literature by employing a comprehensive approach that analyzes various factors such as title, abstract and keywords.
-
In the first step we leverage GPT4 model Da-Vinci to generate keywords. Specifically, we input a condition name from our database, and require the model to output a list of 5 ingredients to facilitate efficient and targeted paper searches. GPT4 utilizes its knowledge to provide relevant ingredients. These ingredients are combined with the input condition to form pairs of keywords (e.g. anxiety + ashwagandha) which act as a guiding compass, allowing us to focus our search on specific areas of interest.
-
To conduct the paper search, we integrate the Semantic Scholar Graph API directly into the process. The API provides us with seamless access to an extensive database of academic papers and publications. By leveraging this API, we can bypass the need for manual searching as well as replace the fully automatic, GPT4-based step of the previous model, which is susceptible to unreliable results.
This step, moreover, replaces the need to cross-check GPT4 results against Google Scholar, as every paper returned is directly seeded from the Semantic Scholar database.
The integration of the semantic scholar API ensures that our search is efficient, comprehensive, and up-to-date, as it directly taps into the latest scholarly research available.
-
Once we have obtained a set of relevant papers, we employ GPT4 once again, this time to rank the papers based on their relevance to the initial keywords.This ranking process enables us to prioritize papers that are most likely to be closely aligned with the intended research or topic of interest so we can serve users with the most valuable and pertinent research material.
-
Unali goes beyond retrieval and ranking by providing tailored summaries of abstracts related to the query, simplifying complex or lengthy content and assisting users in determining the papers' relevance.
Ultimately Unali's approach combines AI-driven analysis, validation through reputable sources, and query-specific summarization to ensure that users receive the most relevant scientific literature in a concise and understandable format. By streamlining the search and evaluation process, Unali enables users to efficiently access valuable research and make informed decisions based on the provided summaries.
Unali's limitations
To help you calibrate how much you can rely on Unali, we’ll share some of the limitations you should be aware of as you use Unali:
Limitations specific to Unali
Part of Unali’s pipeline uses large language models, which have only been around since 2019. While already useful, these early stage technologies are far from “Artificial general intelligence that takes away all of our jobs.”
For example, the models aren’t explicitly trained to be faithful to a body of text by default. We’ve had to introduce verifiers and human feedback to make sure their summaries or extractions are actually what is said in the abstract, and not what the model thinks is likely to be the case in general (sometimes called "hallucination"). While we’ve made a lot of progress and try hard to err on the side of Unali saying nothing rather than saying something wrong, in some cases Unali can miss the nuance of a paper or misunderstand what a number refers to.
Limitations that apply to research or search tools in general
Unali is only as good as the papers underlying it. While we think researchers are a very careful and rigorous group on average, there is research with questionable methodology and even fraud.
Unali does not yet know how to evaluate whether one paper is more trustworthy than another, except by giving you an adjusted Jadad Score. It includes considerations like citation count, journal, critiques from other researchers who cited the paper, and certain methodological details (sample size, study type, etc.).
In the same way that good research involves looking for evidence for and against various arguments, we recommend searching for papers presenting multiple sides of a position to avoid confirmation bias.
Unali works better for some symptoms and conditions than others.
Suggestions for how to approach Unali
Given these limitations, here are some ways to relate to Unali that can be useful without leading to undue confidence in Unali’s abilities.
Find papers that you may not have found elsewhere
The same query may get you different results in different databases. This can be because different search tools have more or different papers or because they rank papers differently. Unali can supplement other search tools to help you discover different papers. The inverse is also true - search engines with different ranking algorithms may return different papers at the top even if they were to have the exact same data as us.
Figure out where to drill in
Unali is a starting point. Some recommended papers might seem relevant, but you may not know whether digging into them would involve getting stuck at a local optimum. Overall, there are way more papers than any of us could ever read in an ideal world. Unali can help with the prioritization decision by showing information about papers and letting you sort or filter by that information.
