Ranking Anything with GPT4

I’ve been fascinated by this paper: Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent and have been trying out the ideas with success in a few personal projects.

They essentially found that GPT4 is excellent at ranking things. Given a set of items (candidates) and some query it can rank these items very well (as well as or better than the current SOTA models for the benchmarks they evaluated).

They focused on benchmarks that include text only data (i.e. passages, news articles, etc), but I’ve found that the same ranking prompts work well for pretty much everything, and more so the ranking can take other attributes into account, things like tags, genres, user ratings, languages, and so on.

Why is this useful?

The prevailing workflow right now for including external data for LLM inference is to get it from some external data store (vector store, database, website HTML, etc) and stuff it into the prompt as context without any additional evaluation, ranking, or filtering.

So from call to call you could be getting great context, or low quality context, or some mixture of the two. This effects your final outcome since the model will rely heavily on whatever data your provided to do what you asked.

Here’s a quick example of adding in a ranker to the above workflow; if you want to just see output jump to the last section.

1. Grab some interesting data.
2. Generate a set of candidate games for a given query.
3. Rank these games via the methods proposed in that paper.

Dataset

For demo data, I’m using this Steam Games Dataset, an example of it below.

{'name': 'Royal Battleships',
 'release_date': 'Apr 6, 2018',
 'required_age': 0,
 'price': 2.99,
 'dlc_count': 0,
 'detailed_description': "Special Offer About the Game Sometimes there is nothing better than a tourney in a classic battleships game. You can see a beautifully designed sea battle, familiar to those from your school times. You have to play against the computer and you need to destroy all enemy ships before yours are destroyed. Features: Very beautiful graphics Classic game You can compete with your friends Supporting all platforms (Windows, Mac, Linux, SteamOS) A lot of Steam achievements The gameplay is very simple. When you are preparing for the match, there are some ships of different sizes - from simple battle boat to a huge battleship. You put them on the battlefield and nobody else than you knows their location. Of course, don't know the enemy's positions neither. Thus the battle is absolutely fair and you can only guess where to shoot. Of course, if you hit a battleship for the first time, it will be easy to find the other tiles. However, finding all enemy boats can be a real challenge. Do not forget that this game is not about speed, but about variety, so that the battle is always different. The game perfectly develops strategic thinking and will be useful to both adults and children. In addition, it will help you to plunge into nostalgia for simple classic games. ",
 'short_description': 'Royal Battleships is a classic sea battle game, which includes battles against the AI and battles for points. Destroy the enemy fleet and try to make yours survive.',
 ...(truncated)...
}

Candidate Generation

For this example, I embedded each game’s detailed_description via OpenAI’s text-embedding-ada-002 model.

Then for a given query I embed it as well, and find the nearest embeddings to it via cosine similarity.

Ranking

For each candidate game I take a subset of the fields, add an index field that just matches the index of where it’s located in the list, and dump the list of dicts as a json string for the context_string below.

Here’s the ranker function, the prompts below are modified versions of prompts from the paper I referenced above.

def ranker(games, user_query):

    system_message = {"role": "system", "content": "You are RankGPT, an intelligent assistant that can rank games based on how helpful they would be to the user."}
    context_string = format_games(games)

    prompt = f"""
    I will provide you with {len(games)} games, each indicated by number `index` field.
    Rank the games based on their relevance to query: {user_query}.

    {context_string}

    Search Query: {user_query}.
    
    Rank the {len(games)} games above based on their relevance to the search query.
    Use all the information available to you to rank the games.
    The games should be listed in descending order using identifiers. 
    The most relevant games should be listed first. If multiple games are equally relevant, use the number of positive votes as a tie-breaker.
    The output format should be a list of identifiers, separated by commas and enclosed in square brackets.
    Only output the list of identifiers, do not output any other text.
    """

    modified_user_message = {'role': 'user', 'content': prompt}

    response = openai.ChatCompletion.create(
        model="gpt-4",
        messages=[system_message, modified_user_message],
        stream = False,
        temperature=0.7,
    )

    ranked_order = json.loads(response.choices[0].message.content)

    return ranked_order

Let’s try it out. I’ll use the same query for both candidate generation and and ranking to make things simple, but they could be different (and should probably be different in real usage).

Here’s a few example queries with tables of the ranked games, along with the original rank (so what index the game was in the candidate set prior to ranking).

I’m looking for games with lots of intrigue and lore. I want to be able to explore a world and find out more about it

I’m looking for satirical JRPG games

I’m looking for soothing, mediative games

I’m looking for a game like Zelda, but with more action and less puzzles

I’m looking for games with lots of intrigue and lore. I want to be able to explore a world and find out more about it.

query = '''I'm looking for games with lots of intrigue and lore. I want to be able to explore a world and find out more about it.'''
games = get_games(query, 10)
ranked_order = ranker(games, query)

I’m looking for satirical JRPG games

query = '''I'm looking for satirical JRPG games'''
games = get_games(query, 10)
ranked_order = ranker(games, query)

I’m looking for soothing, mediative games

query = '''I'm looking for soothing, mediative games'''
games = get_games(query, 10)
ranked_order = ranker(games, query)

I’m looking for a game like Zelda, but with more action and less puzzles.

query = '''I'm looking for a game like Zelda, but with more action and less puzzles.'''
games = get_games(query, 10)
ranked_order = ranker(games, query)