OpenAI just released in face of fierce competition in the market.

When generating model responses, you can extend model capabilities using built-in tools. These tools help models access additional context and information from the web or your files. The example below uses the web search tool to use the latest information from the web to generate a model response.

import OpenAI from "openai";
const client = new OpenAI();

const completion = await client.chat.completions.create({
    model: "gpt-4o-search-preview",
    web_search_options: {},
    messages: [{
        "role": "user",
        "content": "What was a positive news story from today?"
    }],
});

console.log(completion.choices[0].message.content);

Note location of user can be entered to optimize the web search and note The gpt-4o-search-preview and gpt-4o-mini-search-preview models used in Chat Completions only support a subset of API parameters.

import OpenAI from "openai";
const client = new OpenAI();

const completion = await client.chat.completions.create({
    model: "gpt-4o-search-preview",
    web_search_options: {
        user_location: {
            type: "approximate",
            approximate: {
                country: "GB",
                city: "London",
                region: "London",
            },
        },
    },
    messages: [{
        "role": "user",
        "content": "What are the best restaurants around Granary Square?",
    }],
});
console.log(completion.choices[0].message.content);

Recall function calling is kept as before:

from openai import OpenAI

client = OpenAI()

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current temperature for a given location.",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "City and country e.g. Bogotá, Colombia"
                }
            },
            "required": [
                "location"
            ],
            "additionalProperties": False
        },
        "strict": True
    }
}]

completion = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is the weather like in Paris today?"}],
    tools=tools
)

print(completion.choices[0].message.tool_calls)

About file search, note Projects are limited to a total size of 100GB for all Files; Vector stores are limited to a total of 10k files; Individual files can be a max of 512MB (roughly 5M tokens per file)

Computer use is mostly interesting, here is snippet from openAI’s own demo

but in their official doc, there are quite a lot of limitations: “We recommend using the computer-use-preview model for browser-based tasks. The model may be susceptible to inadvertent model mistakes, especially in non-browser environments that it is less used to.

For example, computer-use-preview‘s performance on OSWorld is currently 38.1%, indicating that the model is not yet highly reliable for automating tasks on an OS. More details about the model and related safety work can be found in our updated system card.

Some other behavior limitations to be aware of:

This tool does not support zero data retention (data retention policies).

The computer-use-preview model has constrained rate limits and feature support, described on its model detail page.'”

At the end of their demo, a swarm of agents were used and demoed how it accomplish roles as refund, support etc. service and notion of handoffs.

Even in early stage, it painted the future application of AI to build powerful agents replacing human workers.

My plan is 1. web search function for information retrieval; 2. computer use for streamline/automate the maintenance workflow; 3. computer use for marketing and promoting, maybe the last item be set as priority!