Create an evaluation rule using stable evaluator identifiers on Langfuse

Langfuse: Create an evaluation rule using stable evaluator identifiers. It runs on your own Langfuse account: connect it once in the Relvo app and Relvo adds its credentials server-side. Relvo charges no usage fee; Langfuse may bill that account under its own plan.

Connect LangfuseAll Langfuse toolsAPI docs

Try Create an evaluation rule using stable evaluator identifiers now

Compose the request, copy the prompt, and run it in your agent. It runs on your connected Langfuse account.

Prompt composerMarkdown
Langfuselangfuse__evaluation_rules_create
Open Relvo
Try asking

More Langfuse tools

The rest of the Langfuse shelf. Every endpoint runs on the same Langfuse connection.

124 tools
Get all annotation queueslangfuse__annotation_queues_list_queuesLangfuseyour accountCreate an annotation queuelangfuse__annotation_queues_create_queueLangfuseyour accountGet an annotation queue by IDlangfuse__annotation_queues_get_queueLangfuseyour accountGet items for a specific annotation queuelangfuse__annotation_queues_list_queue_itemsLangfuseyour accountAdd an item to an annotation queuelangfuse__annotation_queues_create_queue_itemLangfuseyour accountGet a specific item from an annotation queuelangfuse__annotation_queues_get_queue_itemLangfuseyour accountUpdate an annotation queue itemlangfuse__annotation_queues_update_queue_itemLangfuseyour accountRemove an item from an annotation queuelangfuse__annotation_queues_delete_queue_itemLangfuseyour accountCreate an assignment for a user to an annotation queuelangfuse__annotation_queues_create_queue_assignmentLangfuseyour accountDelete an assignment for a user to an annotation queuelangfuse__annotation_queues_delete_queue_assignmentLangfuseyour accountGet all blob storage integrations for the organization (requires organization-scoped API key)langfuse__blob_storage_integrations_get_blob_storage_inte_b49accLangfuseyour accountCreate or update a blob storage integration for a specific project (requires organization-scoped API key)langfuse__blob_storage_integrations_upsert_blob_storage_i_2bd6d2Langfuseyour accountGet the sync status of a blob storage integration by integration ID (requires organization-scoped API key)langfuse__blob_storage_integrations_get_blob_storage_inte_f96137Langfuseyour accountDelete a blob storage integration by ID (requires organization-scoped API key)langfuse__blob_storage_integrations_delete_blob_storage_i_deee21Langfuseyour accountCreate a commentlangfuse__comments_createLangfuseyour accountGet all commentslangfuse__comments_getLangfuseyour accountGet a comment by idlangfuse__comments_get_by_idLangfuseyour accountCreate a dataset itemlangfuse__dataset_items_createLangfuseyour accountGet dataset itemslangfuse__dataset_items_listLangfuseyour accountGet a dataset itemlangfuse__dataset_items_getLangfuseyour accountDelete a dataset item and all its run itemslangfuse__dataset_items_deleteLangfuseyour accountGet all datasetslangfuse__datasets_listLangfuseyour accountCreate a datasetlangfuse__datasets_createLangfuseyour accountGet a datasetlangfuse__datasets_getLangfuseyour accountCreate an evaluation rule using stable evaluator identifierslangfuse__evaluation_rules_createYou are hereyour accountList evaluation rules in newest-first creation orderlangfuse__evaluation_rules_listLangfuseyour accountGet one evaluation rule, including a legacy trace or dataset rule, by its stable identifierlangfuse__evaluation_rules_getLangfuseyour accountUpdate an evaluation rule by its stable identifierlangfuse__evaluation_rules_updateLangfuseyour accountDelete an evaluation rulelangfuse__evaluation_rules_deleteLangfuseyour accountCreate an evaluator in the authenticated projectlangfuse__evaluators_createLangfuseyour account

Set up Create an evaluation rule using stable evaluator identifiers for your agent

Pick the client you use, then pick a route: Automatic is that client's own link, command or prompt, and Manual is the same install done by hand, or the tool opened in the Relvo app. Both end at the same OAuth sign-in, with nothing installed per tool.

Automatic way

Open the plugin page with the details ready.

ChatGPT takes custom MCP servers as plugins. The button opens the right page; copy the name and URL into the form and authorise once.

Open ChatGPT plugins

  1. Add a custom MCP server

    On the plugins page, click + and choose Add custom MCP server.

  2. Fill in the connection

    NameRelvo
    Connectionhttps://app.relvo.si/mcp

  3. Authorise and create

    Pick OAuth, tick the confirmation and click Create as a plugin. Approve Relvo in the browser window ChatGPT opens.

  4. Ask for Create an evaluation rule using stable evaluator identifiers

    PROMPT
    Using Relvo, call `langfuse__evaluation_rules_create` on my Langfuse connection to create an evaluation rule using stable evaluator identifiers. Return Langfuse's response as it comes back.

    The client inspects this operation and runs it on your connected account.

Custom MCP servers are set up in ChatGPT on the web, not in the mobile app. Connect once and every chat can call Relvo.

Manual way

Set it up from ChatGPT's settings.

The same connector, reached through the settings menu instead of the plugin page.

  1. Open Apps & Connectors

    In ChatGPT, open Settings → Apps & Connectors and create a new custom connector.

  2. Name it Relvo

    NameRelvo

  3. Point it at Relvo

    Connectionhttps://app.relvo.si/mcp

  4. Authenticate

    Choose OAuth, continue, and approve Relvo in the browser. ChatGPT saves the connector to your account.

  5. Ask for Create an evaluation rule using stable evaluator identifiers

    PROMPT
    Using Relvo, call `langfuse__evaluation_rules_create` on my Langfuse connection to create an evaluation rule using stable evaluator identifiers. Return Langfuse's response as it comes back.

    The agent checks this operation, runs it on your connection, then relays the provider's response verbatim.

  6. Open langfuse__evaluation_rules_create in the app

    URL
    https://app.relvo.si/catalog/langfuse__evaluation_rules_create

    Connect Langfuse under Connections in the Relvo app, then inspect this operation's input schema before running it.

This operation runs on your own Langfuse account. Relvo charges no usage fee; Langfuse may bill that account on its own plan.

Automatic way

One link fills in the dialog.

On a desktop browser this opens Claude's Add custom connector form with Relvo's name and MCP URL already in place. Click Add and sign in — there is nothing to type.

Add the Relvo connector

  1. Open the dialog

    Click the button above in the browser where you are signed in to Claude. Customize → Connectors opens with the custom-connector form ready.

  2. Add and authorise

    Click Add, then choose Sign in now under Authentication. Relvo signs you in over OAuth, so no token or API key is pasted into Claude.

  3. Ask for Create an evaluation rule using stable evaluator identifiers

    PROMPT
    Using Relvo, call `langfuse__evaluation_rules_create` on my Langfuse connection to create an evaluation rule using stable evaluator identifiers. Return Langfuse's response as it comes back.

    The client inspects this operation and runs it on your connected account.

Manual way

The same connector, typed by hand.

Use this on a phone, in a locked-down workspace, or whenever you would rather add the connector through Claude's own menu.

  1. Open Connectors

    In Claude, go to Customize → Connectors and choose + Add → Add custom connector.

  2. Name the connector

    NameRelvo

  3. Paste the MCP URL

    Server URLhttps://app.relvo.si/mcp

  4. Sign in

    Click Add, choose Sign in now under Authentication, and approve Relvo in the OAuth window. The Relvo tools appear under the connector.

  5. Ask for Create an evaluation rule using stable evaluator identifiers

    PROMPT
    Using Relvo, call `langfuse__evaluation_rules_create` on my Langfuse connection to create an evaluation rule using stable evaluator identifiers. Return Langfuse's response as it comes back.

    The agent checks this operation, runs it on your connection, then relays the provider's response verbatim.

  6. Open langfuse__evaluation_rules_create in the app

    URL
    https://app.relvo.si/catalog/langfuse__evaluation_rules_create

    Connect Langfuse under Connections in the Relvo app, then inspect this operation's input schema before running it.

This operation runs on your own Langfuse account. Relvo charges no usage fee; Langfuse may bill that account on its own plan.

Automatic way

One command, then one sign-in.

The CLI registers Relvo with HTTP transport and writes it into your Claude Code config; authentication happens in the browser.

  1. Run the command

    BASH
    claude mcp add --transport http relvo https://app.relvo.si/mcp

  2. Authenticate once

    Open a session and run /mcp. Choose Relvo → Authenticate; Claude Code opens Relvo's OAuth page and stores the token for the project.

  3. Check the list

    BASH
    claude mcp list

    Relvo shows as connected, with every catalog tool behind it.

  4. Ask for Create an evaluation rule using stable evaluator identifiers

    PROMPT
    Using Relvo, call `langfuse__evaluation_rules_create` on my Langfuse connection to create an evaluation rule using stable evaluator identifiers. Return Langfuse's response as it comes back.

    The client inspects this operation and runs it on your connected account.

The command uses local scope by default. Add --scope user to make Relvo available in every project, or --scope project to share it through the repository.

Manual way

Write it into .mcp.json.

Project scope lives in a file, so the whole team gets Relvo without running a command.

  1. Add the server

    JSON · .mcp.json
    {
  "mcpServers": {
    "relvo": {
      "type": "http",
      "url": "https://app.relvo.si/mcp"
    }
  }
}

    Create .mcp.json in the project root and add relvo as an HTTP server.

  2. Reload and sign in

    Start a Claude Code session, run /mcp, choose Relvo → Authenticate, then reload. The tools are available in that project.

  3. Ask for Create an evaluation rule using stable evaluator identifiers

    PROMPT
    Using Relvo, call `langfuse__evaluation_rules_create` on my Langfuse connection to create an evaluation rule using stable evaluator identifiers. Return Langfuse's response as it comes back.

    The agent checks this operation, runs it on your connection, then relays the provider's response verbatim.

  4. Open langfuse__evaluation_rules_create in the app

    URL
    https://app.relvo.si/catalog/langfuse__evaluation_rules_create

    Connect Langfuse under Connections in the Relvo app, then inspect this operation's input schema before running it.

The same entry works for every project when it is added at user scope instead of project scope. This operation runs on your own Langfuse account. Relvo charges no usage fee; Langfuse may bill that account on its own plan.

Automatic way

Hand the setup to Hermes.

Hermes can edit its own config, so one instruction is enough: it writes Relvo into mcp_servers, reloads, and reports the tools it can now call.

  1. Copy the setup prompt

    PROMPT
    Add Relvo as a remote MCP server named "relvo": url https://app.relvo.si/mcp, auth oauth. Follow https://app.relvo.si/SKILL.md, reload MCP, then list the Relvo tools.

  2. Paste it into a Hermes session

    Hermes adds the remote server to ~/.hermes/config.yaml and reloads its MCP connections.

  3. Approve the sign-in

    Open the OAuth link Hermes prints and approve Relvo once. The token is cached, so later sessions do not ask again.

  4. Ask for Create an evaluation rule using stable evaluator identifiers

    PROMPT
    Using Relvo, call `langfuse__evaluation_rules_create` on my Langfuse connection to create an evaluation rule using stable evaluator identifiers. Return Langfuse's response as it comes back.

    The client inspects this operation and runs it on your connected account.

Manual way

Add it to ~/.hermes/config.yaml.

Two settings and one reload — the same entry Hermes would write itself.

  1. Open the config

    Edit ~/.hermes/config.yaml, the file Hermes reads at start-up.

  2. Add the server

    YAML
    mcp_servers:
  Relvo:
    url: https://app.relvo.si/mcp
    auth: oauth

  3. Reload and sign in

    In a session, run /reload-mcp, then approve the OAuth prompt Hermes shows. Tokens are cached under ~/.hermes/mcp-tokens/.

  4. Ask for Create an evaluation rule using stable evaluator identifiers

    PROMPT
    Using Relvo, call `langfuse__evaluation_rules_create` on my Langfuse connection to create an evaluation rule using stable evaluator identifiers. Return Langfuse's response as it comes back.

    The agent checks this operation, runs it on your connection, then relays the provider's response verbatim.

  5. Open langfuse__evaluation_rules_create in the app

    URL
    https://app.relvo.si/catalog/langfuse__evaluation_rules_create

    Connect Langfuse under Connections in the Relvo app, then inspect this operation's input schema before running it.

This operation runs on your own Langfuse account. Relvo charges no usage fee; Langfuse may bill that account on its own plan.

Automatic way

Hand the install to Cursor.

The button opens Cursor's MCP install dialog with the server name and URL already attached.

Add Relvo to Cursor

  1. Open the install prompt

    Click the button on the machine where Cursor is installed. Cursor receives a server named relvo pointing at https://app.relvo.si/mcp.

  2. Install and authenticate

    Confirm the install, then click Authenticate. Cursor completes Relvo's OAuth in the browser.

  3. Ask for Create an evaluation rule using stable evaluator identifiers

    PROMPT
    Using Relvo, call `langfuse__evaluation_rules_create` on my Langfuse connection to create an evaluation rule using stable evaluator identifiers. Return Langfuse's response as it comes back.

    The client inspects this operation and runs it on your connected account.

The install link opens Cursor directly, so it does nothing in a browser that has never installed the editor.

Manual way

Edit ~/.cursor/mcp.json.

Use the global or the project file when you want the entry under your own control.

  1. Open the MCP config

    Edit ~/.cursor/mcp.json for every project, or .cursor/mcp.json inside the repository for one.

  2. Add Relvo

    JSON
    {
  "mcpServers": {
    "relvo": {
      "url": "https://app.relvo.si/mcp"
    }
  }
}

  3. Reload Cursor

    Save the file and reload the window. Relvo appears in the MCP panel, and Authenticate finishes the sign-in.

  4. Ask for Create an evaluation rule using stable evaluator identifiers

    PROMPT
    Using Relvo, call `langfuse__evaluation_rules_create` on my Langfuse connection to create an evaluation rule using stable evaluator identifiers. Return Langfuse's response as it comes back.

    The agent checks this operation, runs it on your connection, then relays the provider's response verbatim.

  5. Open langfuse__evaluation_rules_create in the app

    URL
    https://app.relvo.si/catalog/langfuse__evaluation_rules_create

    Connect Langfuse under Connections in the Relvo app, then inspect this operation's input schema before running it.

This operation runs on your own Langfuse account. Relvo charges no usage fee; Langfuse may bill that account on its own plan.

Automatic way

Ask the Bot to wire it up.

Grok Bot configures connections from the conversation, so the quickest route is a single instruction with Relvo's address.

  1. Copy the setup prompt

    PROMPT
    Add a custom MCP server called relvo at https://app.relvo.si/mcp with OAuth sign-in, then list the Relvo tools.

  2. Paste it into a Bot chat

    The Bot repeats the server name and URL and asks you to confirm. Check them and select Add it.

  3. Authorise

    Select Authorize on the card the Bot sends and approve Relvo in the browser once. The Bot can then call the catalog from any chat.

  4. Ask for Create an evaluation rule using stable evaluator identifiers

    PROMPT
    Using Relvo, call `langfuse__evaluation_rules_create` on my Langfuse connection to create an evaluation rule using stable evaluator identifiers. Return Langfuse's response as it comes back.

    The client inspects this operation and runs it on your connected account.

Manual way

Name the server yourself in the chat.

Grok Bot's settings have no field for a server URL; custom MCP servers are added from the conversation.

  1. Describe the server

    NameRelvo
    Server URLhttps://app.relvo.si/mcp

    Ask the Bot to add a custom server with this name and URL. Say custom server, or it may offer a catalog plugin instead.

  2. Confirm

    The Bot repeats the name and URL. Check that the URL is https://app.relvo.si/mcp, then select Add it.

  3. Authorise

    Select Authorize on the card the Bot sends, sign in and allow access. The Relvo tools appear once you are signed in.

  4. Ask for Create an evaluation rule using stable evaluator identifiers

    PROMPT
    Using Relvo, call `langfuse__evaluation_rules_create` on my Langfuse connection to create an evaluation rule using stable evaluator identifiers. Return Langfuse's response as it comes back.

    The agent checks this operation, runs it on your connection, then relays the provider's response verbatim.

  5. Open langfuse__evaluation_rules_create in the app

    URL
    https://app.relvo.si/catalog/langfuse__evaluation_rules_create

    Connect Langfuse under Connections in the Relvo app, then inspect this operation's input schema before running it.

This operation runs on your own Langfuse account. Relvo charges no usage fee; Langfuse may bill that account on its own plan.

Use Langfuse with any agent

Pick the client you already use. Every one reaches the same catalog through one Relvo key.

All agents
ChatGPTUse Langfuse with ChatGPTClaudeUse Langfuse with ClaudeClaude CodeUse Langfuse with Claude CodeHermes AgentUse Langfuse with Hermes AgentCursorUse Langfuse with CursorGrok BotUse Langfuse with Grok BotCodexUse Langfuse with CodexOpenClawUse Langfuse with OpenClawAny MCP ClientUse Langfuse with Any MCP Client

Explore other platforms

Other shelves in the same catalog. One key reaches all of them.

Browse all platforms
Google SearchBrowse Google Search in the Relvo app. Inspect each tool and compare providers before running a call.11 endpointsXBrowse X in the Relvo app. Inspect each tool and compare providers before running a call.332 endpointsInstagramBrowse Instagram in the Relvo app. Inspect each tool and compare providers before running a call.208 endpointsTikTokBrowse TikTok in the Relvo app. Inspect each tool and compare providers before running a call.231 endpoints

Frequently asked questions

Short answers about the key, the price and the call itself.

Give your agent Langfuse and 147,361 tools

One token reaches your Langfuse account and every catalog tool. Start free.

Start freeBrowse the catalog