List evaluators

cURL

curl --request GET \
  --url https://api.fireworks.ai/v1/accounts/{account_id}/evaluators \
  --header 'Authorization: Bearer <token>'

{
  "evaluators": [
    {
      "name": "<string>",
      "displayName": "<string>",
      "description": "<string>",
      "createTime": "2023-11-07T05:31:56Z",
      "createdBy": "<string>",
      "updateTime": "2023-11-07T05:31:56Z",
      "state": "STATE_UNSPECIFIED",
      "multiMetrics": true,
      "criteria": [
        {
          "type": "TYPE_UNSPECIFIED",
          "name": "<string>",
          "description": "<string>",
          "codeSnippets": {
            "language": "<string>",
            "fileContents": {},
            "entryFile": "<string>",
            "entryFunc": "<string>"
          }
        }
      ],
      "requirements": "<string>",
      "rollupSettings": {
        "criteriaWeights": {},
        "pythonCode": "<string>",
        "successThreshold": 123,
        "skipRollup": true
      },
      "entryPoint": "<string>",
      "status": {
        "code": "OK",
        "message": "<string>"
      },
      "commitHash": "<string>",
      "source": {
        "type": "TYPE_UNSPECIFIED",
        "githubRepositoryName": "<string>"
      },
      "defaultDataset": "<string>"
    }
  ],
  "nextPageToken": "<string>",
  "totalSize": 123
}

GET

accounts

{account_id}

evaluators

cURL

curl --request GET \
  --url https://api.fireworks.ai/v1/accounts/{account_id}/evaluators \
  --header 'Authorization: Bearer <token>'

{
  "evaluators": [
    {
      "name": "<string>",
      "displayName": "<string>",
      "description": "<string>",
      "createTime": "2023-11-07T05:31:56Z",
      "createdBy": "<string>",
      "updateTime": "2023-11-07T05:31:56Z",
      "state": "STATE_UNSPECIFIED",
      "multiMetrics": true,
      "criteria": [
        {
          "type": "TYPE_UNSPECIFIED",
          "name": "<string>",
          "description": "<string>",
          "codeSnippets": {
            "language": "<string>",
            "fileContents": {},
            "entryFile": "<string>",
            "entryFunc": "<string>"
          }
        }
      ],
      "requirements": "<string>",
      "rollupSettings": {
        "criteriaWeights": {},
        "pythonCode": "<string>",
        "successThreshold": 123,
        "skipRollup": true
      },
      "entryPoint": "<string>",
      "status": {
        "code": "OK",
        "message": "<string>"
      },
      "commitHash": "<string>",
      "source": {
        "type": "TYPE_UNSPECIFIED",
        "githubRepositoryName": "<string>"
      },
      "defaultDataset": "<string>"
    }
  ],
  "nextPageToken": "<string>",
  "totalSize": 123
}

Authorizations

Authorization

string

header

required

Bearer authentication using your Fireworks API key. Format: Bearer <API_KEY>

Path Parameters

account_id

string

required

The Account Id

Query Parameters

pageSize

integer<int32>

pageToken

string

filter

string

orderBy

string

readMask

string

The fields to be returned in the response. If empty or "*", all fields will be returned.

Response

200 - application/json

A successful response.

evaluators

Next ID: 17 · object[]

Show child attributes

evaluators.name

string

evaluators.displayName

string

evaluators.description

string

evaluators.createTime

string<date-time>

evaluators.createdBy

string

evaluators.updateTime

string<date-time>

evaluators.state

enum<string>

default:STATE_UNSPECIFIED

Available options:

STATE_UNSPECIFIED,

ACTIVE,

BUILDING,

BUILD_FAILED

evaluators.multiMetrics

boolean

If true, the criteria will report multiple metric-score pairs Otherwise, each criteria will report the score assigned to the criteria name as metric.

evaluators.criteria

Criteria for the evaluator, it should produce a score for the metric (name of criteria) · object[]

Show child attributes

evaluators.criteria.type

enum<string>

default:TYPE_UNSPECIFIED

Available options:

TYPE_UNSPECIFIED,

CODE_SNIPPETS

evaluators.criteria.name

string

evaluators.criteria.description

string

evaluators.criteria.codeSnippets

Criteria for code snippet · object

Show child attributes

evaluators.criteria.codeSnippets.language

string

evaluators.criteria.codeSnippets.fileContents

File name to code snippet, default is main.py · object

Show child attributes

evaluators.criteria.codeSnippets.fileContents.{key}

string

evaluators.criteria.codeSnippets.entryFile

string

evaluators.criteria.codeSnippets.entryFunc

string

evaluators.requirements

string

evaluators.rollupSettings

object

Strategy for metrics reports summary/rollup. e.g. {metric1: 1, metric2: 0.3}, rollup_settings could be criteria_weights: {metric1: 0.5, metric2: 0.5}, then final score will be 0.5 * 1 + 0.5 * 0.3 = 0.65 If skip_rollup is true, the rollup step will be skipped since the criteria will also report the rollup score and metrics altogether.

Show child attributes

evaluators.rollupSettings.criteriaWeights

from criteria name to weights in rollup score · object

Show child attributes

evaluators.rollupSettings.criteriaWeights.{key}

number<float>

evaluators.rollupSettings.pythonCode

string

evaluators.rollupSettings.successThreshold

number<float>

evaluators.rollupSettings.skipRollup

boolean

evaluators.entryPoint

string

evaluators.status

Status of the evaluator, used to expose build status to the user · object

Show child attributes

evaluators.status.code

enum<string>

default:OK

The status code.

Available options:

OK,

CANCELLED,

UNKNOWN,

INVALID_ARGUMENT,

DEADLINE_EXCEEDED,

NOT_FOUND,

ALREADY_EXISTS,

PERMISSION_DENIED,

UNAUTHENTICATED,

RESOURCE_EXHAUSTED,

FAILED_PRECONDITION,

ABORTED,

OUT_OF_RANGE,

UNIMPLEMENTED,

INTERNAL,

UNAVAILABLE,

DATA_LOSS

evaluators.status.message

string

A developer-facing error message in English.

evaluators.commitHash

string

evaluators.source

object

Source information for the evaluator codebase.

Show child attributes

evaluators.source.type

enum<string>

default:TYPE_UNSPECIFIED

Identifies how the evaluator source code is provided.

Available options:

TYPE_UNSPECIFIED,

TYPE_UPLOAD,

TYPE_GITHUB,

TYPE_TEMPORARY

evaluators.source.githubRepositoryName

string

Normalized GitHub repository name (e.g. owner/repository) when the source is GitHub.

evaluators.defaultDataset

string

nextPageToken

string

totalSize

integer<int32>

Delete Evaluation Job

⌘I

API Reference

Inference

Deployments

Fine-tuning

Evals

Multimedia

Admin

Build SDK (Deprecated)

Authorizations

Path Parameters

Query Parameters

Response