U ܟj%d@sddlmZddlmZmZddlmZddlZddlm Z ddl m Z m Z m Z ddlmZmZmZmZmZdd lmZmZdd lmZd d lmZmZmZmZmZmZdd l m!Z!m"Z"ddl#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7ddgZ8Gddde!Z9Gddde"Z:GdddZ;GdddZdS)#) annotations)IterableOptional)LiteralN)_legacy_response)eval_list_paramseval_create_paramseval_update_params) NOT_GIVENBodyQueryHeadersNotGiven)maybe_transformasync_maybe_transform)cached_property)Runs AsyncRunsRunsWithRawResponseAsyncRunsWithRawResponseRunsWithStreamingResponseAsyncRunsWithStreamingResponse)SyncAPIResourceAsyncAPIResource)to_streamed_response_wrapper"async_to_streamed_response_wrapper)SyncCursorPageAsyncCursorPage)AsyncPaginatormake_request_options)EvalListResponse)EvalCreateResponse)EvalDeleteResponse)EvalUpdateResponse)EvalRetrieveResponse)MetadataEvals AsyncEvalsc @seZdZeddddZeddddZeddd d Zeeed d d ed d dddddddddd ddZd d d eddddddddddZ eed d d ed dddddddd!d"d#d$Z eeeed d d ed%dd&d'd(ddddd)d* d+d,Z d d d eddddddd-dd.d/Z d S)0r(rreturncCs t|jSN)r_clientselfr0@/tmp/pip-unpacked-wheel-2qj3ofc_/openai/resources/evals/evals.pyruns)sz Evals.runsEvalsWithRawResponsecCst|Sa This property can be used as a prefix for any HTTP method call to return the raw response object instead of the parsed content. For more information, see https://www.github.com/openai/openai-python#accessing-raw-response-data-eg-headers )r3r.r0r0r1with_raw_response-szEvals.with_raw_responseEvalsWithStreamingResponsecCst|Sz An alternative to `.with_raw_response` that doesn't eagerly read the response body. For more information, see https://www.github.com/openai/openai-python#with_streaming_response )r6r.r0r0r1with_streaming_response7szEvals.with_streaming_responseNmetadatanameshare_with_openai extra_headers extra_query extra_bodytimeout#eval_create_params.DataSourceConfig-Iterable[eval_create_params.TestingCriterion]Optional[Metadata] | NotGivenstr | NotGivenbool | NotGivenHeaders | None Query | None Body | None'float | httpx.Timeout | None | NotGivenr# data_source_configtesting_criteriar:r;r<r=r>r?r@r+c Cs2|jdt|||||dtjt|||| dtdS) Create the structure of an evaluation that can be used to test a model's performance. An evaluation is a set of testing criteria and a datasource. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources. For more information, see the [Evals guide](https://platform.openai.com/docs/guides/evals). Args: data_source_config: The configuration for the data source used for the evaluation runs. testing_criteria: A list of graders for all eval runs in this group. metadata: Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters. name: The name of the evaluation. share_with_openai: Indicates whether the evaluation is shared with OpenAI. extra_headers: Send extra headers extra_query: Add additional query parameters to the request extra_body: Add additional JSON properties to the request timeout: Override the client-level default timeout for this request, in seconds /evalsrKrLr:r;r<r=r>r?r@bodyoptionscast_to)_postrr EvalCreateParamsr!r# r/rKrLr:r;r<r=r>r?r@r0r0r1create@s&. z Evals.createrPstrr&eval_idr=r>r?r@r+cCs4|std||jd|t||||dtdS)L Get an evaluation by ID. Args: extra_headers: Send extra headers extra_query: Add additional query parameters to the request extra_body: Add additional JSON properties to the request timeout: Override the client-level default timeout for this request, in seconds 6Expected a non-empty value for `eval_id` but received /evals/rPrSrT ValueError_getr!r&r/r[r=r>r?r@r0r0r1retrieveszEvals.retriever:r;r=r>r?r@r%r[r:r;r=r>r?r@r+c CsD|std||jd|t||dtjt||||dtdS) Update certain properties of an evaluation. Args: metadata: Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters. name: Rename the evaluation. extra_headers: Send extra headers extra_query: Add additional query parameters to the request extra_body: Add additional JSON properties to the request timeout: Override the client-level default timeout for this request, in seconds r]r^r:r;rPrQ)rarUrr EvalUpdateParamsr!r%r/r[r:r;r=r>r?r@r0r0r1updates$"z Evals.updateafterlimitorderorder_byr=r>r?r@int | NotGiven#Literal[('asc', 'desc')] | NotGiven0Literal[('created_at', 'updated_at')] | NotGivenz SyncCursorPage[EvalListResponse] rmrnrorpr=r>r?r@r+c Cs6|jdttt||||t||||dtjdtdSa List evaluations for a project. Args: after: Identifier for the last eval from the previous pagination request. limit: Number of evals to retrieve. order: Sort order for evals by timestamp. Use `asc` for ascending order or `desc` for descending order. order_by: Evals can be ordered by creation time or last updated time. Use `created_at` for creation time or `updated_at` for last updated time. extra_headers: Send extra headers extra_query: Add additional query parameters to the request extra_body: Add additional JSON properties to the request timeout: Override the client-level default timeout for this request, in seconds rN)rmrnrorp)r=r>r?r@query)pagerSmodel) _get_api_listrr"r!rrEvalListParams r/rmrnrorpr=r>r?r@r0r0r1lists&$z Evals.listr$cCs4|std||jd|t||||dtdS)I Delete an evaluation. Args: extra_headers: Send extra headers extra_query: Add additional query parameters to the request extra_body: Add additional JSON properties to the request timeout: Override the client-level default timeout for this request, in seconds r]r^rPr_ra_deleter!r$rcr0r0r1delete sz Evals.delete __name__ __module__ __qualname__rr2r5r8r rXrdrkr|rr0r0r0r1r((sP  $F% 6"?c @seZdZeddddZeddddZeddd d Zeeed d d ed d dddddddddd ddZd d d eddddddddddZ eed d d ed dddddddd!d"d#d$Z eeeed d d ed%dd&d'd(ddddd)d* d+d,Z d d d eddddddd-dd.d/Z d S)0r)rr*cCs t|jSr,)rr-r.r0r0r1r20szAsyncEvals.runsAsyncEvalsWithRawResponsecCst|Sr4)rr.r0r0r1r54szAsyncEvals.with_raw_responseAsyncEvalsWithStreamingResponsecCst|Sr7)rr.r0r0r1r8>sz"AsyncEvals.with_streaming_responseNr9rArBrCrDrErFrGrHrIr#rJc s>|jdt|||||dtjIdHt|||| dtdIdHS)rMrNrONrPrQ)rUrr rVr!r#rWr0r0r1rXGs&. zAsyncEvals.createrPrYr&rZcs:|std||jd|t||||dtdIdHS)r\r]r^rPr_Nr`rcr0r0r1rdszAsyncEvals.retrieverer%rfc sP|std||jd|t||dtjIdHt||||dtdIdHS)rgr]r^rhNrPrQ)rarUrr rir!r%rjr0r0r1rks$"zAsyncEvals.updaterlrqrrrszCAsyncPaginator[EvalListResponse, AsyncCursorPage[EvalListResponse]]rtc Cs6|jdttt||||t||||dtjdtdSru)ryrr"r!rrrzr{r0r0r1r|s&$zAsyncEvals.listr$cs:|std||jd|t||||dtdIdHS)r}r]r^rPr_Nr~rcr0r0r1rszAsyncEvals.deleterr0r0r0r1r)/sP  $F% 6"?c@s.eZdZdddddZedddd Zd S) r3r(Noneevalsr+cCsP||_t|j|_t|j|_t|j|_t|j|_t|j|_dSr,)_evalsrZto_raw_response_wrapperrXrdrkr|rr/rr0r0r1__init__7s zEvalsWithRawResponse.__init__rr*cCs t|jjSr,)rrr2r.r0r0r1r2JszEvalsWithRawResponse.runsNrrrrrr2r0r0r0r1r36sr3c@s.eZdZdddddZedddd Zd S) rr)rrcCsP||_t|j|_t|j|_t|j|_t|j|_t|j|_dSr,)rrZasync_to_raw_response_wrapperrXrdrkr|rrr0r0r1rPs z"AsyncEvalsWithRawResponse.__init__rr*cCs t|jjSr,)rrr2r.r0r0r1r2cszAsyncEvalsWithRawResponse.runsNrr0r0r0r1rOsrc@s.eZdZdddddZedddd Zd S) r6r(rrcCsF||_t|j|_t|j|_t|j|_t|j|_t|j|_dSr,)rrrXrdrkr|rrr0r0r1ris z#EvalsWithStreamingResponse.__init__rr*cCs t|jjSr,)rrr2r.r0r0r1r2|szEvalsWithStreamingResponse.runsNrr0r0r0r1r6hsr6c@s.eZdZdddddZedddd Zd S) rr)rrcCsF||_t|j|_t|j|_t|j|_t|j|_t|j|_dSr,)rrrXrdrkr|rrr0r0r1rs z(AsyncEvalsWithStreamingResponse.__init__rr*cCs t|jjSr,)rrr2r.r0r0r1r2sz$AsyncEvalsWithStreamingResponse.runsNrr0r0r0r1rsr)? __future__rtypingrrtyping_extensionsrZhttpxrtypesrr r _typesr r r rr_utilsrr_compatrZ runs.runsrrrrrrZ _resourcerr _responserrZ paginationrrZ _base_clientr r!Ztypes.eval_list_responser"Ztypes.eval_create_responser#Ztypes.eval_delete_responser$Ztypes.eval_update_responser%Ztypes.eval_retrieve_responser&Ztypes.shared_params.metadatar'__all__r(r)r3rr6rr0r0r0r1s<