170 Commits

Author SHA1 Message Date
Ian Arawjo
7aa597775b remove autorun 2025-03-06 14:44:18 -05:00
Ian Arawjo
c9df5478b1 typing fixes 2024-12-30 12:28:36 -05:00
HelenWeixuChen
2ac79355e3 Evalgen (#299)
* Add files via upload

Update README.md

* Add files via upload

Counting responses, thumbs up/down, new criteria card, EvalGen Report, Suggest Criteria

* Add files via upload

Counting responses, thumbs up/down, Suggest Criteria, EvalGen Report

* Delete utils.ts

* Delete typing.ts

* Delete text-fields-node.css

* Delete oai_utils.ts

* Delete executor.ts

* Delete MultiEvalNode.tsx

* Delete EvalGenModal.tsx

* Add files via upload

Maintain uid unique

* Add files via upload
2024-12-30 12:28:33 -05:00
Shreya Shankar
7031ebc6e8 Hook up backend executor with evalgen2 (#289)
* Hook up backend executor with evalgen2

* Adding UI indicators of how many LLM calls are executed
2024-12-30 12:27:39 -05:00
Ian Arawjo
c14ae4fac4 wip 2024-12-30 12:27:39 -05:00
Ian Arawjo
9f4d067341 wip 2024-12-30 12:27:39 -05:00
Ian Arawjo
94ba3b59af Add mb to Skeleton 2024-12-30 12:27:39 -05:00
Ian Arawjo
50407b8c9b First working version (no backend) 2024-12-30 12:27:39 -05:00
Ian Arawjo
1b093b8560 Fix typing and merge evalgen backend 2024-12-30 12:27:39 -05:00
Ian Arawjo
3a8a216630 wip 2024-12-30 12:27:39 -05:00
Ian Arawjo
949844f83b Fix feedback box in bottom of EvalGen modal 2024-12-30 12:27:39 -05:00
Ian Arawjo
f562532601 wip 2024-12-30 12:27:39 -05:00
Ian Arawjo
2775f7e163 wip 2024-12-30 12:27:38 -05:00
Ian Arawjo
b2da479ff1 init commit 2024-12-30 12:27:38 -05:00
ianarawjo
9ec7a3a4fc
Add prompt previews and cancel button to LLM scorer (#319)
Fix bug with default model not showing the selected one when adding new LLM model from the dropdown list.
2024-12-29 11:31:47 -05:00
Ian Arawjo
f5882768ba Update Google Gemini models 2024-12-28 17:56:51 -05:00
ianarawjo
ff813c7255
Revamped Example Flows (#316)
* Fix Try Me button spacing

* Added new examples

* Updated package version
2024-12-27 18:47:32 -05:00
ianarawjo
7e86f19aac
GenAI Data Synthesis for Tabular Data Node (#315)
* TabularDataNode supports Replace and Extend for AiGen (#312)

* Testing Values

* Fixed typing issue with Models in fromModelId

* TabularDataNode now supports table generation.
modified:   src/AiPopover.tsx
            Added support for table replacement
            and future support for extension.
modified:   src/TabularDataNode.tsx
            Added the AiPopover button and
            functionality for table replacement.
modified:   src/backend/ai.ts
            Added specific prompts and decoding
            for markdown table generation.
new file:   src/backend/tableUtils.ts
            Seperated the parsing for tables into
            a seperate utility file for better
            organization and future extensibility.

* Fixed typing issue with Models in fromModelId

* TabularDataNode now supports table generation.
modified:   src/AiPopover.tsx
            Added support for table replacement
            and future support for extension.
modified:   src/TabularDataNode.tsx
            Added the AiPopover button and
            functionality for table replacement.
modified:   src/backend/ai.ts
            Added specific prompts and decoding
            for markdown table generation.
new file:   src/backend/tableUtils.ts
            Seperated the parsing for tables into
            a seperate utility file for better
            organization and future extensibility.

Testing Values

* Added Extend Functionality to Table Popover.
modified:   src/AiPopover.tsx
            Removed unnecessary import.
            Changed handleCommandFill to work with
            autofillTable function in ai.ts.
modified:   src/TabularDataNode.tsx
            Removed Skeleton from Popover.
            Changed addMultipleRows such that it
            now renders the new rows correctly
            and removes the blank row.
modified:   src/backend/ai.ts
            Added autofillTable function and
            changed decodeTable so that they
            are flexible with both proper and
            improper markdown tables.
            Added new system message prompt
            specific to autofillTable.
            Removed unneccessary log statements.
removed:    src/backend/utils.ts
            Removed change.

* Added "add column" prompt & button in TablePopover

modified:   src/AiPopover.tsx
            Added handleGenerateColumn so that
            a column can be generated given
            a prompt.
            Added changes to the TablePopover UI
            Now extend is diveded into AddRow
            and AddColumn sections.
modified:   src/TabularDataNode.tsx
            Modified addColumns so that its safer.
            Added optional pass of rowValue to
            support generateColumn.
modified:   src/backend/ai.ts
            Added generateColumn and it's
            corresponding system message.
Cleaned up some comments and added missing commas.

* Generate Columns now considers item-by-item
processing of the rows for generating the
new column values.

modified:   src/AiPopover.tsx
            Fixed the key issue for onAddColumn.
modified:   src/TabularDataNode.tsx
            Changed addColumns to filter out
            previously added columns.
modified:   src/backend/ai.ts
            Changed generateColumns to process
            item-by-item to generate new columns.

* Fix bugs. Change OpenAI small model for GenAI features to GPT-4o.

* Update package version

* Remove gen diverse outputs switch in genAI for table

---------

Co-authored-by: Kraft-Cheese <114844630+Kraft-Cheese@users.noreply.github.com>
2024-12-19 15:38:23 -05:00
ianarawjo
1641abe975
Structured outputs support for Ollama, OpenAI, and Anthropic models (#313)
* Add structured outputs support for OpenAI and Ollama

* Extract outputs from tool_calls and refusal in OpenAI API responses

* Add tool use for Anthropic API. Add new Anthropic models.

* Add num_ctx to Ollama API call

* Update package version

* Update function calling example
2024-12-16 16:24:55 -05:00
Ian Arawjo
dd28754959 Remove rate limiting ceiling in bottleneck. Fix eslint to <9.0. 2024-12-11 10:17:04 -05:00
Ian Arawjo
98b140b5fa Add newest OpenAI and Anthropic models 2024-08-15 22:06:13 -04:00
ianarawjo
f6e1bfa38a
Add Together.ai and update Bedrock (#283)
* feat(bedrock_llama3): added support for Llama3 (#270)

- added also Claude 3 Opus to the list of models
- replaced hardcoded model Id strings with refs to NativeLLM enum

* chore: bump @mirai73/bedrock-fm library (#277)

- the new version adds source code to facilitate debugging

Co-authored-by: ianarawjo <fatso784@gmail.com>

* Adding together.ai support (#280)


---------

Co-authored-by: ianarawjo <fatso784@gmail.com>

* Add Together.ai and update Bedrock models

---------

Co-authored-by: Massimiliano Angelino <angmas@amazon.com>
Co-authored-by: Can Bal <canbal@users.noreply.github.com>
2024-05-17 20:17:18 -10:00
Ian Arawjo
e3259ecc1b Add new OpenAI models 2024-05-14 07:21:48 -10:00
Ian Arawjo
735268e331 Fix Claude carrying system message issue and bug with OpenAI_BaseURL loading 2024-04-29 21:17:49 -04:00
Ian Arawjo
af7f53f76e Fix bug loading OPENAI_BASE_URL from environ var 2024-04-28 14:29:58 -04:00
Ian Arawjo
4fa4b7bcc0 Escape braces in LLM scorer. Add OpenAI_BaseURL setting. 2024-04-26 07:24:31 -04:00
ianarawjo
6fa3092cd0
Add Multi-Eval node (#265)
* Port over and type MultiEvalNode code from the `multi-eval` branch

* Merge css changes from `multi-eval`

* Merge changes to inspector table view from `multi-eval`

* Criteria progress rings

* Debounce renders on text edits

* Add sandbox toggle to Python evals inside MultiEval

* Add uids to evals in MultiEval, for correct cache ids not dependent on name

* <Stack> scores

* Add debounce to editing code or prompts in eval UI

* Update package version
2024-04-25 13:51:25 -04:00
Ian Arawjo
2998c99f08 Bug fix for loading example flows in web version 2024-04-19 19:50:32 -04:00
Ian Arawjo
7126f4f4d4 Fix typing error and update package vers 2024-04-17 19:19:54 -04:00
Massimiliano Angelino
ffd947e636
Update to Bedrock integration (#258)
* fix(aws credentials): correct check for credentials

* chore(bedrock): bump @mirai73/ bedrock-fm library

* feat(bedrock): updating library and adding new mistral large model

- fix stop_sequences
2024-04-15 12:55:17 -04:00
Ian Arawjo
6b65d96369 Fix bug w/ non-updating custom providers in model list 2024-04-02 12:16:35 -04:00
yipengfei
5d4d196260
Fixed a bug that prevented custom models from appearing in model list. (#255) 2024-04-02 12:03:37 -04:00
ianarawjo
583ea6506f
Convert entire front-end to TypeScript. Add image model support. (And more!) (#252)
* Refactor: modularize response boxes into a separate component

* Type store.js. Change info to vars. NOTE: This may break backwards compat.

* Refactor addNodes in App.tsx to be simpler.

* Turn AlertModal into a Provider with useContext

* Remove fetch_from_backend.

* Add build/ to gitignore

* Add support for image models and add Dall-E models.

* Better rate limiting with Bottleneck

* Fix new Chrome bug with file import readers not appearing as arrays; and fix bug with exportCache

* Add ability to add custom right-click context menu items per node

* Convert to/from TF and Items nodes

* Add lazyloader for images

* Add compression to images by default before storing in cache

* Add image compression toggle in Global Settings

* Move Alert Provider to top level of index.js
2024-03-30 22:43:40 -04:00
ianarawjo
eb51d1cee9
Add Amazon Bedrock models to main (#251)
* Adding support for Amazon Bedrock models (#247)

* Create global setting for GenAI features provider, to support Bedrock (Anthropic) models as an alternative

* Reformats dropdown in PromptNode to use Mantine ContextMenu with a nested menu, to save space. 

* Remove build folder from git

* Fix context menu to close on click-off. Refactor context menu array code.

* Ensure context menu is positioned below the Add+ button, like a proper dropdown. 

* Toggle context menu off when clicking btn again.

---------

Co-authored-by: Massimiliano Angelino <angmas@amazon.com>
2024-03-30 17:59:17 -04:00
Ian Arawjo
ad84cfdecc Add Claude Haiku 2024-03-21 21:33:16 -04:00
Ian Arawjo
b832b7ba21 Rebuild 2024-03-17 00:01:58 -04:00
Ian Arawjo
91afd61c6c Bug fix toggle visibility
The visibility toggle was screwed up due to an eslint autofix code rewrite, it appears.
2024-03-16 23:59:47 -04:00
Ian Arawjo
fac5579eee Rebuild 2024-03-15 19:59:05 -04:00
Ian Arawjo
f4e06fd00d Bug fix fill templated settings vars 2024-03-15 19:56:19 -04:00
ianarawjo
bdfeb5c26f
Add human ratings to inspectors (#244)
* Add human ratings to inspectors

* Store human labels in cache, not resp objs

* Change rating UI to pull from Zustand store

* Lazy load inspectors

* Update version and rebuild app
2024-03-14 13:02:47 -04:00
Ian Arawjo
677073ef62 Clean escaped braces before eval 2024-03-07 15:36:40 -05:00
ianarawjo
4cb97d87f7
Bug fix code on save (#241) 2024-03-07 14:31:55 -05:00
ianarawjo
0f4275bc75
Add Claude 3 and Pyodide sandboxing (#237)
Adds pyodide WebWorker to run Python scripts, thanks to idea by Shreya.

* Add sandbox option to Python eval nodes.

* Add new Anthropic models

* Disable guards for Python evals on server

* Fix bug with detecting async func in runOverResponses

---------

Co-authored-by: Shreya Shankar <ss.shankar505@gmail.com>
2024-03-05 23:15:35 -05:00
ianarawjo
0a45383b95
Generate code evaluators (#231)
* Adds a purple GenAI button to Code Evaluator Nodes, to allow easier creation of evaluation functions. (NOTE: This, like the TextFields and Items Nodes GenAI features, is experimental and requires an OpenAI API key to access.)

* Adds a drop-down to LLM evaluators

* Ensures LLM evaluators load cache'd responses on load

* Fixes a bug where right-clicking in pop-up Inspectors would bring up the node context menu.

* Internally, refactors evaluator nodes to have inner components that take care of running evaluations, in preparation for multi-eval and running evals elsewhere
2024-02-27 20:27:41 -05:00
Ian Arawjo
5d666643bd Fix TypeScript type errors 2024-02-24 14:31:36 -05:00
ianarawjo
bd35ecddb2
Add Prettier and ESLint formatting (#227)
* Add and run prettier

* Add eslint and code fixes after formatting (#223)

* chore(formatting): config files and packages

* chore(formatting): package.json

* chore(formatting): applying formatting

changes obtained by applying the previous commit and running `npx prettier -w .`

* chore(formatting): added formatting and linting to react app

* chore(formatting): fixes

* chore(eslint): apply fixes to utils.ts

* rebuild

---------

Co-authored-by: Massimiliano Angelino <angmas@amazon.com>

* Rebuild

---------

Co-authored-by: Massimiliano Angelino <angmas@amazon.com>
2024-02-24 11:32:05 -05:00
ianarawjo
d8d02424e2
Add new OpenAI models. (#225) 2024-02-17 12:35:38 -05:00
Ian Arawjo
625619c0b0 Rebuild app and update package version 2024-02-11 16:25:08 -05:00
Ian Arawjo
cff7a470bc Export metavars to excel 2024-02-11 16:22:55 -05:00
Ian Arawjo
08047d2c55 Bug fix join node 2024-01-22 12:56:34 -05:00