ChainForge/chainforge/oaievals/unsolvable_questions.cforge

1 line
265 KiB
Plaintext
Raw Permalink Normal View History

TypeScript backend, HuggingFace models, JavaScript evaluators, Comment Nodes, and more (#81) * Beginning to convert Python backend to Typescript * Change all fetch() calls to fetch_from_backend switcher * wip converting query.py to query.ts * wip started utils.js conversion. Tested that OpenAI API call works * more progress on converting utils.py to Typescript * jest tests for query, utils, template.ts. Confirmed PromptPipeline works. * wip converting queryLLM in flask_app to TS * Tested queryLLM and StorageCache compressed saving/loading * wip execute() in backend.ts * Added execute() and tested w concrete func. Need to test eval() * Added craco for optional webpack config. Config'd for TypeScript with Node.js packages browserify'd * Execute JS code on iframe sandbox * Tested and working JS Evaluator execution. * wip swapping backends * Tested TypeScript backendgit status! :) woot * Added fetchEnvironAPIKeys to Flask server to fetch os.environ keys when running locally * Route Anthropic calls through Flask when running locally * Added info button to Eval nodes. Rebuilt react * Edits to info modal on Eval node * Remove/error out on Python eval nodes when not running locally. * Check browser compat and display error if not supported * Changed all example flows to use JS. Bug fix in query.ts * Refactored to LLMProvider to streamline model additions * Added HuggingFace models API * Added back Dalai call support, routing through Flask * Remove flask app calls and socketio server that are no longer used * Added Comment Nodes. Rebuilt react. * Fix PaLM temp=0 build, update package vers and rebuild react
2023-06-30 15:11:20 -04:00
{"flow": {"nodes": [{"width": 312, "height": 311, "id": "prompt-unsolvable_questions", "type": "prompt", "data": {"prompt": "{prompt}", "n": 1, "llms": [{"key": "aa3c0f03-22bd-416e-af4d-4bf5c4278c99", "settings": {"system_msg": "You are UnsolvableGPT. You will be provided a question and some context for the question. Using only the context to answer the question determine if it is \"Solveable\" or \"Unsolveable\". Respond with only one word without punctuation, either: \"Solveable\": The submitted question is solveable with the context provided alongside it and no other outside information OR \"Unsolveable\": The submitted question is unsolveable with the context provided alongside it. There is not enough context to answer the question. Remember, only answer with \"Solveable\" OR \"Unsolveable\", do not include anything else.", "temperature": 1, "functions": [], "function_call": "", "top_p": 1, "stop": [], "presence_penalty": 0, "frequency_penalty": 0}, "name": "GPT3.5", "emoji": "\ud83d\ude42", "model": "gpt-3.5-turbo", "base_model": "gpt-3.5-turbo", "temp": 1, "formData": {"shortname": "GPT3.5", "model": "gpt-3.5-turbo", "system_msg": "You are UnsolvableGPT. You will be provided a question and some context for the question. Using only the context to answer the question determine if it is \"Solveable\" or \"Unsolveable\". Respond with only one word without punctuation, either: \"Solveable\": The submitted question is solveable with the context provided alongside it and no other outside information OR \"Unsolveable\": The submitted question is unsolveable with the context provided alongside it. There is not enough context to answer the question. Remember, only answer with \"Solveable\" OR \"Unsolveable\", do not include anything else.", "temperature": 1, "functions": "", "function_call": "", "top_p": 1, "stop": "", "presence_penalty": 0, "frequency_penalty": 0}}]}, "position": {"x": 448, "y": 224}, "selected": false, "positionAbsolute": {"x": 448, "y": 224}, "dragging": false}, {"width": 333, "height": 182, "id": "eval-unsolvable_questions", "type": "evaluator", "data": {"code": "function evaluate(response) {\n\tlet ideal = response.meta['Ideal'];\n\treturn response.text.startsWith(ideal);\n}", "language": "javascript"}, "position": {"x": 820, "y": 150}, "positionAbsolute": {"x": 820, "y": 150}}, {"width": 228, "height": 196, "id": "vis-unsolvable_questions", "type": "vis", "data": {"input": "eval-unsolvable_questions"}, "position": {"x": 1200, "y": 250}, "positionAbsolute": {"x": 1200, "y": 250}}, {"width": 302, "height": 260, "id": "inspect-unsolvable_questions", "type": "inspect", "data": {"input": "prompt-unsolvable_questions"}, "position": {"x": 820, "y": 400}, "positionAbsolute": {"x": 820, "y": 400}}, {"width": 423, "height": 417, "id": "table-unsolvable_questions", "type": "table", "data": {"rows": [{"prompt": "Question: Who did some early female settlers marry?\n Context: Some early male settlers married Indigenous American women and had informal unions with them. Early contact between Indigenous Americans and Europeans was often charged with tension, but also had moments of friendship, cooperation, and intimacy. Marriages took place in both English and Latin colonies between European men and Native women. For instance, on April 5, 1614, Pocahontas, a Powhatan woman in present-day Virginia, married the Englishman John Rolfe of Jamestown. Their son Thomas Rolfe was an ancestor to many descendants in First Families of Virginia. As a result, English laws did not exclude people with some Indigenous American ancestry from being considered English or white.", "ideal": "Unsolveable"}, {"prompt": "Question: What did Walter von Dyck publish in 1870?\n Context: The convergence of these various sources into a uniform theory of groups started with Camille Jordan's Trait\u00bb des substitutions et des \u00bbquations alg\u00bbbriques (1870). Walther von Dyck (1882) introduced the idea of specifying a group by means of generators and relations, and was also the first to give an axiomatic definition of an \"abstract group\", i