mirror of
https://github.com/ianarawjo/ChainForge.git
synced 2025-03-14 08:16:37 +00:00
Merge branch 'main' of https://github.com/ianarawjo/ChainForge
This commit is contained in:
commit
8f88061dc8
44
README.md
44
README.md
@ -1,7 +1,7 @@
|
||||
# ⛓️🛠️ ChainForge
|
||||
**An open-source visual programming environment for battle-testing prompts to LLMs.**
|
||||
|
||||
<img width="1599" alt="prompt-injection-test" src="https://github.com/ianarawjo/ChainForge/assets/5251713/83757804-4288-4fc2-b28d-fd0826bae6a1">
|
||||
<img width="1517" alt="banner" src="https://github.com/ianarawjo/ChainForge/assets/5251713/570879ef-ef8a-4e00-b37c-b49bc3c1a370">
|
||||
|
||||
ChainForge is a data flow prompt engineering environment for analyzing and evaluating LLM responses. It is geared towards early-stage, quick-and-dirty exploration of prompts and response quality that goes beyond ad-hoc chatting with individual LLMs. With ChainForge, you can:
|
||||
- Query multiple LLMs at once to test prompt ideas and variations quickly and effectively.
|
||||
@ -10,13 +10,17 @@ ChainForge is a data flow prompt engineering environment for analyzing and evalu
|
||||
|
||||
ChainForge comes with a number of example evaluation flows to give you a sense of what's possible, including 188 example flows generated from benchmarks in OpenAI evals.
|
||||
|
||||
**This is an open alpha of Chainforge.** Functionality is powerful but limited. We currently support OpenAI models GPT3.5 and GPT4, HuggingFace models on the Inference API, Anthropic's Claude, Google PaLM2, and [Dalai](https://github.com/cocktailpeanut/dalai)-hosted models Alpaca and Llama. You can change the exact model and individual model settings. Visualization nodes support numeric and boolean evaluation metrics. Try it and let us know what you think! :)
|
||||
# Try it @ https://chainforge.ai/play/
|
||||
|
||||
**This is an open beta of Chainforge.** Functionality is powerful but limited. We currently support OpenAI models GPT3.5 and GPT4, HuggingFace models on the Inference API, Anthropic's Claude, Google PaLM2, Azure OpenAI endpoints, and [Dalai](https://github.com/cocktailpeanut/dalai)-hosted models Alpaca and Llama. You can change the exact model and individual model settings. Visualization nodes support numeric and boolean evaluation metrics. Try it and let us know what you think! :)
|
||||
|
||||
ChainForge is built on [ReactFlow](https://reactflow.dev) and [Flask](https://flask.palletsprojects.com/en/2.3.x/).
|
||||
|
||||
# Installation
|
||||
# Installation (local machine)
|
||||
|
||||
To install Chainforge alpha, make sure you have Python 3.8 or higher, then run
|
||||
The web version of ChainForge (https://chainforge.ai/play/) has a limited feature set. In a locally installed version you can load API keys automatically from environment variables, write Python code to evaluate LLM responses, or query locally-run Alpaca/Llama models hosted via Dalai.
|
||||
|
||||
To install Chainforge on your machine, make sure you have Python 3.8 or higher, then run
|
||||
|
||||
```bash
|
||||
pip install chainforge
|
||||
@ -28,26 +32,26 @@ Once installed, do
|
||||
chainforge serve
|
||||
```
|
||||
|
||||
Open [localhost:8000](http://localhost:8000/) in a Google Chrome or Firefox browser (other browsers are currently unsupported).
|
||||
Open [localhost:8000](http://localhost:8000/) in a Google Chrome or Firefox browser.
|
||||
|
||||
You can set your API keys by clicking the Settings icon in the top-right corner. If you prefer to not worry about this everytime you open ChainForge, we recommend that save your OpenAI, Anthropic, and/or Google PaLM API keys to your local environment. For more details, see the [Installation Guide](https://github.com/ianarawjo/ChainForge/blob/main/INSTALL_GUIDE.md).
|
||||
|
||||
## Example evaluation flows
|
||||
# Example evaluation flows
|
||||
|
||||
We've prepared a couple example flows to give you a sense of what's possible with Chainforge.
|
||||
Click the "Example Flows" button on the top-right corner and select one. Here is a basic comparison example, plotting the length of responses across different models and arguments for the prompt parameter `{game}`:
|
||||
|
||||
<img width="1593" alt="basic-compare" src="https://github.com/ianarawjo/ChainForge/assets/5251713/43c87ab7-aabd-41ba-8d9b-e7e9ebe25c75">
|
||||
|
||||
You can also conduct **ground truth evaluations** using Tabular Data nodes. For instance, we can compare each LLM's ability to answer math problems by comparing each response to the expected answer:
|
||||
You can also conduct **ground truth evaluations** using Tabular Data nodes. For instance, we can compare each LLM's ability to answer math problems by comparing each response to the expected answer:
|
||||
|
||||
<img width="1770" alt="Screen Shot 2023-06-11 at 11 51 28 AM" src="https://github.com/ianarawjo/ChainForge/assets/5251713/3a038fa6-46af-42d8-ac82-e94f7c239b10">
|
||||
<img width="1775" alt="Screen Shot 2023-07-04 at 9 21 50 AM" src="https://github.com/ianarawjo/ChainForge/assets/5251713/6d842f7a-f747-44f9-b317-95bec73653c5">
|
||||
|
||||
For finer details about the features of available nodes, check out the [Node Guide](https://github.com/ianarawjo/ChainForge/blob/main/GUIDE.md).
|
||||
|
||||
# Features
|
||||
|
||||
A key goal of ChainForge is facilitating **comparison** and **evaluation** of prompts and models, and (in the future) prompt chains. Basic features are:
|
||||
A key goal of ChainForge is facilitating **comparison** and **evaluation** of prompts and models. Basic features are:
|
||||
- **Prompt permutations**: Setup a prompt template and feed it variations of input variables. ChainForge will prompt all selected LLMs with all possible permutations of the input prompt, so that you can get a better sense of prompt quality. You can also chain prompt templates at arbitrary depth (e.g., to compare templates).
|
||||
- **Model settings**: Change the settings of supported models, and compare across settings. For instance, you can measure the impact of a system message on ChatGPT by adding several ChatGPT models, changing individual settings, and nicknaming each one. ChainForge will send out queries to each version of the model.
|
||||
- **Evaluation nodes**: Probe LLM responses in a chain and test them (classically) for some desired behavior. At a basic level, this is Python script based. We plan to add preset evaluator nodes for common use cases in the near future (e.g., name-entity recognition). Note that you can also chain LLM responses into prompt templates to help evaluate outputs cheaply before more extensive evaluation methods.
|
||||
@ -61,6 +65,23 @@ We've also found that some users simply want to use ChainForge to make tons of p
|
||||
|
||||
For more specific details, see the [User Guide](https://github.com/ianarawjo/ChainForge/blob/main/GUIDE.md).
|
||||
|
||||
# Share with others
|
||||
|
||||
The web version of ChainForge (https://chainforge.ai/play/) includes a Share button.
|
||||
|
||||
Simply click Share to generate a unique link for your flow and copy it to your clipboard:
|
||||
|
||||

|
||||
|
||||
For instance, here's a experiment I made that tries to get an LLM to reveal a secret key: https://chainforge.ai/play/?f=28puvwc788bog
|
||||
|
||||
> **Note**
|
||||
> To prevent abuse, you can only share up to 10 flows at a time, and each flow must be <5MB after compression.
|
||||
> If you share more than 10 flows, the oldest link will break, so make sure to always Export important flows to `cforge` files,
|
||||
> and use Share to only pass data ephemerally.
|
||||
|
||||
----------------------------------
|
||||
|
||||
# Development
|
||||
|
||||
ChainForge was created by [Ian Arawjo](http://ianarawjo.com/index.html), a postdoctoral scholar in Harvard HCI's [Glassman Lab](http://glassmanlab.seas.harvard.edu/) with support from the Harvard HCI community, especially PhD student [Priyan Vaithilingam](https://priyan.info).
|
||||
@ -72,7 +93,6 @@ We provide ongoing releases of this tool in the hopes that others find it useful
|
||||
## Future Planned Features
|
||||
|
||||
Highest priority:
|
||||
- **Hosting on chainforge.ai**: Host a version that works entirely in the browser, no install or login necessary
|
||||
- **Improved vis and inspect nodes**: Better UX and more features, such as collapsing variable groups in response inspectors and more control over visualizations displayed in vis nodes beyond the default
|
||||
|
||||
Medium-to-low priority:
|
||||
@ -96,7 +116,9 @@ Unlike these projects, we are focusing on supporting evaluation across prompts,
|
||||
|
||||
## How to collaborate?
|
||||
|
||||
We are looking for open-source collaborators. The best way to do this, at the moment, is simply to implement the requested feature / bug fix and submit a Pull Request. If you want to report a bug or request a feature, open an [Issue](https://github.com/ianarawjo/ChainForge/issues).
|
||||
We are looking for open-source collaborators. The best way to collaborate, at the moment, is simply to implement the requested feature / bug fix and submit a Pull Request. If you want to report a bug or request a feature, open an [Issue](https://github.com/ianarawjo/ChainForge/issues).
|
||||
|
||||
_(If you are an investor or funder, send us a message via email.)_
|
||||
|
||||
# License
|
||||
|
||||
|
@ -1,15 +1,15 @@
|
||||
{
|
||||
"files": {
|
||||
"main.css": "/static/css/main.98db168d.css",
|
||||
"main.js": "/static/js/main.c56530db.js",
|
||||
"main.js": "/static/js/main.b4009f20.js",
|
||||
"static/js/787.4c72bb55.chunk.js": "/static/js/787.4c72bb55.chunk.js",
|
||||
"index.html": "/index.html",
|
||||
"main.98db168d.css.map": "/static/css/main.98db168d.css.map",
|
||||
"main.c56530db.js.map": "/static/js/main.c56530db.js.map",
|
||||
"main.b4009f20.js.map": "/static/js/main.b4009f20.js.map",
|
||||
"787.4c72bb55.chunk.js.map": "/static/js/787.4c72bb55.chunk.js.map"
|
||||
},
|
||||
"entrypoints": [
|
||||
"static/css/main.98db168d.css",
|
||||
"static/js/main.c56530db.js"
|
||||
"static/js/main.b4009f20.js"
|
||||
]
|
||||
}
|
@ -1 +1 @@
|
||||
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="/favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="Web site created using create-react-app"/><link rel="apple-touch-icon" href="/logo192.png"/><link rel="manifest" href="/manifest.json"/><title>Chainforge</title><script defer="defer" src="/static/js/main.c56530db.js"></script><link href="/static/css/main.98db168d.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
|
||||
<!doctype html><html lang="en"><head><meta charset="utf-8"/><script async src="https://www.googletagmanager.com/gtag/js?id=G-RN3FDBLMCR"></script><script>function gtag(){dataLayer.push(arguments)}window.dataLayer=window.dataLayer||[],gtag("js",new Date),gtag("config","G-RN3FDBLMCR")</script><link rel="icon" href="/favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="A visual programming environment for prompt engineering"/><link rel="apple-touch-icon" href="/logo192.png"/><link rel="manifest" href="/manifest.json"/><title>ChainForge</title><script defer="defer" src="/static/js/main.b4009f20.js"></script><link href="/static/css/main.98db168d.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
|
3
chainforge/react-server/build/static/js/main.b4009f20.js
Normal file
3
chainforge/react-server/build/static/js/main.b4009f20.js
Normal file
File diff suppressed because one or more lines are too long
@ -0,0 +1,238 @@
|
||||
/*
|
||||
* @copyright 2016 Sean Connelly (@voidqk), http://syntheti.cc
|
||||
* @license MIT
|
||||
* @preserve Project Home: https://github.com/voidqk/polybooljs
|
||||
*/
|
||||
|
||||
/*
|
||||
object-assign
|
||||
(c) Sindre Sorhus
|
||||
@license MIT
|
||||
*/
|
||||
|
||||
/*
|
||||
* based on code from:
|
||||
*
|
||||
* @license RequireJS text 0.25.0 Copyright (c) 2010-2011, The Dojo Foundation All Rights Reserved.
|
||||
* Available via the MIT or new BSD license.
|
||||
* see: http://github.com/jrburke/requirejs for details
|
||||
*/
|
||||
|
||||
/* @license
|
||||
Papa Parse
|
||||
v5.4.1
|
||||
https://github.com/mholt/PapaParse
|
||||
License: MIT
|
||||
*/
|
||||
|
||||
/*!
|
||||
Copyright (c) 2018 Jed Watson.
|
||||
Licensed under the MIT License (MIT), see
|
||||
http://jedwatson.github.io/classnames
|
||||
*/
|
||||
|
||||
/*!
|
||||
* The buffer module from node.js, for the browser.
|
||||
*
|
||||
* @author Feross Aboukhadijeh <https://feross.org>
|
||||
* @license MIT
|
||||
*/
|
||||
|
||||
/*!
|
||||
* Determine if an object is a Buffer
|
||||
*
|
||||
* @author Feross Aboukhadijeh <https://feross.org>
|
||||
* @license MIT
|
||||
*/
|
||||
|
||||
/*!
|
||||
* pad-left <https://github.com/jonschlinkert/pad-left>
|
||||
*
|
||||
* Copyright (c) 2014-2015, Jon Schlinkert.
|
||||
* Licensed under the MIT license.
|
||||
*/
|
||||
|
||||
/*!
|
||||
* repeat-string <https://github.com/jonschlinkert/repeat-string>
|
||||
*
|
||||
* Copyright (c) 2014-2015, Jon Schlinkert.
|
||||
* Licensed under the MIT License.
|
||||
*/
|
||||
|
||||
/*!
|
||||
* The buffer module from node.js, for the browser.
|
||||
*
|
||||
* @author Feross Aboukhadijeh <feross@feross.org> <http://feross.org>
|
||||
* @license MIT
|
||||
*/
|
||||
|
||||
/*!
|
||||
* The buffer module from node.js, for the browser.
|
||||
*
|
||||
* @author Feross Aboukhadijeh <https://feross.org>
|
||||
* @license MIT
|
||||
*/
|
||||
|
||||
/*!
|
||||
* decimal.js v10.4.3
|
||||
* An arbitrary-precision Decimal type for JavaScript.
|
||||
* https://github.com/MikeMcl/decimal.js
|
||||
* Copyright (c) 2022 Michael Mclaughlin <M8ch88l@gmail.com>
|
||||
* MIT Licence
|
||||
*/
|
||||
|
||||
/*!
|
||||
* The buffer module from node.js, for the browser.
|
||||
*
|
||||
* @author Feross Aboukhadijeh <https://feross.org>
|
||||
* @license MIT
|
||||
*/
|
||||
|
||||
/*! @license
|
||||
==========================================================================
|
||||
SproutCore -- JavaScript Application Framework
|
||||
copyright 2006-2009, Sprout Systems Inc., Apple Inc. and contributors.
|
||||
Permission is hereby granted, free of charge, to any person obtaining a
|
||||
copy of this software and associated documentation files (the "Software"),
|
||||
to deal in the Software without restriction, including without limitation
|
||||
the rights to use, copy, modify, merge, publish, distribute, sublicense,
|
||||
and/or sell copies of the Software, and to permit persons to whom the
|
||||
Software is furnished to do so, subject to the following conditions:
|
||||
The above copyright notice and this permission notice shall be included in
|
||||
all copies or substantial portions of the Software.
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
||||
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
|
||||
DEALINGS IN THE SOFTWARE.
|
||||
SproutCore and the SproutCore logo are trademarks of Sprout Systems, Inc.
|
||||
For more information about SproutCore, visit http://www.sproutcore.com
|
||||
==========================================================================
|
||||
@license */
|
||||
|
||||
/*! Native Promise Only
|
||||
v0.8.1 (c) Kyle Simpson
|
||||
MIT License: http://getify.mit-license.org
|
||||
*/
|
||||
|
||||
/*! ieee754. BSD-3-Clause License. Feross Aboukhadijeh <https://feross.org/opensource> */
|
||||
|
||||
/*! regenerator-runtime -- Copyright (c) 2014-present, Facebook, Inc. -- license (MIT): https://github.com/facebook/regenerator/blob/main/LICENSE */
|
||||
|
||||
/*! sheetjs (C) 2013-present SheetJS -- http://sheetjs.com */
|
||||
|
||||
/**
|
||||
* @license Complex.js v2.1.1 12/05/2020
|
||||
*
|
||||
* Copyright (c) 2020, Robert Eisele (robert@xarg.org)
|
||||
* Dual licensed under the MIT or GPL Version 2 licenses.
|
||||
**/
|
||||
|
||||
/**
|
||||
* @license Fraction.js v4.2.0 05/03/2022
|
||||
* https://www.xarg.org/2014/03/rational-numbers-in-javascript/
|
||||
*
|
||||
* Copyright (c) 2021, Robert Eisele (robert@xarg.org)
|
||||
* Dual licensed under the MIT or GPL Version 2 licenses.
|
||||
**/
|
||||
|
||||
/**
|
||||
* @license React
|
||||
* react-dom.production.min.js
|
||||
*
|
||||
* Copyright (c) Facebook, Inc. and its affiliates.
|
||||
*
|
||||
* This source code is licensed under the MIT license found in the
|
||||
* LICENSE file in the root directory of this source tree.
|
||||
*/
|
||||
|
||||
/**
|
||||
* @license React
|
||||
* react-is.production.min.js
|
||||
*
|
||||
* Copyright (c) Facebook, Inc. and its affiliates.
|
||||
*
|
||||
* This source code is licensed under the MIT license found in the
|
||||
* LICENSE file in the root directory of this source tree.
|
||||
*/
|
||||
|
||||
/**
|
||||
* @license React
|
||||
* react-jsx-runtime.production.min.js
|
||||
*
|
||||
* Copyright (c) Facebook, Inc. and its affiliates.
|
||||
*
|
||||
* This source code is licensed under the MIT license found in the
|
||||
* LICENSE file in the root directory of this source tree.
|
||||
*/
|
||||
|
||||
/**
|
||||
* @license React
|
||||
* react.production.min.js
|
||||
*
|
||||
* Copyright (c) Facebook, Inc. and its affiliates.
|
||||
*
|
||||
* This source code is licensed under the MIT license found in the
|
||||
* LICENSE file in the root directory of this source tree.
|
||||
*/
|
||||
|
||||
/**
|
||||
* @license React
|
||||
* scheduler.production.min.js
|
||||
*
|
||||
* Copyright (c) Facebook, Inc. and its affiliates.
|
||||
*
|
||||
* This source code is licensed under the MIT license found in the
|
||||
* LICENSE file in the root directory of this source tree.
|
||||
*/
|
||||
|
||||
/**
|
||||
* @license React
|
||||
* use-sync-external-store-shim.production.min.js
|
||||
*
|
||||
* Copyright (c) Facebook, Inc. and its affiliates.
|
||||
*
|
||||
* This source code is licensed under the MIT license found in the
|
||||
* LICENSE file in the root directory of this source tree.
|
||||
*/
|
||||
|
||||
/**
|
||||
* @license React
|
||||
* use-sync-external-store-shim/with-selector.production.min.js
|
||||
*
|
||||
* Copyright (c) Facebook, Inc. and its affiliates.
|
||||
*
|
||||
* This source code is licensed under the MIT license found in the
|
||||
* LICENSE file in the root directory of this source tree.
|
||||
*/
|
||||
|
||||
/**
|
||||
* Prism: Lightweight, robust, elegant syntax highlighting
|
||||
*
|
||||
* @license MIT <https://opensource.org/licenses/MIT>
|
||||
* @author Lea Verou <https://lea.verou.me>
|
||||
* @namespace
|
||||
* @public
|
||||
*/
|
||||
|
||||
/** @license React v16.13.1
|
||||
* react-is.production.min.js
|
||||
*
|
||||
* Copyright (c) Facebook, Inc. and its affiliates.
|
||||
*
|
||||
* This source code is licensed under the MIT license found in the
|
||||
* LICENSE file in the root directory of this source tree.
|
||||
*/
|
||||
|
||||
/** @license React v17.0.2
|
||||
* react-is.production.min.js
|
||||
*
|
||||
* Copyright (c) Facebook, Inc. and its affiliates.
|
||||
*
|
||||
* This source code is licensed under the MIT license found in the
|
||||
* LICENSE file in the root directory of this source tree.
|
||||
*/
|
||||
|
||||
/** @license URI.js v4.4.1 (c) 2011 Gary Court. License: http://github.com/garycourt/uri-js */
|
File diff suppressed because one or more lines are too long
@ -2,13 +2,22 @@
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<!-- Google tag (gtag.js) -->
|
||||
<script async src="https://www.googletagmanager.com/gtag/js?id=G-RN3FDBLMCR"></script>
|
||||
<script>
|
||||
window.dataLayer = window.dataLayer || [];
|
||||
function gtag(){dataLayer.push(arguments);}
|
||||
gtag('js', new Date());
|
||||
|
||||
gtag('config', 'G-RN3FDBLMCR');
|
||||
</script>
|
||||
<!-- Logo from bqlqn @ flaticon.com; used with attribution: https://www.flaticon.com/free-icon/link_1209950 -->
|
||||
<link rel="icon" href="%PUBLIC_URL%/favicon.ico" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<meta name="theme-color" content="#000000" />
|
||||
<meta
|
||||
name="description"
|
||||
content="Web site created using create-react-app"
|
||||
content="A visual programming environment for prompt engineering"
|
||||
/>
|
||||
<link rel="apple-touch-icon" href="%PUBLIC_URL%/logo192.png" />
|
||||
<!--
|
||||
@ -25,7 +34,7 @@
|
||||
work correctly both with client-side routing and a non-root public URL.
|
||||
Learn how to configure a non-root public URL by running `npm run build`.
|
||||
-->
|
||||
<title>Chainforge</title>
|
||||
<title>ChainForge</title>
|
||||
</head>
|
||||
<body>
|
||||
<noscript>You need to enable JavaScript to run this app.</noscript>
|
||||
|
207
chainforge/react-server/src/App.js
vendored
207
chainforge/react-server/src/App.js
vendored
@ -5,12 +5,10 @@ import React, { useState, useCallback, useRef, useEffect } from 'react';
|
||||
import ReactFlow, {
|
||||
Controls,
|
||||
Background,
|
||||
useReactFlow,
|
||||
useViewport,
|
||||
setViewport,
|
||||
} from 'react-flow-renderer';
|
||||
import { Button, Menu, LoadingOverlay, Text, Box, List } from '@mantine/core';
|
||||
import { IconSettings, IconTextPlus, IconTerminal, IconCsv, IconSettingsAutomation } from '@tabler/icons-react';
|
||||
import { Button, Menu, LoadingOverlay, Text, Box, List, Loader } from '@mantine/core';
|
||||
import { useClipboard } from '@mantine/hooks';
|
||||
import { IconSettings, IconTextPlus, IconTerminal, IconCsv, IconSettingsAutomation, IconFileSymlink } from '@tabler/icons-react';
|
||||
import TextFieldsNode from './TextFieldsNode'; // Import a custom node
|
||||
import PromptNode from './PromptNode';
|
||||
import EvaluatorNode from './EvaluatorNode';
|
||||
@ -26,6 +24,7 @@ import ExampleFlowsModal from './ExampleFlowsModal';
|
||||
import AreYouSureModal from './AreYouSureModal';
|
||||
import { getDefaultModelFormData, getDefaultModelSettings } from './ModelSettingSchemas';
|
||||
import { v4 as uuid } from 'uuid';
|
||||
import LZString from 'lz-string';
|
||||
import { EXAMPLEFLOW_1 } from './example_flows';
|
||||
import './text-fields-node.css';
|
||||
|
||||
@ -87,6 +86,27 @@ const nodeTypes = {
|
||||
// we have access to the Flask backend for, e.g., Python code evaluation.
|
||||
const IS_RUNNING_LOCALLY = APP_IS_RUNNING_LOCALLY();
|
||||
|
||||
// Try to get a GET param in the URL, representing the shared flow.
|
||||
// Returns undefined if not found.
|
||||
const getSharedFlowURLParam = () => {
|
||||
// Get the current URL
|
||||
const curr_url = new URL(window.location.href);
|
||||
|
||||
// Get the search parameters from the URL
|
||||
const params = new URLSearchParams(curr_url.search);
|
||||
|
||||
// Try to retrieve an 'f' parameter (short for flow)
|
||||
const shared_flow_uid = params.get('f');
|
||||
|
||||
if (shared_flow_uid) {
|
||||
// Check if it's a base36 string:
|
||||
const is_base36 = /^[0-9a-z]+$/i;
|
||||
if (shared_flow_uid.length > 1 && is_base36.test(shared_flow_uid))
|
||||
return shared_flow_uid;
|
||||
}
|
||||
return undefined;
|
||||
};
|
||||
|
||||
// const connectionLineStyle = { stroke: '#ddd' };
|
||||
const snapGrid = [16, 16];
|
||||
let saveIntervalInitialized = false;
|
||||
@ -100,6 +120,10 @@ const App = () => {
|
||||
const [rfInstance, setRfInstance] = useState(null);
|
||||
const [autosavingInterval, setAutosavingInterval] = useState(null);
|
||||
|
||||
// For 'share' button
|
||||
const clipboard = useClipboard({ timeout: 1500 });
|
||||
const [waitingForShare, setWaitingForShare] = useState(false);
|
||||
|
||||
// For modal popup to set global settings like API keys
|
||||
const settingsModal = useRef(null);
|
||||
|
||||
@ -186,6 +210,7 @@ const App = () => {
|
||||
|
||||
const handleError = (err) => {
|
||||
setIsLoading(false);
|
||||
setWaitingForShare(false);
|
||||
if (alertModal.current)
|
||||
alertModal.current.trigger(err.message);
|
||||
console.error(err.message);
|
||||
@ -321,11 +346,13 @@ const App = () => {
|
||||
}).catch(handleError);
|
||||
}, [handleError]);
|
||||
|
||||
const importFlowFromJSON = useCallback((flowJSON) => {
|
||||
const importFlowFromJSON = useCallback((flowJSON, rf_inst) => {
|
||||
const rf = rf_inst || rfInstance;
|
||||
|
||||
// Detect if there's no cache data
|
||||
if (!flowJSON.cache) {
|
||||
// Support for loading old flows w/o cache data:
|
||||
loadFlow(flowJSON, rfInstance);
|
||||
loadFlow(flowJSON, rf);
|
||||
return;
|
||||
}
|
||||
|
||||
@ -337,11 +364,11 @@ const App = () => {
|
||||
// before we can load the flow itself...
|
||||
importCache(cache).then(() => {
|
||||
// We load the ReactFlow instance last
|
||||
loadFlow(flow, rfInstance);
|
||||
loadFlow(flow, rf);
|
||||
}).catch(err => {
|
||||
// On an error, still try to load the flow itself:
|
||||
handleError("Error encountered when importing cache data:" + err.message + "\n\nTrying to load flow regardless...");
|
||||
loadFlow(flow, rfInstance);
|
||||
loadFlow(flow, rf);
|
||||
});
|
||||
}, [rfInstance]);
|
||||
|
||||
@ -444,11 +471,93 @@ const App = () => {
|
||||
confirmationModal.current.trigger();
|
||||
}, [confirmationModal, resetFlow, setConfirmationDialogProps]);
|
||||
|
||||
// When the user clicks the 'Share Flow' button
|
||||
const onClickShareFlow = useCallback(async () => {
|
||||
if (IS_RUNNING_LOCALLY) {
|
||||
handleError(new Error('Cannot upload flow to server database when running locally: Feature only exists on hosted version of ChainForge.'));
|
||||
return;
|
||||
} else if (waitingForShare === true) {
|
||||
handleError(new Error('A share request is already in progress. Wait until the current share finishes before clicking again.'));
|
||||
return;
|
||||
}
|
||||
|
||||
// Helper function
|
||||
function isFileSizeLessThan5MB(str) {
|
||||
const encoder = new TextEncoder();
|
||||
const encodedString = encoder.encode(str);
|
||||
const fileSizeInBytes = encodedString.length;
|
||||
const fileSizeInMB = fileSizeInBytes / (1024 * 1024); // Convert bytes to megabytes
|
||||
return fileSizeInMB < 5;
|
||||
}
|
||||
|
||||
setWaitingForShare(true);
|
||||
|
||||
// Package up the current flow:
|
||||
const flow = rfInstance.toObject();
|
||||
const all_node_ids = nodes.map(n => n.id);
|
||||
const cforge_data = await fetch_from_backend('exportCache', {
|
||||
'ids': all_node_ids,
|
||||
}).then(function(json) {
|
||||
if (!json || !json.files)
|
||||
throw new Error('There was no response from the backend.');
|
||||
|
||||
// Now we append the cache file data to the flow
|
||||
return {
|
||||
flow: flow,
|
||||
cache: json.files,
|
||||
};
|
||||
}).catch(handleError);
|
||||
|
||||
if (!cforge_data) return;
|
||||
|
||||
// Compress the data and check it's compressed size < 5MB:
|
||||
const compressed = LZString.compressToUTF16(JSON.stringify(cforge_data));
|
||||
if (!isFileSizeLessThan5MB(compressed)) {
|
||||
handleError(new Error("Flow filesize exceeds 5MB. You can only share flows up to 5MB or less. But, don't despair! You can still use 'Export Flow' to share your flow manually as a .cforge file."))
|
||||
return;
|
||||
}
|
||||
|
||||
// Try to upload the compressed cforge data to the server:
|
||||
fetch('/db/shareflow.php', {
|
||||
method: 'POST',
|
||||
body: compressed
|
||||
})
|
||||
.then(r => r.text())
|
||||
.then(uid => {
|
||||
if (!uid) {
|
||||
throw new Error("Received no response from server.");
|
||||
} else if (uid.startsWith('Error')) {
|
||||
// Error encountered during the query; alert the user
|
||||
// with the error message:
|
||||
throw new Error(uid);
|
||||
}
|
||||
|
||||
// Share completed!
|
||||
setWaitingForShare(false);
|
||||
|
||||
// The response should be a uid we can put in a GET request.
|
||||
// Generate the link:
|
||||
let base_url = new URL(window.location.origin + window.location.pathname); // the current address e.g., https://chainforge.ai/play
|
||||
let get_params = new URLSearchParams(base_url.search);
|
||||
// Add the 'f' parameter
|
||||
get_params.set('f', uid); // set f=uid
|
||||
// Update the URL with the modified search parameters
|
||||
base_url.search = get_params.toString();
|
||||
// Get the modified URL
|
||||
const get_url = base_url.toString();
|
||||
|
||||
// Copies the GET URL to user's clipboard
|
||||
// and updates the 'Share This' button state:
|
||||
clipboard.copy(get_url);
|
||||
})
|
||||
.catch(err => {
|
||||
handleError(err);
|
||||
});
|
||||
|
||||
}, [rfInstance, nodes, IS_RUNNING_LOCALLY, handleError, clipboard, waitingForShare]);
|
||||
|
||||
// Run once upon ReactFlow initialization
|
||||
const onInit = (rf_inst) => {
|
||||
localStorage.removeItem('chainforge-flow');
|
||||
localStorage.removeItem('chainforge-state');
|
||||
|
||||
setRfInstance(rf_inst);
|
||||
|
||||
// Autosave the flow to localStorage every minute:
|
||||
@ -456,30 +565,60 @@ const App = () => {
|
||||
const interv = setInterval(() => saveFlow(rf_inst), 60000); // 60000 milliseconds = 1 minute
|
||||
setAutosavingInterval(interv);
|
||||
|
||||
if (!IS_RUNNING_LOCALLY) {
|
||||
|
||||
// Check if there's a shared flow UID in the URL as a GET param
|
||||
// If so, we need to look it up in the database and attempt to load it:
|
||||
const shared_flow_uid = getSharedFlowURLParam();
|
||||
if (shared_flow_uid !== undefined) {
|
||||
try {
|
||||
// The format passed a basic smell test;
|
||||
// now let's query the server for a flow with that UID:
|
||||
fetch('/db/get_sharedflow.php', {
|
||||
method: 'POST',
|
||||
body: shared_flow_uid,
|
||||
})
|
||||
.then(r => r.text())
|
||||
.then(response => {
|
||||
if (!response || response.startsWith('Error')) {
|
||||
// Error encountered during the query; alert the user
|
||||
// with the error message:
|
||||
handleError(new Error(response || 'Unknown error'));
|
||||
return;
|
||||
}
|
||||
|
||||
// Attempt to parse the response as a compressed flow + import it:
|
||||
try {
|
||||
const cforge_json = JSON.parse(LZString.decompressFromUTF16(response));
|
||||
importFlowFromJSON(cforge_json, rf_inst);
|
||||
setIsLoading(false);
|
||||
} catch (err) {
|
||||
handleError(err);
|
||||
}
|
||||
})
|
||||
.catch(err => {
|
||||
handleError(err);
|
||||
});
|
||||
} catch(err) {
|
||||
// Soft fail
|
||||
setIsLoading(false);
|
||||
console.error(err);
|
||||
}
|
||||
|
||||
// Since we tried to load from the shared flow ID, don't try to load from autosave
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
// Attempt to load an autosaved flow, if one exists:
|
||||
if (autosavedFlowExists())
|
||||
loadFlowFromAutosave(rf_inst);
|
||||
else {
|
||||
// Load an interesting default starting flow for new users
|
||||
importFlowFromJSON(EXAMPLEFLOW_1);
|
||||
rf_inst.setViewport(EXAMPLEFLOW_1.flow.viewport);
|
||||
importFlowFromJSON(EXAMPLEFLOW_1, rf_inst);
|
||||
|
||||
// Open a welcome pop-up
|
||||
// openWelcomeModal();
|
||||
|
||||
// NOTE: We need to create a unique ID using the current date,
|
||||
// because of the way ReactFlow saves and restores states.
|
||||
// const uid = (id) => `${id}-${Date.now()}`;
|
||||
// setNodes([
|
||||
// { id: uid('prompt'), type: 'prompt', data: {
|
||||
// llms: [ INITIAL_LLM() ],
|
||||
// prompt: 'What is the opening sentence of Pride and Prejudice?',
|
||||
// n: 1 }, position: { x: 450, y: 200 } },
|
||||
// { id: uid('eval'), type: 'evaluator', data: { language: "javascript", code: "function evaluate(response) {\n return response.text.length;\n}" }, position: { x: 820, y: 150 } },
|
||||
// { id: uid('textfields'), type: 'textfields', data: {}, position: { x: 80, y: 270 } },
|
||||
// { id: uid('vis'), type: 'vis', data: {}, position: { x: 1200, y: 250 } },
|
||||
// { id: uid('inspect'), type: 'inspect', data: {}, position: { x:820, y:400 } },
|
||||
// ]);
|
||||
}
|
||||
|
||||
// Turn off loading wheel
|
||||
@ -574,10 +713,22 @@ const App = () => {
|
||||
<Button onClick={importFlowFromFile} size="sm" variant="outline" compact>Import</Button>
|
||||
</div>
|
||||
<div style={{position: 'fixed', right: '10px', top: '10px', zIndex: 8}}>
|
||||
{IS_RUNNING_LOCALLY ? (<></>) : (
|
||||
<Button onClick={onClickShareFlow}
|
||||
size="sm" variant="outline" compact
|
||||
color={clipboard.copied ? 'teal' : 'blue'}
|
||||
mr='xs' style={{float: 'left'}}>
|
||||
{waitingForShare ? <Loader size='xs' mr='4px' /> : <IconFileSymlink size="16px"/>}
|
||||
{clipboard.copied ? 'Link copied!' : (waitingForShare ? 'Sharing...' : 'Share')}
|
||||
</Button>
|
||||
)}
|
||||
<Button onClick={onClickNewFlow} size="sm" variant="outline" compact mr='xs' style={{float: 'left'}}> New Flow </Button>
|
||||
<Button onClick={onClickExamples} size="sm" variant="filled" compact mr='xs' style={{float: 'left'}}> Example Flows </Button>
|
||||
<Button onClick={onClickSettings} size="sm" variant="gradient" compact><IconSettings size={"90%"} /></Button>
|
||||
</div>
|
||||
<div style={{position: 'fixed', right: '10px', bottom: '20px', zIndex: 8}}>
|
||||
<a href='https://forms.gle/AA82Rbn1X8zztcbj8' target="_blank" style={{color: '#666', fontSize:'11pt'}}>Send us feedback</a>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
2
chainforge/react-server/src/CommentNode.js
vendored
2
chainforge/react-server/src/CommentNode.js
vendored
@ -1,4 +1,4 @@
|
||||
import React, { useState, useEffect, useRef, useCallback } from 'react';
|
||||
import React, { useState } from 'react';
|
||||
import useStore from './store';
|
||||
import NodeLabel from './NodeLabelComponent'
|
||||
import { Textarea } from '@mantine/core';
|
||||
|
4
chainforge/react-server/src/EditableTable.js
vendored
4
chainforge/react-server/src/EditableTable.js
vendored
@ -1,5 +1,5 @@
|
||||
import React, { useState, useRef, useEffect, useCallback, forwardRef } from 'react';
|
||||
import { Table, Textarea, Menu, Button, TextInput } from '@mantine/core';
|
||||
import React, { useState, useEffect } from 'react';
|
||||
import { Table, Textarea, Menu } from '@mantine/core';
|
||||
import { IconDots, IconPencil, IconArrowLeft, IconArrowRight, IconX } from '@tabler/icons-react';
|
||||
|
||||
const cellTextareaStyle = {
|
||||
|
@ -1,8 +1,7 @@
|
||||
import React, { useState, useEffect, useRef, useCallback, forwardRef, useImperativeHandle } from 'react';
|
||||
import { SimpleGrid, Card, Modal, Image, Group, Text, Button, Badge, Tabs, Alert, Code } from '@mantine/core';
|
||||
import React, { forwardRef, useImperativeHandle } from 'react';
|
||||
import { SimpleGrid, Card, Modal, Text, Button, Tabs } from '@mantine/core';
|
||||
import { useDisclosure } from '@mantine/hooks';
|
||||
import { IconChartDots3, IconAlertCircle } from '@tabler/icons-react';
|
||||
import { BASE_URL } from './store';
|
||||
import { IconChartDots3 } from '@tabler/icons-react';
|
||||
|
||||
/** The preconverted OpenAI evals we can load from,
|
||||
* along with their descriptions, extracted from the evals registry. */
|
||||
|
@ -1,5 +1,5 @@
|
||||
import React, { useState, forwardRef, useImperativeHandle } from 'react';
|
||||
import { TextInput, Checkbox, Button, Group, Box, Modal, Divider, Text } from '@mantine/core';
|
||||
import { TextInput, Button, Group, Box, Modal, Divider, Text } from '@mantine/core';
|
||||
import { useDisclosure } from '@mantine/hooks';
|
||||
import { useForm } from '@mantine/form';
|
||||
import useStore from './store';
|
||||
|
2
chainforge/react-server/src/InspectorNode.js
vendored
2
chainforge/react-server/src/InspectorNode.js
vendored
@ -1,4 +1,4 @@
|
||||
import React, { useState, useEffect, useRef, useCallback } from 'react';
|
||||
import React, { useState, useEffect } from 'react';
|
||||
import { Handle } from 'react-flow-renderer';
|
||||
import useStore from './store';
|
||||
import NodeLabel from './NodeLabelComponent'
|
||||
|
@ -1,6 +1,6 @@
|
||||
import { useState, useEffect, useCallback, useRef } from "react";
|
||||
import { DragDropContext, Droppable, Draggable } from "react-beautiful-dnd";
|
||||
import LLMListItem, { DragItem, LLMListItemClone } from "./LLMListItem";
|
||||
import { DragDropContext, Draggable } from "react-beautiful-dnd";
|
||||
import LLMListItem, { LLMListItemClone } from "./LLMListItem";
|
||||
import { StrictModeDroppable } from './StrictModeDroppable'
|
||||
import ModelSettingsModal from "./ModelSettingsModal"
|
||||
|
||||
|
@ -5,7 +5,7 @@
|
||||
* be deployed in multiple locations.
|
||||
*/
|
||||
import React, { useState, useEffect, useRef } from 'react';
|
||||
import { Collapse, MultiSelect, ScrollArea } from '@mantine/core';
|
||||
import { Collapse, MultiSelect } from '@mantine/core';
|
||||
import { useDisclosure } from '@mantine/hooks';
|
||||
import * as XLSX from 'xlsx';
|
||||
import useStore from './store';
|
||||
|
@ -1,5 +1,5 @@
|
||||
import React, { useState, useRef, useCallback, forwardRef, useImperativeHandle, useEffect } from 'react';
|
||||
import { TextInput, Checkbox, Button, Group, Box, Modal, Popover } from '@mantine/core';
|
||||
import React, { useState, useCallback, forwardRef, useImperativeHandle, useEffect } from 'react';
|
||||
import { Button, Modal, Popover } from '@mantine/core';
|
||||
import { useDisclosure } from '@mantine/hooks';
|
||||
|
||||
import emojidata from '@emoji-mart/data';
|
||||
|
2
chainforge/react-server/src/ScriptNode.js
vendored
2
chainforge/react-server/src/ScriptNode.js
vendored
@ -1,4 +1,4 @@
|
||||
import React, { useState, useRef, useEffect, useCallback } from 'react';
|
||||
import React, { useState, useEffect, useCallback } from 'react';
|
||||
import useStore from './store';
|
||||
import NodeLabel from './NodeLabelComponent';
|
||||
import { IconSettingsAutomation } from '@tabler/icons-react';
|
||||
|
@ -1,11 +1,10 @@
|
||||
import React, { useState, useRef, useEffect, useCallback, forwardRef, useId } from 'react';
|
||||
import { Handle } from 'react-flow-renderer';
|
||||
import { Menu, Modal, TextInput, Button, Tooltip, Textarea, Table } from '@mantine/core';
|
||||
import React, { useState, useRef, useEffect, useCallback } from 'react';
|
||||
import { Menu, Tooltip } from '@mantine/core';
|
||||
import EditableTable from './EditableTable';
|
||||
import * as XLSX from 'xlsx';
|
||||
import Papa from 'papaparse';
|
||||
import { v4 as uuidv4 } from 'uuid';
|
||||
import { IconPencil, IconArrowLeft, IconArrowRight, IconX, IconArrowBarToUp, IconArrowBarToDown } from '@tabler/icons-react';
|
||||
import { IconX, IconArrowBarToUp, IconArrowBarToDown } from '@tabler/icons-react';
|
||||
import TemplateHooks from './TemplateHooksComponent';
|
||||
import NodeLabel from './NodeLabelComponent';
|
||||
import AlertModal from './AlertModal';
|
||||
|
2
chainforge/react-server/src/VisNode.js
vendored
2
chainforge/react-server/src/VisNode.js
vendored
@ -1,6 +1,6 @@
|
||||
import React, { useState, useEffect, useCallback, useRef } from 'react';
|
||||
import { Handle } from 'react-flow-renderer';
|
||||
import { MultiSelect, NativeSelect } from '@mantine/core';
|
||||
import { NativeSelect } from '@mantine/core';
|
||||
import useStore, { colorPalettes } from './store';
|
||||
import Plot from 'react-plotly.js';
|
||||
import NodeLabel from './NodeLabelComponent';
|
||||
|
@ -247,7 +247,6 @@ export async function call_azure_openai(prompt: string, model: LLM, n: number =
|
||||
if (error?.response) {
|
||||
throw new Error(error.response.data?.error?.message);
|
||||
} else {
|
||||
console.log(error?.message || error);
|
||||
throw new Error(error?.message || error);
|
||||
}
|
||||
}
|
||||
@ -299,19 +298,43 @@ export async function call_anthropic(prompt: string, model: LLM, n: number = 1,
|
||||
console.log(`Calling Anthropic model '${model}' with prompt '${prompt}' (n=${n}). Please be patient...`);
|
||||
|
||||
// Make a REST call to Anthropic
|
||||
const url = 'https://api.anthropic.com/v1/complete';
|
||||
const headers = {
|
||||
'accept': 'application/json',
|
||||
'anthropic-version': '2023-06-01',
|
||||
'content-type': 'application/json',
|
||||
'x-api-key': ANTHROPIC_API_KEY,
|
||||
};
|
||||
|
||||
// Repeat call n times, waiting for each response to come in:
|
||||
let responses: Array<Dict> = [];
|
||||
while (responses.length < n) {
|
||||
const resp = await route_fetch(url, 'POST', headers, query);
|
||||
responses.push(resp);
|
||||
|
||||
if (APP_IS_RUNNING_LOCALLY()) {
|
||||
// If we're running locally, route the request through the Flask backend,
|
||||
// where we can use the Anthropic Python API to make the API call:
|
||||
const url = 'https://api.anthropic.com/v1/complete';
|
||||
const headers = {
|
||||
'Accept': 'application/json',
|
||||
"anthropic-version": "2023-06-01",
|
||||
'Content-Type': 'application/json',
|
||||
'User-Agent': "Anthropic/JS 0.5.0",
|
||||
'X-Api-Key': ANTHROPIC_API_KEY,
|
||||
};
|
||||
const resp = await route_fetch(url, 'POST', headers, query);
|
||||
responses.push(resp);
|
||||
|
||||
} else {
|
||||
// We're on the server; route API call through a proxy on the server, since Anthropic has CORS policy on their API:
|
||||
const resp = await fetch('/db/call_anthropic.php', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'X-Api-Key': ANTHROPIC_API_KEY,
|
||||
},
|
||||
body: JSON.stringify(query)
|
||||
}).then(r => r.json());
|
||||
|
||||
// Check for error from server
|
||||
if (resp?.error !== undefined) {
|
||||
throw new Error(`${resp.error.type}: ${resp.error.message}`);
|
||||
}
|
||||
|
||||
// console.log('Received Anthropic response from server proxy:', resp);
|
||||
responses.push(resp);
|
||||
}
|
||||
}
|
||||
|
||||
return [query, responses];
|
||||
@ -498,17 +521,11 @@ export async function call_huggingface(prompt: string, model: LLM, n: number = 1
|
||||
// Merge responses
|
||||
const resp_text: string = result[0].generated_text;
|
||||
|
||||
console.log(curr_cont, 'curr_text', curr_text);
|
||||
console.log(curr_cont, 'resp_text', resp_text);
|
||||
|
||||
continued_response.generated_text += resp_text;
|
||||
curr_text += resp_text;
|
||||
|
||||
curr_cont += 1;
|
||||
}
|
||||
|
||||
console.log(continued_response);
|
||||
|
||||
// Continue querying
|
||||
responses.push(continued_response);
|
||||
}
|
||||
|
2
chainforge/react-server/src/example_flows.js
vendored
2
chainforge/react-server/src/example_flows.js
vendored
@ -313,7 +313,7 @@ export const EXAMPLEFLOW_1 = {
|
||||
}
|
||||
],
|
||||
"viewport": {
|
||||
"x": 450,
|
||||
"x": 474,
|
||||
"y": 63,
|
||||
"zoom": 1
|
||||
}
|
||||
|
@ -1,4 +1,4 @@
|
||||
import { queryLLM, executejs, executepy, FLASK_BASE_URL,
|
||||
import { queryLLM, executejs, executepy,
|
||||
fetchExampleFlow, fetchOpenAIEval, importCache,
|
||||
exportCache, countQueries, grabResponses,
|
||||
createProgressFile } from "./backend/backend";
|
||||
|
1
chainforge/react-server/src/store.js
vendored
1
chainforge/react-server/src/store.js
vendored
@ -3,7 +3,6 @@ import {
|
||||
addEdge,
|
||||
applyNodeChanges,
|
||||
applyEdgeChanges,
|
||||
useViewport,
|
||||
} from 'react-flow-renderer';
|
||||
|
||||
// Initial project settings
|
||||
|
Loading…
x
Reference in New Issue
Block a user