ChainForge/chainforge/oaievals/russian-english-homonym-context-resolution.cforge

{"flow": {"nodes": [{"width": 312, "height": 311, "id": "prompt-russian-english-homonym-context-resolution", "type": "prompt", "data": {"prompt": "{prompt}", "n": 1, "llms": [{"key": "aa3c0f03-22bd-416e-af4d-4bf5c4278c99", "settings": {"system_msg": "For the given context, resolve the ambiguity and determine the most appropriate response. If there is one, output just one word; otherwise, output unknown. The responses must be lowercase with no punctuation.", "temperature": 1, "functions": [], "function_call": "", "top_p": 1, "stop": [], "presence_penalty": 0, "frequency_penalty": 0}, "name": "GPT3.5", "emoji": "\ud83d\ude42", "model": "gpt-3.5-turbo", "base_model": "gpt-3.5-turbo", "temp": 1, "formData": {"shortname": "GPT3.5", "model": "gpt-3.5-turbo", "system_msg": "For the given context, resolve the ambiguity and determine the most appropriate response. If there is one, output just one word; otherwise, output unknown. The responses must be lowercase with no punctuation.", "temperature": 1, "functions": "", "function_call": "", "top_p": 1, "stop": "", "presence_penalty": 0, "frequency_penalty": 0}}]}, "position": {"x": 448, "y": 224}, "selected": false, "positionAbsolute": {"x": 448, "y": 224}, "dragging": false}, {"width": 333, "height": 182, "id": "eval-russian-english-homonym-context-resolution", "type": "evaluator", "data": {"code": "def evaluate(response):\n\tideal = response.meta['Ideal']\n\treturn response.text.startswith(ideal)"}, "position": {"x": 820, "y": 150}, "positionAbsolute": {"x": 820, "y": 150}}, {"width": 228, "height": 196, "id": "vis-russian-english-homonym-context-resolution", "type": "vis", "data": {"input": "eval-russian-english-homonym-context-resolution"}, "position": {"x": 1200, "y": 250}, "positionAbsolute": {"x": 1200, "y": 250}}, {"width": 302, "height": 260, "id": "inspect-russian-english-homonym-context-resolution", "type": "inspect", "data": {"input": "prompt-russian-english-homonym-context-resolution"}, "position": {"x": 820, "y": 400}, "positionAbsolute": {"x": 820, "y": 400}}, {"width": 423, "height": 417, "id": "table-russian-english-homonym-context-resolution", "type": "table", "data": {"rows": [{"prompt": "You see a sign with the words: \"\u0432\u043e\u0437\u044c\u043c\u0438 \u0437\u0435\u043b\u0451\u043d\u044b\u0439 \u043b\u0443\u043a.\" It's early spring, and the ground is still frozen. You see onions and a broken bow near it. What do you need to pick up?", "ideal": "bow"}, {"prompt": "You see a sign with the words: \"\u0432\u043e\u0437\u044c\u043c\u0438 \u0437\u0435\u043b\u0451\u043d\u044b\u0439 \u043b\u0443\u043a\". It is summer. You see onions and a broken bow near it. What would you need to pick up?", "ideal": "onions"}, {"prompt": "You are coming to work, the factory floor is loud and busy. You see a sign with the words: \"\u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0435 \u0431\u0440\u0430\u043a\u0430 \u043d\u0435\u0434\u043e\u043e\u0446\u0435\u043d\u0435\u043d\u043e\". Is it referring to defects or marriage?", "ideal": "defects"}, {"prompt": "You are coming to work, the factory floor is loud and busy. One of the workers looks at the picture of their family and says: \"\u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0435 \u0431\u0440\u0430\u043a\u0430 \u043d\u0435\u0434\u043e\u043e\u0446\u0435\u043d\u0435\u043d\u043e\". Is it referring to defects or marriage?", "ideal": "marriage"}, {"prompt": "You are coming to work, the community center facility focused around family activities. You see a sign with the words: \"\u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0435 \u0431\u0440\u0430\u043a\u0430 \u043d\u0435\u0434\u043e\u043e\u0446\u0435\u043d\u0435\u043d\u043e\". Is it referring to defects or marriage?", "ideal": "marriage"}, {"prompt": "You are traveling in the wilderness and see the sign \"\u0433\u043e\u0440\u044f\u0447\u0438\u0439 \u043a\u043b\u044e\u0447\" pointing down the hill. Is it referring to a key or a stream?", "ideal": "stream"}, {"prompt": "You are at the steel melting factory. One of the workers says: \"\u0433\u043e\u0440\u044f\u0447\u0438\u0439 \u043a\u043b\u044e\u0447!\" pointing on the object on the floor. Is it referring to a key or a stream?", "ideal": "key"}, {"prompt": "You are traveling in the wilderness and see the sign \"\u043a\u043b\u044e\u0447 \u043e\u0442 \u0434\u043e\u043c\u0430 \u0443 \u0441\u0442\u043e\u0440\u043e\u0436\u0430\"  pointing down the hill. Is it referring key or stream?", "ideal": "key"}, {"prompt": "You are at the construction site. There is a massive steel machine getting assembled in the yard. You see a sign saying \"\u043f\u0440\u043e\u0445\u043e\u0434 \u043a \u043a\u0440\u0430\u043d\u0443 \u0437\u0430\u043a\u0440\u044b\u0442\". Is it referring crane or tap?", "ideal": "crane"}, {"prompt": "You are at the construction site. There is a massive steel machine getting assembled in the yard. You enter a building looking for a bathroom. You find one, but a sign saying \"\u043f\u0440\u043e\u0445\u043e\u0434 \u043a \u043a\u0440\u0430\u043d\u0443 \u0437\u0430\u043a\u0440\u044b\u0442\". Is it referring crane or tap?", "ideal": "tap"}, {"prompt": "You are at the limber mill, you see a sign near heavy equipment \"\u043f\u0438\u043b\u0430 \u0414\u0436\u0435\u043d\u043d\u0438\". Is it referring drinking or saw?", "ideal": "saw"}, {"prompt": "You are at the limber mill, you see a paper near heavy equipment listing workers penalties for breaking safety requirements. \"\u0414\u0436\u0435\u043d\u043d\u0438 \u043f\u0438\u043b\u0430\". Is it referring drinking or saw?", "ideal": "drinking"}, {"prompt": "A child coming from school and saying: \"\u042f \u043f\u043e\u043b\u0443\u0447\u0438\u043b \u043a\u043e\u043b\". Is it referring grade or pole?", "ideal": "grade"}, {"prompt": "A construction worker is building a palisade. He said: \"\u042f \u043f\u043e\u043b\u0443\u0447\u0438\u043b \u043a\u043e\u043b\". Is it referring grade or pole?", "ideal": "pole"}, {"prompt": "You look at the table. It's was covered with a clean sheet of paper and there is a cup on it. Someone says: \"\u043a\u0440\u0430\u0441\u0438\u0432\u0430\u044f \u043a\u0440\u0443\u0436\u043a\u0430\". Is it referring to a circle or a cup?", "ideal": "cup"}, {"prompt": "You look at the table. It's was covered with a clean sheet of paper and there is a cup on it. Someone says: \"\u043a\u0440\u0443\u0436\u043a\u0430 \u043d\u0435 \u0432\u0438\u0434\u043d\u043e\". Is it referring to a circle or a cup?", "ideal": "circle"}, {"prompt": "You look at the table. It's was covered with a clean sheet of paper and there is a cup on it. Someone says: \"\u043a\u0440\u0443\u0436\u043a\u0430 \u043e\u0442 \u043a\u0440\u0443\u0436\u043a\u0438 \u043d\u0435 \u0432\u0438\u0434\u043d\u043e\". Is it referring to a circle or a cup?", "ideal": "circle"}, {"prompt": "You are staying at the old castle to enjoy some wine. Someone takes a drink and says: \"\u043a\u0440\u0435\u043f\u043e\u0441\u0442\u044c \u0438\u0434\u0435\u0430\u043b\u044c\u043d\u0430\". Do they refer strength or castle?", "ideal": "strength"}, {"prompt": "You are staying at the old castle to enjoy some wine. A person took a photo during the drive and said \"\u043a\u0440\u0435\u043f\u043e\u0441\u0442\u044c \u0438\u0434\u0435\u0430\u043b\u044c\u043d\u0430\" after looking at the photo. Do they refer strength or castle?", "ideal": "strength"}, {"prompt": "You walking on a castle wall. You hear \"meow\" and quickly say: \"\u043a\u043e\u0448\u043a\u0430!\". Do they refer hook or cat?", "ideal": "cat"}, {"prompt": "You walking on a castle wall. You hear metal clink and quickly say: \"\u043a\u043e\u0448\u043a\u0430!\". Do they refer hook or cat?", "ideal": "hook"}, {"prompt": "You are coming to a community facility to drop a child for hockey practice. The road is under construction and big machines are paving it. What would \"\u043a\u0430\u0442\u043e\u043a\" mean in this context? rink or roller", "ideal": "unknown"}, {"prompt": "You are coming to a community facility to drop a child for hockey practice. The road is under construction and big machines are paving it. Child pointing at the machine and says: \"\u043a\u0430\u0442\u043e\u043a\". What does he mean: rink or roller?", "ideal": "roller"}, {"prompt": "You are coming to a community facility to drop a child for hockey practice. The road is under construction and big machines are paving it. Child pointing at the building and says: \"\u043a\u0430\u0442\u043e\u043a\". What does he mean: rink or roller?", "ideal": "rink"}, {"prompt": "A woman with long braided hair is working in the field. She cuts the grass with scythe. Someone says \"\u0445\u043e\u0440\u043e\u0448\u0430\u044f \u043a\u043e\u0441\u0430\". Do they refer scythe or hairstyle?", "ideal": "unknown"}, {"prompt": "A woman with long braided hair is working in the field. She cuts the grass with scythe. Someone points at the quality of her work and says \"\u0445\u043e\u0440\u043e\u0448\u0430\u044f \u043a\u043e\u0441\u0430\". Do they refer scythe or hairstyle?", "ideal": "scythe"}, {"prompt": "A woman with long braided hair is working in the field. She cuts the grass with scythe. Someone points at her head and says \"\u0445\u043e\u0440\u043e\u0448\u0430\u044f \u043a\u043e\u0441\u0430\". Do they refer scythe or hairstyle?", "ideal": "hairstyle"}, {"prompt": "Scythe is found on a sandbar. A person is saying: \"\u043a\u043e\u0441\u0430\". Do they refer scythe or sandbar?", "ideal": "unknown"}, {"prompt": "Scythe is found on a sandbar. A person is saying: \"\u0440\u0436\u0430\u0432\u0430\u044f \u043a\u043e\u0441\u0430\". Do they refer scythe or sandbar?", "ideal": "scythe"}, {"prompt": "Scythe is found on a sandbar. Scythe is a pin for a backpack. A person is saying: \"\u0434\u043b\u0438\u043d\u043d\u0430\u044f \u043a\u043e\u0441\u0430\". Do they refer scythe or sandbar?", "ideal": "sandbar"}, {"prompt": "Scythe is found on a sandbar. Scythe is a word on a piece of paper. A person is saying: \"\u0434\u043b\u0438\u043d\u043d\u0430\u044f \u043a\u043e\u0441\u0430\". Do they refer scythe or sandbar?", "ideal": "sandbar"}, {"prompt": "A witch applies perfume in order to communicate with ghosts. When she says, \"\u043c\u043d\u0435 \u043d\u0443\u0436\u043d\u044b \u0434\u0443\u0445\u0438,\" does she refer to ghosts or perfume?", "ideal": "unknown"}, {"prompt": "A witch applies perfume in order to communicate with ghosts. She activates crystal ball and chants:, \"\u043c\u043d\u0435 \u043d\u0443\u0436\u043d\u044b \u0434\u0443\u0445\u0438,\" does she refer to ghosts or perfume?", "ideal": "ghosts"}, {"prompt": "A witch applies perfume in order to communicate with ghosts. She looks at the empty flask and says \"\u043c\u043d\u0435 \u043d\u0443\u0436\u043d\u044b \u0434\u0443\u0445\u0438,\" does she refer to ghosts or perfume?", "ideal": "perfume"}, {"prompt": "A patron at the bar loads a bullet into a gun and says, \"\u043f\u043e\u0441\u043b\u0435\u0434\u043d\u0438\u0439 \u043f\u0430\u0442\u0440\u043e\u043d\" Does he refer to the patron or the bullet?", "ideal": "bullet"}, {"prompt": "A patron at the bar loads a bullet into a gun. An alien watching him says, \"\u043f\u043e\u0441\u043b\u0435\u0434\u043d\u0438\u0439 \u043f\u0430\u0442\u0440\u043e\u043d\" Does the alien refer to the patron or the bullet?", "ideal": "unknown"}, {"prompt": "A patron at the bar loads a bullet into a gun from a fully loaded belt. An alien watching him says, \"\u043f\u043e\u0441\u043b\u0435\u0434\u043d\u0438\u0439 \u043f\u0430\u0442\u0440\u043e\u043d\" Does the alien refer to the patron or the bullet?", "ideal": "patron"}, {"prompt": "At the end of World War, a soldier says, \"\u043c\u0438\u0440 \u043d\u0430\u0448\" Does he refer to the world or peace?", "ideal": "unknown"}, {"prompt": "There is a Galactic War. At the end of the war on one of the planets, a soldier says, \"\u043c\u0438\u0440 \u043d\u0430\u0448.\" Does he refer to the world or peace?", "ideal": "world"}, {"prompt": "At the end of World War, the whole world is conquered and a soldier says, \"\u043c\u0438\u0440 \u043d\u0430\u0448\" Does he refer to the world or peace?", "ideal": "world"}, {"prompt": "At the end of World War, the whole world is conquered and a soldier says, \"\u043d\u0430\u043a\u043e\u043d\u0435\u0446-\u0442\u043e \u043c\u0438\u0440\" Does he refer to the world or peace?", "ideal": "peace"}, {"prompt": "A person was tasked to rub the flask three times. He asked a question, and the reply was \"\u0442\u0440\u0438\" Did he ask about the action or the quantity?", "ideal": "unknown"}, {"prompt": "A person was tasked to rub the flask three times. He asked: \"\u0447\u0442\u043e \u043c\u043d\u0435 \u0434\u0435\u043b\u0430\u0442\u044c\", and the reply was \"\u0442\u0440\u0438\" Did he ask about the action or the quantity?", "ideal": "action"}, {"prompt": "A person was tasked to rub the flask three times. He asked: \"\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0440\u0430\u0437\", and the reply was \"\u0442\u0440\u0438\" Did he ask about the action or the quantity?", "ideal": "quantity"}, {"prompt": "In this part of the world, light is generated via bioluminescence. The ad says, \"\u0438\u0437\u0432\u0435\u0441\u0442\u043d\u044b\u0439 \u043d\u0430 \u0432\u0435\u0441\u044c \u0441\u0432\u0435\u0442\" Do they refer to light or the world?", "ideal": "world"}, {"prompt": "In this part of the world, light is generated via bioluminescence. The ad says, \"\u0432\u0435\u0441\u044c \u0441\u0432\u0435\u0442 \u0441\u043c\u043e\u0442\u0440\u0438\u0442 \u043d\u0430 \u043d\u0430\u0441\". Do they refer to light or the world?", "ideal": "world"}, {"prompt": "In this part of the world, light is generated via bioluminescence. The ad says, \"\u0441\u0438\u043d\u0438\u0439 \u0441\u0432\u0435\u0442\" . Do they refer to light or the world?", "ideal": "light"}, {"prompt": "A person is planning to prepare a test in a liquid environment for the next Wednesday. They take a note: \"\u0441\u0440\u0435\u0434\u0430: \u0442\u0435\u0441\u0442\". Do they refer to the weekday or the environment?", "ideal": "unknown"}, {"prompt": "A person is planning to prepare a test in a liquid environment for the next Wednesday. They take a note: \"\u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0430\u044f \u0441\u0440\u0435\u0434\u0430: \u0442\u0435\u0441\u0442\". Do they refer to the weekday or the environment?", "ideal": "weekday"}, {"prompt": "A person is planning to prepare a test in a liquid environment for the next Wednesday. They take a note: \"\u0432\u043e\u0437\u0434\u0443\u0448\u043d\u0430\u044f \u0441\u0440\u0435\u0434\u0430: \u0442\u0435\u0441\u0442\". Do they refer to the weekday or the environment?", "ideal": "environment"}], "columns": [{"key": "prompt", "header": "Prompt"}, {"key": "ideal", "header": "Ideal"}]}, "position": {"x": -16, "y": 160}, "selected": false, "positionAbsolute": {"x": -16, "y": 160}, "dragging": false}], "edges": [{"source": "prompt-russian-english-homonym-context-resolution", "sourceHandle": "prompt", "target": "eval-russian-english-homonym-context-resolution", "targetHandle": "responseBatch", "interactionWidth": 100, "markerEnd": {"type": "arrow", "width": "22px", "height": "22px"}, "id": "reactflow__edge-prompt-1686756357355prompt-eval-1686756357355responseBatch"}, {"source": "prompt-russian-english-homonym-context-resolution", "sourceHandle": "prompt", "target": "inspect-russian-english-homonym-context-resolution", "targetHandle": "input", "interactionWidth": 100, "markerEnd": {"type": "arrow", "width": "22px", "height": "22px"}, "id": "reactflow__edge-prompt-1686756357355prompt-inspect-1686756357355input"}, {"source": "eval-russian-english-homonym-context-resolution", "sourceHandle": "output", "target": "vis-russian-english-homonym-context-resolution", "targetHandle": "input", "interactionWidth": 100, "markerEnd": {"type": "arrow", "width": "22px", "height": "22px"}, "id": "reactflow__edge-eval-1686756357355output-vis-1686756357355input"}, {"source": "table-russian-english-homonym-context-resolution", "sourceHandle": "Prompt", "target": "prompt-russian-english-homonym-context-resolution", "targetHandle": "prompt", "interactionWidth": 100, "markerEnd": {"type": "arrow", "width": "22px", "height": "22px"}, "id": "reactflow__edge-table-1686756385002Prompt-prompt-1686756357355prompt"}], "viewport": {"x": 144, "y": 37, "zoom": 1}}, "cache": {"eval-1686756357355.json": {}, "inspect-1686756357355.json": {}, "prompt-1686756357355.json": {}, "table-1686756385002.json": {}, "vis-1686756357355.json": {}}}