{"id":154206,"date":"2023-11-07T17:03:12","date_gmt":"2023-11-07T08:03:12","guid":{"rendered":"https:\/\/support.questetra.com\/?p=154206"},"modified":"2023-11-14T16:33:00","modified_gmt":"2023-11-14T07:33:00","slug":"openai-chat-answer-prompt-with-images-202311","status":"publish","type":"post","link":"https:\/\/support.questetra.com\/en\/addons\/openai-chat-answer-prompt-with-images-202311\/","title":{"rendered":"OpenAI #Chat: Answer Prompt with Images"},"content":{"rendered":"\n

OpenAI #Chat: Answer Prompt with Images<\/strong><\/h4>\n\n\n\n

translate<\/span> OpenAI #Chat: \u753b\u50cf\u4ed8\u304d\u30d7\u30ed\u30f3\u30d7\u30c8\u306b\u56de\u7b54<\/a><\/p>\n\n\n\n

Creates a response for a prompt with images. MODEL (GPT-4 with Vision) takes in images and answers questions related to them, such as what the image represents, what is in the image, and more (eg. ideas for dinner based on what is in fridge).<\/p>\n\n\n\n

\"Auto<\/figure>\n\n\n\n
\n
Configs for this Auto Step<\/summary>
\n
\n
AuthzConfU1<\/dt>
U1: Select HTTP_Authz Setting (Secret API Key as “Fixed Value”) *<\/span><\/dd>\n
StrConfA1<\/dt>
A1: Set Text PROMPT *<\/span>#{EL}<\/sup><\/span><\/dd>\n
StrConfA2<\/dt>
A2: Set Image URLs on each line *<\/span>#{EL}<\/sup><\/span><\/dd>\n
SelectConfC1<\/dt>
C1: Select STRING that stores Generated Text (update)<\/dd>\n<\/div>\n
StrConfM<\/dt>
M: Set MODEL Name (default “gpt-4-vision-preview”)#{EL}<\/sup><\/span><\/dd>\n
StrConfU2<\/dt>
U2: Set OpenAI Organization ID (“org-xxxx”)#{EL}<\/sup><\/span><\/dd>\n
StrConfB1<\/dt>
B1: Set DETAIL parameter (“high” or “low”:default)#{EL}<\/sup><\/span><\/dd>\n
StrConfB2<\/dt>
B2: Set MaxTokens#{EL}<\/sup><\/span><\/dd>\n
SelectConfC2<\/dt>
C2: Select NUMERIC that stores PROMPT Tokens (update)<\/dd>\n
SelectConfC3<\/dt>
C3: Select NUMERIC that stores Total Tokens (update)<\/dd>\n<\/div>\n\n<\/dl>\n<\/div><\/details><\/div>\n<\/div>\n\n\n\n
\n
Script (click to open)<\/summary>
\n
\/\/ Script Example of Business Process Automation\n\/\/ for 'engine type: 3' ("GraalJS standard mode")\n\/\/ cf. 'engine type: 2' ("GraalJS Nashorn compatible mode") (renamed from "GraalJS" at 20230526)\n\n\/\/\/\/\/\/\/\/ START "main()" \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\n\nmain();\nfunction main(){ \n\n\/\/\/\/\/\/ == Config Retrieving \/ \u5de5\u7a0b\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u53c2\u7167 ==\nconst strAuthzSetting  = configs.get      ( "AuthzConfU1" );                       \/\/\/ REQUIRED\n  engine.log( " AutomatedTask Config: Authz Setting: " + strAuthzSetting );\nconst strOrgId         = configs.get      ( "StrConfU2" );                         \/\/ NotRequired\n  engine.log( " AutomatedTask Config: OpenAI-Organization: " + strOrgId );\nconst strModel         = configs.get      ( "StrConfM" ) !== "" ?                  \/\/ NotRequired\n                         configs.get      ( "StrConfM" ) : "gpt-4-vision-preview"; \/\/ (default)\n  engine.log( " AutomatedTask Config: OpenAI Model: " + strModel );\n\nconst strTextPrompt    = configs.get      ( "StrConfA1" );                         \/\/\/ REQUIRED\n  if( strTextPrompt  === "" ){\n    throw new Error( "\\n AutomatedTask ConfigError:" +\n                     " Config {A1: Prompt} must be non-empty \\n" );\n  }\nconst strImageUrls     = configs.get      ( "StrConfA2" );                         \/\/\/ REQUIRED\n  if( strImageUrls   === "" ){\n    throw new Error( "\\n AutomatedTask ConfigError:" +\n                     " Config {A2: ImageUrls} must be non-empty \\n" );\n  }\nconst arrImageUrls     = strImageUrls.split("\\n");\n\nconst strDetail        = configs.get      ( "StrConfB1" ) !== "" ?                 \/\/ NotRequired\n                         configs.get      ( "StrConfB1" ) : "low";                 \/\/ (default)\nconst strMaxTokens     = configs.get      ( "StrConfB2" );                         \/\/ NotRequired\nconst numMaxTokens     = parseInt ( strMaxTokens, 10 );\n  engine.log( " AutomatedTask Config: Max Tokens: " + numMaxTokens );\n\nconst strPocketGenerated = configs.getObject ( "SelectConfC1" );                   \/\/\/ REQUIRED NotRequired\nconst numPocketPrompt    = configs.getObject ( "SelectConfC2" );                   \/\/ NotRequired\nconst numPocketTotal     = configs.getObject ( "SelectConfC3" );                   \/\/ NotRequired\n\n\n\n\/\/\/\/\/\/ == Data Retrieving \/ \u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u30c7\u30fc\u30bf\u306e\u53c2\u7167 ==\n\/\/ (Nothing. Retrieved via Expression Language in Config Retrieving)\n\n\n\/\/\/\/\/\/ == Calculating \/ \u6f14\u7b97 ==\n\n\/\/\/\/ OpenAI API > Documentation > API REFERENCE > CHAT\n\/\/\/\/ https:\/\/platform.openai.com\/docs\/api-reference\/chat\/create (not updated)\n\/\/\/\/ https:\/\/platform.openai.com\/docs\/guides\/vision\n\n\/\/\/ prepare json\nlet strJson = {};\n    strJson.model = strModel;\n    if ( ! isNaN(numMaxTokens) ) {\n      strJson.max_tokens       = numMaxTokens;\n    }\n\/\/    strJson.response_format = {};\n\/\/    strJson.response_format.type = "json_object";\n\n    strJson.messages = [];\n    strJson.messages[0] = {};\n    strJson.messages[0].role = "user";\n    strJson.messages[0].content = [];\n    strJson.messages[0].content[0] = {};\n    strJson.messages[0].content[0].type = "text";\n    strJson.messages[0].content[0].text = strTextPrompt;\n\n    for ( let i = 0; i < arrImageUrls.length; i++ ) {\n      const objTmp = {};\n      objTmp.type = "image_url";\n      objTmp.image_url = arrImageUrls[i];\n      strJson.messages[0].content.push ( objTmp );\n    }\n\n\/\/\/ prepare request1\nlet request1Uri = "https:\/\/api.openai.com\/v1\/chat\/completions";\nlet request1 = httpClient.begin(); \/\/ HttpRequestWrapper\n    request1 = request1.queryParam( "detail", strDetail );\n    request1 = request1.authSetting( strAuthzSetting ); \/\/ with "Authorization: Bearer XX"\n    request1 = request1.body( JSON.stringify( strJson ), "application\/json" );\n    if ( strOrgId !== "" ){\n      request1 = request1.header( "OpenAI-Organization", strOrgId );\n    }\n\n\/\/\/ try request1\nconst response1     = request1.post( request1Uri ); \/\/ HttpResponseWrapper\nengine.log( " AutomatedTask ApiRequest1 Start: " + request1Uri );\nconst response1Code = response1.getStatusCode() + ""; \/\/ JavaNum to string\nconst response1Body = response1.getResponseAsString();\nengine.log( " AutomatedTask ApiResponse1 Status: " + response1Code );\nif( response1Code !== "200"){\n  throw new Error( "\\n AutomatedTask UnexpectedResponseError: " +\n                    response1Code + "\\n" + response1Body + "\\n" );\n}\n\n\n\/\/\/ parse response1\n\/* engine.log( response1Body ); \/\/ debug\n{\n  "id": "chatcmpl-8I9gORGEvsLaVE10Dir0pOMZI5I37",\n  "object": "chat.completion",\n  "created": 1699337816,\n  "model": "gpt-4-1106-vision-preview",\n  "usage": {\n    "prompt_tokens": 1887, \n    "completion_tokens": 16,\n    "total_tokens": 1903\n  },\n  "choices": [{\n    "message": {\n      "role": "assistant", \n      "content": "\\u6700\\u521d\\u306e\\u753b\\u50cf\\u306b\\u306f\\u3001\\"Fine tuning workflow\\" \\u3068\\u3044\\u3046"\n    },\n    "finish_details": {\n      "type": "max_tokens"\n    },\n    "index": 0\n    }]\n}\n*\/\nconst response1Obj = JSON.parse( response1Body );\nengine.log( " AutomatedTask ApiResponse1 finish_details: " + response1Obj.choices[0].finish_details.type );\n\n\n\/\/\/\/\/\/ == Data Updating \/ \u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u30c7\u30fc\u30bf\u3078\u306e\u4ee3\u5165 ==\n\nif( strPocketGenerated !== null ){\n  engine.setData( strPocketGenerated, response1Obj.choices[0].message.content );\n}\nif( numPocketPrompt !== null ){\n  engine.setData( numPocketPrompt, new java.math.BigDecimal( response1Obj.usage.prompt_tokens ) );\n}\nif( numPocketTotal !== null ){\n  engine.setData( numPocketTotal, new java.math.BigDecimal( response1Obj.usage.total_tokens ) );\n}\n\n} \/\/\/\/\/\/\/\/ END "main()" \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\n\n\n\/*\nNotes:\n- This [Automated Step] obtains the answer text via OpenAI API (Chat endpoint).\n    - Specify the instruction (prompt) using Text and Image-Url.\n    - Also possible to specify multiple images (Image-Urls).\n    - Compatible with GPT-4V preview version (gpt-4-vision-preview).\n        - Specifications are subject to change.\n        - https:\/\/platform.openai.com\/docs\/guides\/vision\n- If place this [Automated Atep] in the workflow diagram, communication will occur every time a process arrives.\n    - Request from the Questetra BPM Suite server to the OpenAI server.\n    - Analyzes the response from the OpenAI server and stores the necessary information.\n- [HTTP Authz Settings] is required for workflow apps that include this [Automated Step].\n    - An API key is required to use OpenAI API. Please obtain an API key in advance.\n        - https:\/\/platform.openai.com\/api-keys\n    - Set 'Secret API Key' as communication token. [HTTP Authz Settings] > [Token Fixed Value]\n\nAPPENDIX\n- For low res mode, a 512px x 512px image is expected.\n- For high res mode,\n    - the short side of the image should be less than 768px and\n    - the long side of the image should be less than 2,000px.\n- Supported type of files\n    - PNG (.png), JPEG (.jpeg .jpg), WEBP (.webp), and non-animated GIF (.gif)\n- An error may occur depending on the timing.\n    - 429 error ('Too Many Requests')\n    - Timeout script error `java.util.concurrent.TimeoutException`\n\n\nNotes-ja:\n- \u3053\u306e\uff3b\u81ea\u52d5\u5de5\u7a0b\uff3d\u306f\u3001OpenAI API \uff08Chat \u30a8\u30f3\u30c9\u30dd\u30a4\u30f3\u30c8\uff09\u3092\u901a\u3058\u3066\u3001\u56de\u7b54\u6587\u3092\u53d6\u5f97\u3057\u307e\u3059\u3002\n    - \u6307\u793a\u6587\uff08\u30d7\u30ed\u30f3\u30d7\u30c8\uff09\u306f Text \u304a\u3088\u3073 Image-Url \u306b\u3066\u6307\u5b9a\u3057\u307e\u3059\u3002\n    - \u8907\u6570\u753b\u50cf\uff08Image-Urls\uff09\u306e\u6307\u5b9a\u3082\u53ef\u80fd\u3067\u3059\u3002\n    - GPT-4V \u30d7\u30ec\u30d3\u30e5\u30fc\u7248\uff08gpt-4-vision-preview\uff09\u306b\u5bfe\u5fdc\u3057\u307e\u3059\u3002\n        - \u4ed5\u69d8\u304c\u5909\u66f4\u3055\u308c\u308b\u53ef\u80fd\u6027\u304c\u3042\u308a\u307e\u3059\u3002\n        - https:\/\/platform.openai.com\/docs\/guides\/vision\n- \u3053\u306e\uff3b\u81ea\u52d5\u5de5\u7a0b\uff3d\u3092\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u56f3\u306b\u914d\u7f6e\u3059\u308c\u3070\u3001\u6848\u4ef6\u5230\u9054\u306e\u5ea6\u306b\u901a\u4fe1\u304c\u767a\u751f\u3057\u307e\u3059\u3002\n    - Questetra BPM Suite \u30b5\u30fc\u30d0\u304b\u3089 OpenAI \u30b5\u30fc\u30d0\u306b\u5bfe\u3057\u3066\u30ea\u30af\u30a8\u30b9\u30c8\u3057\u307e\u3059\u3002\n    - OpenAI \u30b5\u30fc\u30d0\u304b\u3089\u306e\u30ec\u30b9\u30dd\u30f3\u30b9\u3092\u89e3\u6790\u3057\u3001\u5fc5\u8981\u60c5\u5831\u3092\u683c\u7d0d\u3057\u307e\u3059\u3002\n- \u3053\u306e\uff3b\u81ea\u52d5\u5de5\u7a0b\uff3d\u3092\u542b\u3080\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u30a2\u30d7\u30ea\u306b\u306f\u3001\uff3bHTTP \u8a8d\u8a3c\u8a2d\u5b9a\uff3d\u304c\u5fc5\u8981\u3067\u3059\u3002\n    - OpenAI API \u306e\u5229\u7528\u306b\u306f API key \u304c\u5fc5\u8981\u3067\u3059\u3002\u3042\u3089\u304b\u3058\u3081 API Key \u3092\u53d6\u5f97\u3057\u3066\u304a\u3044\u3066\u304f\u3060\u3055\u3044\u3002\n        - https:\/\/platform.openai.com\/api-keys\n    - 'Secret API Key' \u3092\u901a\u4fe1\u30c8\u30fc\u30af\u30f3\u3068\u3057\u3066\u30bb\u30c3\u30c8\u3057\u307e\u3059\u3002\uff3bHTTP \u8a8d\u8a3c\u8a2d\u5b9a\uff3d\uff1e\uff3b\u30c8\u30fc\u30af\u30f3\u76f4\u63a5\u6307\u5b9a\uff3d\n\nAPPENDIX-ja\n- \u4f4e\u89e3\u50cf\u5ea6\u30e2\u30fc\u30c9\uff08low\uff09\u306e\u5834\u5408\u3001512px x 512px \u753b\u50cf\u304c\u63a8\u5968\u3067\u3059\u3002\n- \u9ad8\u89e3\u50cf\u5ea6\u30e2\u30fc\u30c9\uff08high\uff09\u306e\u5834\u5408\u3001\n    - \u753b\u50cf\u306e\u77ed\u8fba\u306f 768px \u672a\u6e80\u3001\n    - \u753b\u50cf\u306e\u9577\u8fba\u306f 2,000px \u672a\u6e80\u3067\u3042\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002\n- \u30b5\u30dd\u30fc\u30c8\u3055\u308c\u308b\u753b\u50cf\u30d5\u30a9\u30fc\u30de\u30c3\u30c8\n    - PNG (.png), JPEG (.jpeg .jpg), WEBP (.webp), and non-animated GIF (.gif)\n- \u30bf\u30a4\u30df\u30f3\u30b0\u306b\u3088\u3063\u3066\u3001\u30a8\u30e9\u30fc\u306b\u306a\u308b\u5834\u5408\u304c\u3042\u308a\u307e\u3059\u3002\n    - 429\u30a8\u30e9\u30fc\uff08'Too Many Requests'\uff09\n    - Timeout \u30b9\u30af\u30ea\u30d7\u30c8\u30a8\u30e9\u30fc `java.util.concurrent.TimeoutException`\n*\/\n\n<\/code><\/pre><\/div>\n<\/div><\/details><\/div>\n<\/div>\n\n\n\n
Download<\/span><\/h3>\n\n\n\n