{"id":154209,"date":"2023-11-07T17:07:13","date_gmt":"2023-11-07T08:07:13","guid":{"rendered":"https:\/\/support.questetra.com\/?p=154209"},"modified":"2023-11-14T16:34:16","modified_gmt":"2023-11-14T07:34:16","slug":"openai-chat-answer-prompt-with-images-202311","status":"publish","type":"post","link":"https:\/\/support.questetra.com\/ja\/addons\/openai-chat-answer-prompt-with-images-202311\/","title":{"rendered":"OpenAI #Chat: \u753b\u50cf\u4ed8\u304d\u30d7\u30ed\u30f3\u30d7\u30c8\u306b\u56de\u7b54"},"content":{"rendered":"\n

OpenAI #Chat: \u753b\u50cf\u4ed8\u304d\u30d7\u30ed\u30f3\u30d7\u30c8\u306b\u56de\u7b54<\/strong><\/h4>\n\n\n\n

translate<\/span> OpenAI #Chat: Answer Prompt with Images<\/a><\/p>\n\n\n\n

\u753b\u50cf\u4ed8 Prompt \u306b\u5bfe\u3059\u308b\u56de\u7b54\u6587\u3092\u751f\u6210\u3057\u307e\u3059\u3002MODEL \u306f\u753b\u50cf\u3092\u53d6\u308a\u8fbc\u307f\u3001\u753b\u50cf\u306b\u95a2\u9023\u3059\u308b\u8cea\u554f\u306b\u7b54\u3048\u307e\u3059\u3002\u753b\u50cf\u304c\u4f55\u3092\u8868\u73fe\u3057\u3066\u3044\u308b\u304b\u3001\u753b\u50cf\u5185\u306b\u4f55\u304c\u5b58\u5728\u3059\u308b\u304b\u3001\u305d\u308c\u3089\u3092\u8e0f\u307e\u3048\u305f\u8cea\u554f\uff08\u51b7\u8535\u5eab\u306e\u4e2d\u8eab\u304b\u3089\u5915\u98df\u30ec\u30b7\u30d4\u306a\u3069\uff09\u306b\u7b54\u3048\u307e\u3059\u3002 (GPT-4 with Vision)<\/p>\n\n\n\n

\"Auto<\/figure>\n\n\n\n
\n
Configs for this Auto Step<\/summary>
\n
\n
AuthzConfU1<\/dt>
U1: HTTP\u8a8d\u8a3c\u8a2d\u5b9a\u3092\u9078\u629e\u3057\u3066\u304f\u3060\u3055\u3044\uff08Secret API Key \uff20\u30c8\u30fc\u30af\u30f3\u76f4\u63a5\u6307\u5b9a\uff09 *<\/span><\/dd>\n
StrConfA1<\/dt>
A1: \u30c6\u30ad\u30b9\u30c8 PROMPT \u3092\u30bb\u30c3\u30c8\u3057\u3066\u304f\u3060\u3055\u3044 *<\/span>#{EL}<\/sup><\/span><\/dd>\n
StrConfA2<\/dt>
A2: \u753b\u50cf URL \u3092\u5404\u884c\u306b\u30bb\u30c3\u30c8\u3057\u3066\u304f\u3060\u3055\u3044 *<\/span>#{EL}<\/sup><\/span><\/dd>\n
SelectConfC1<\/dt>
C1: \u751f\u6210\u30c6\u30ad\u30b9\u30c8\u304c\u683c\u7d0d\u3055\u308c\u308b\u6587\u5b57\u5217\u578b\u30c7\u30fc\u30bf\u9805\u76ee\u3092\u9078\u629e\u3057\u3066\u304f\u3060\u3055\u3044\uff08\u66f4\u65b0\uff09<\/dd>\n<\/div>\n
StrConfM<\/dt>
M: \u30e2\u30c7\u30eb\u540d\uff08AI\u30a8\u30f3\u30b8\u30f3\u540d\uff09\u3092\u30bb\u30c3\u30c8\u3057\u3066\u304f\u3060\u3055\u3044 (default “gpt-4-vision-preview”)#{EL}<\/sup><\/span><\/dd>\n
StrConfU2<\/dt>
U2: OpenAI Organization ID \u3092\u30bb\u30c3\u30c8\u3057\u3066\u304f\u3060\u3055\u3044\uff08”org-xxxx”\uff09#{EL}<\/sup><\/span><\/dd>\n
StrConfB1<\/dt>
B1: \u8a73\u7d30\u5ea6\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u30bb\u30c3\u30c8\u3057\u3066\u304f\u3060\u3055\u3044 (“high” or “low”:default)#{EL}<\/sup><\/span><\/dd>\n
StrConfB2<\/dt>
B2: MaxTokens \u3092\u30bb\u30c3\u30c8\u3057\u3066\u304f\u3060\u3055\u3044#{EL}<\/sup><\/span><\/dd>\n
SelectConfC2<\/dt>
C2: PROMPT\u30c8\u30fc\u30af\u30f3\u6570\u304c\u683c\u7d0d\u3055\u308c\u308b\u6570\u5024\u578b\u30c7\u30fc\u30bf\u9805\u76ee\u3092\u9078\u629e\u3057\u3066\u304f\u3060\u3055\u3044\uff08\u66f4\u65b0\uff09<\/dd>\n
SelectConfC3<\/dt>
C3: \u5408\u8a08\u30c8\u30fc\u30af\u30f3\u6570\u304c\u683c\u7d0d\u3055\u308c\u308b\u6570\u5024\u578b\u30c7\u30fc\u30bf\u9805\u76ee\u3092\u9078\u629e\u3057\u3066\u304f\u3060\u3055\u3044\uff08\u66f4\u65b0\uff09<\/dd>\n<\/div>\n\n<\/dl>\n<\/div><\/details><\/div>\n<\/div>\n\n\n\n
\n
Script (click to open)<\/summary>
\n
\/\/ Script Example of Business Process Automation\n\/\/ for 'engine type: 3' ("GraalJS standard mode")\n\/\/ cf. 'engine type: 2' ("GraalJS Nashorn compatible mode") (renamed from "GraalJS" at 20230526)\n\n\/\/\/\/\/\/\/\/ START "main()" \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\n\nmain();\nfunction main(){ \n\n\/\/\/\/\/\/ == Config Retrieving \/ \u5de5\u7a0b\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u53c2\u7167 ==\nconst strAuthzSetting  = configs.get      ( "AuthzConfU1" );                       \/\/\/ REQUIRED\n  engine.log( " AutomatedTask Config: Authz Setting: " + strAuthzSetting );\nconst strOrgId         = configs.get      ( "StrConfU2" );                         \/\/ NotRequired\n  engine.log( " AutomatedTask Config: OpenAI-Organization: " + strOrgId );\nconst strModel         = configs.get      ( "StrConfM" ) !== "" ?                  \/\/ NotRequired\n                         configs.get      ( "StrConfM" ) : "gpt-4-vision-preview"; \/\/ (default)\n  engine.log( " AutomatedTask Config: OpenAI Model: " + strModel );\n\nconst strTextPrompt    = configs.get      ( "StrConfA1" );                         \/\/\/ REQUIRED\n  if( strTextPrompt  === "" ){\n    throw new Error( "\\n AutomatedTask ConfigError:" +\n                     " Config {A1: Prompt} must be non-empty \\n" );\n  }\nconst strImageUrls     = configs.get      ( "StrConfA2" );                         \/\/\/ REQUIRED\n  if( strImageUrls   === "" ){\n    throw new Error( "\\n AutomatedTask ConfigError:" +\n                     " Config {A2: ImageUrls} must be non-empty \\n" );\n  }\nconst arrImageUrls     = strImageUrls.split("\\n");\n\nconst strDetail        = configs.get      ( "StrConfB1" ) !== "" ?                 \/\/ NotRequired\n                         configs.get      ( "StrConfB1" ) : "low";                 \/\/ (default)\nconst strMaxTokens     = configs.get      ( "StrConfB2" );                         \/\/ NotRequired\nconst numMaxTokens     = parseInt ( strMaxTokens, 10 );\n  engine.log( " AutomatedTask Config: Max Tokens: " + numMaxTokens );\n\nconst strPocketGenerated = configs.getObject ( "SelectConfC1" );                   \/\/\/ REQUIRED NotRequired\nconst numPocketPrompt    = configs.getObject ( "SelectConfC2" );                   \/\/ NotRequired\nconst numPocketTotal     = configs.getObject ( "SelectConfC3" );                   \/\/ NotRequired\n\n\n\n\/\/\/\/\/\/ == Data Retrieving \/ \u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u30c7\u30fc\u30bf\u306e\u53c2\u7167 ==\n\/\/ (Nothing. Retrieved via Expression Language in Config Retrieving)\n\n\n\/\/\/\/\/\/ == Calculating \/ \u6f14\u7b97 ==\n\n\/\/\/\/ OpenAI API > Documentation > API REFERENCE > CHAT\n\/\/\/\/ https:\/\/platform.openai.com\/docs\/api-reference\/chat\/create (not updated)\n\/\/\/\/ https:\/\/platform.openai.com\/docs\/guides\/vision\n\n\/\/\/ prepare json\nlet strJson = {};\n    strJson.model = strModel;\n    if ( ! isNaN(numMaxTokens) ) {\n      strJson.max_tokens       = numMaxTokens;\n    }\n\/\/    strJson.response_format = {};\n\/\/    strJson.response_format.type = "json_object";\n\n    strJson.messages = [];\n    strJson.messages[0] = {};\n    strJson.messages[0].role = "user";\n    strJson.messages[0].content = [];\n    strJson.messages[0].content[0] = {};\n    strJson.messages[0].content[0].type = "text";\n    strJson.messages[0].content[0].text = strTextPrompt;\n\n    for ( let i = 0; i < arrImageUrls.length; i++ ) {\n      const objTmp = {};\n      objTmp.type = "image_url";\n      objTmp.image_url = arrImageUrls[i];\n      strJson.messages[0].content.push ( objTmp );\n    }\n\n\/\/\/ prepare request1\nlet request1Uri = "https:\/\/api.openai.com\/v1\/chat\/completions";\nlet request1 = httpClient.begin(); \/\/ HttpRequestWrapper\n    request1 = request1.queryParam( "detail", strDetail );\n    request1 = request1.authSetting( strAuthzSetting ); \/\/ with "Authorization: Bearer XX"\n    request1 = request1.body( JSON.stringify( strJson ), "application\/json" );\n    if ( strOrgId !== "" ){\n      request1 = request1.header( "OpenAI-Organization", strOrgId );\n    }\n\n\/\/\/ try request1\nconst response1     = request1.post( request1Uri ); \/\/ HttpResponseWrapper\nengine.log( " AutomatedTask ApiRequest1 Start: " + request1Uri );\nconst response1Code = response1.getStatusCode() + ""; \/\/ JavaNum to string\nconst response1Body = response1.getResponseAsString();\nengine.log( " AutomatedTask ApiResponse1 Status: " + response1Code );\nif( response1Code !== "200"){\n  throw new Error( "\\n AutomatedTask UnexpectedResponseError: " +\n                    response1Code + "\\n" + response1Body + "\\n" );\n}\n\n\n\/\/\/ parse response1\n\/* engine.log( response1Body ); \/\/ debug\n{\n  "id": "chatcmpl-8I9gORGEvsLaVE10Dir0pOMZI5I37",\n  "object": "chat.completion",\n  "created": 1699337816,\n  "model": "gpt-4-1106-vision-preview",\n  "usage": {\n    "prompt_tokens": 1887, \n    "completion_tokens": 16,\n    "total_tokens": 1903\n  },\n  "choices": [{\n    "message": {\n      "role": "assistant", \n      "content": "\\u6700\\u521d\\u306e\\u753b\\u50cf\\u306b\\u306f\\u3001\\"Fine tuning workflow\\" \\u3068\\u3044\\u3046"\n    },\n    "finish_details": {\n      "type": "max_tokens"\n    },\n    "index": 0\n    }]\n}\n*\/\nconst response1Obj = JSON.parse( response1Body );\nengine.log( " AutomatedTask ApiResponse1 finish_details: " + response1Obj.choices[0].finish_details.type );\n\n\n\/\/\/\/\/\/ == Data Updating \/ \u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u30c7\u30fc\u30bf\u3078\u306e\u4ee3\u5165 ==\n\nif( strPocketGenerated !== null ){\n  engine.setData( strPocketGenerated, response1Obj.choices[0].message.content );\n}\nif( numPocketPrompt !== null ){\n  engine.setData( numPocketPrompt, new java.math.BigDecimal( response1Obj.usage.prompt_tokens ) );\n}\nif( numPocketTotal !== null ){\n  engine.setData( numPocketTotal, new java.math.BigDecimal( response1Obj.usage.total_tokens ) );\n}\n\n} \/\/\/\/\/\/\/\/ END "main()" \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\n\n\n\/*\nNotes:\n- This [Automated Step] obtains the answer text via OpenAI API (Chat endpoint).\n    - Specify the instruction (prompt) using Text and Image-Url.\n    - Also possible to specify multiple images (Image-Urls).\n    - Compatible with GPT-4V preview version (gpt-4-vision-preview).\n        - Specifications are subject to change.\n        - https:\/\/platform.openai.com\/docs\/guides\/vision\n- If place this [Automated Atep] in the workflow diagram, communication will occur every time a process arrives.\n    - Request from the Questetra BPM Suite server to the OpenAI server.\n    - Analyzes the response from the OpenAI server and stores the necessary information.\n- [HTTP Authz Settings] is required for workflow apps that include this [Automated Step].\n    - An API key is required to use OpenAI API. Please obtain an API key in advance.\n        - https:\/\/platform.openai.com\/api-keys\n    - Set 'Secret API Key' as communication token. [HTTP Authz Settings] > [Token Fixed Value]\n\nAPPENDIX\n- For low res mode, a 512px x 512px image is expected.\n- For high res mode,\n    - the short side of the image should be less than 768px and\n    - the long side of the image should be less than 2,000px.\n- Supported type of files\n    - PNG (.png), JPEG (.jpeg .jpg), WEBP (.webp), and non-animated GIF (.gif)\n- An error may occur depending on the timing.\n    - 429 error ('Too Many Requests')\n    - Timeout script error `java.util.concurrent.TimeoutException`\n\n\nNotes-ja:\n- \u3053\u306e\uff3b\u81ea\u52d5\u5de5\u7a0b\uff3d\u306f\u3001OpenAI API \uff08Chat \u30a8\u30f3\u30c9\u30dd\u30a4\u30f3\u30c8\uff09\u3092\u901a\u3058\u3066\u3001\u56de\u7b54\u6587\u3092\u53d6\u5f97\u3057\u307e\u3059\u3002\n    - \u6307\u793a\u6587\uff08\u30d7\u30ed\u30f3\u30d7\u30c8\uff09\u306f Text \u304a\u3088\u3073 Image-Url \u306b\u3066\u6307\u5b9a\u3057\u307e\u3059\u3002\n    - \u8907\u6570\u753b\u50cf\uff08Image-Urls\uff09\u306e\u6307\u5b9a\u3082\u53ef\u80fd\u3067\u3059\u3002\n    - GPT-4V \u30d7\u30ec\u30d3\u30e5\u30fc\u7248\uff08gpt-4-vision-preview\uff09\u306b\u5bfe\u5fdc\u3057\u307e\u3059\u3002\n        - \u4ed5\u69d8\u304c\u5909\u66f4\u3055\u308c\u308b\u53ef\u80fd\u6027\u304c\u3042\u308a\u307e\u3059\u3002\n        - https:\/\/platform.openai.com\/docs\/guides\/vision\n- \u3053\u306e\uff3b\u81ea\u52d5\u5de5\u7a0b\uff3d\u3092\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u56f3\u306b\u914d\u7f6e\u3059\u308c\u3070\u3001\u6848\u4ef6\u5230\u9054\u306e\u5ea6\u306b\u901a\u4fe1\u304c\u767a\u751f\u3057\u307e\u3059\u3002\n    - Questetra BPM Suite \u30b5\u30fc\u30d0\u304b\u3089 OpenAI \u30b5\u30fc\u30d0\u306b\u5bfe\u3057\u3066\u30ea\u30af\u30a8\u30b9\u30c8\u3057\u307e\u3059\u3002\n    - OpenAI \u30b5\u30fc\u30d0\u304b\u3089\u306e\u30ec\u30b9\u30dd\u30f3\u30b9\u3092\u89e3\u6790\u3057\u3001\u5fc5\u8981\u60c5\u5831\u3092\u683c\u7d0d\u3057\u307e\u3059\u3002\n- \u3053\u306e\uff3b\u81ea\u52d5\u5de5\u7a0b\uff3d\u3092\u542b\u3080\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u30a2\u30d7\u30ea\u306b\u306f\u3001\uff3bHTTP \u8a8d\u8a3c\u8a2d\u5b9a\uff3d\u304c\u5fc5\u8981\u3067\u3059\u3002\n    - OpenAI API \u306e\u5229\u7528\u306b\u306f API key \u304c\u5fc5\u8981\u3067\u3059\u3002\u3042\u3089\u304b\u3058\u3081 API Key \u3092\u53d6\u5f97\u3057\u3066\u304a\u3044\u3066\u304f\u3060\u3055\u3044\u3002\n        - https:\/\/platform.openai.com\/api-keys\n    - 'Secret API Key' \u3092\u901a\u4fe1\u30c8\u30fc\u30af\u30f3\u3068\u3057\u3066\u30bb\u30c3\u30c8\u3057\u307e\u3059\u3002\uff3bHTTP \u8a8d\u8a3c\u8a2d\u5b9a\uff3d\uff1e\uff3b\u30c8\u30fc\u30af\u30f3\u76f4\u63a5\u6307\u5b9a\uff3d\n\nAPPENDIX-ja\n- \u4f4e\u89e3\u50cf\u5ea6\u30e2\u30fc\u30c9\uff08low\uff09\u306e\u5834\u5408\u3001512px x 512px \u753b\u50cf\u304c\u63a8\u5968\u3067\u3059\u3002\n- \u9ad8\u89e3\u50cf\u5ea6\u30e2\u30fc\u30c9\uff08high\uff09\u306e\u5834\u5408\u3001\n    - \u753b\u50cf\u306e\u77ed\u8fba\u306f 768px \u672a\u6e80\u3001\n    - \u753b\u50cf\u306e\u9577\u8fba\u306f 2,000px \u672a\u6e80\u3067\u3042\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002\n- \u30b5\u30dd\u30fc\u30c8\u3055\u308c\u308b\u753b\u50cf\u30d5\u30a9\u30fc\u30de\u30c3\u30c8\n    - PNG (.png), JPEG (.jpeg .jpg), WEBP (.webp), and non-animated GIF (.gif)\n- \u30bf\u30a4\u30df\u30f3\u30b0\u306b\u3088\u3063\u3066\u3001\u30a8\u30e9\u30fc\u306b\u306a\u308b\u5834\u5408\u304c\u3042\u308a\u307e\u3059\u3002\n    - 429\u30a8\u30e9\u30fc\uff08'Too Many Requests'\uff09\n    - Timeout \u30b9\u30af\u30ea\u30d7\u30c8\u30a8\u30e9\u30fc `java.util.concurrent.TimeoutException`\n*\/\n\n<\/code><\/pre><\/div>\n<\/div><\/details><\/div>\n<\/div>\n\n\n\n
Download<\/span><\/h3>\n\n\n\n