Explore how you can use the new Gemini Pro Vision model with the Gemini API to handle multimodal input data including text and image prompts to receive a text result. In the snippet we added, and in the notebook, you’ll change the visualizing for masks by including the mask annotator from supervision (opens in a new tab), as well as adding it to the annotation loop. This code imports libraries for tasks like reading images (cv2, PIL), handling JSON data (json), interacting with the API (google.generativeai), and utility functions (ultralytics). This is where data annotation becomes essential, particularly for supervised learning tasks like training a custom object detector. We’ll also touch on data annotation tools (like those used with YOLO) to give context for custom training scenarios. This guide walks you through the steps to leverage Google Gemini for computer vision, including how to set up your environment, send images with instructions, and interpret the model’s outputs for object detection, caption generation, and OCR.
By accurately identifying and delineating objects with bounding boxes, developers can unlock a wide range of applications and enhance the intelligence of their projects. Upload an image, specify the objects you want to detect, and receive accurate bounding boxes and labels, all powered by the Gemini API. These fully functional, end-to-end applications showcase the power of Gemini in real-world scenarios. The quickstarts section contains step-by-step tutorials to get you started with Gemini and learn about its specific features. You can ask a Gemini model to generate text by prompting with text and images—providing each input file’s mimeType and the file itself.
Explore various examples of interesting ways that Gemini’s multimodal image and text inputs can be combined to extract text output about images across a variety of different use cases. Learn how to generate text from multimodal text-and-image input data Chatbot using the Gemini Pro Vision model in NodeJS. You can call GenerateContentStreamAsync() to stream generated text from multimodal input of text and images. You can call generateContentStream() to stream generated text from multimodal input of text and images. You can call GenerateContentAsync() to generate text from multimodal input of text and images. You can call generateContent() to generate text from multimodal input of text and images.
The application identifies objects in images and visualizes them with bounding boxes, creating a seamless and user-friendly experience. You have successfully used Gemini 2.5 and Supervision to perform zero-shot object detection and instance segmentation—all without writing a single training loop or collecting labeled data. The notebook snippets describe the same steps we took for object detection, except modifying the second part of the prompt to output segmentation masks in JSON, rather than bounding boxes.
🔹 Advanced AI Capabilities ✔ World Knowledge Integration (Identifying dog breeds) ✔ Handwritten Text-Based Object Detection ✔ Spatial Reasoning & Scene Understanding Each line of the JSON Lines file contains all objects found in a single image file. We also showcase Demos in separate repositories, illustrating end-to-end applications of the Gemini API.
This project demonstrates how to leverage Google Gemini 2.5 Pro for multimodal tasks including object detection, image captioning, visual reasoning, and more — using just a single API. In this codelab, you will write a NodeJS script that leverages the Gemini Pro Vision model to analyze a local HTML document and generate accessible descriptions of the images in the page if necessary. Learn how to use the multimodal features of the Gemini model https://aicurated.com/ to analyze HTML documents and image files for the purpose of adding accessible descriptions to a webpage in a NodeJS script.
Overall, execution should print out a response including the coordinates/measurements of bounding boxes on the image in JSON format, as instructed by the prompt. The colab notebook provides some sample images by executing 2 commands, however, if you wish to run them on different images, you can follow this guide showing how to upload your own files to google colab. By sending images along with clear text instructions, you can guide the model’s understanding and get usable, real-time results. This command installs the google-genai library to communicate with the Gemini API and the ultralytics library, which contains helpful functions for handling images and drawing on them.
Explore the Get Started and Thinking guides, including the migration guide. This cookbook provides a structured learning path for using the Gemini API, focusing on hands-on tutorials and practical examples. The notebook combines multimodal capabilities to interpret and analyze images https://aicurated.com/computer-vision-and-perception/google-gemini-puts-text-prompts-to-work-in-computer-vision/ using natural language prompts. This project demonstrates how to use Google Gemini 2.5 Pro to perform object detection, image captioning, visual reasoning, and OCR — all directly through a single API.
These features allow developers to use Gemini for vision-related tasks through Intelligent agent an API without training a separate model for each job. Google Gemini is a family of AI models built to handle multiple data types, such as text, images, audio, and code together. In this experimental launch, we are providing developers with a powerful tool for object detection and localization within images and video.
While Google Gemini for Computer Vision, can detect general objects without prior annotation, if you needed a model to detect very specific, custom objects (e.g., unique types of industrial equipment, specific product defects), you would likely need to collect images and annotate them using a tool like a YOLO annotator to train a dedicated YOLO model. These examples demonstrate how to combine multiple Gemini API features or 3rd-party tools to build more complex applications. The YOLO Annotator (often referring to tools compatible with the YOLO format, like Labeling, CVAT, or Roboflow) is designed to create labeled datasets. While Gemini models provide powerful zero-shot or few-shot capabilities for these computer vision tasks, building highly specialized computer vision models requires training on a dataset tailored to the specific problem. It showcases the model’s powerful multimodal capabilities by combining image inputs with a text-based prompt to perform advanced computer vision tasks. One of the more under appreciated features of multimodal LLMs is their ability to generate bounding boxes to detect objects.
No Comments