Skip to content

API reference

Generated from the docstrings.

Frames

frame2text4llm.framer.video.VideoReader

__init__(video_path, engine='opencv')

Initialize the video reader.

Parameters:

Name Type Description Default
video_path str

Path to the video file

required
engine str

Video processing engine ('opencv' or 'ffmpeg')

'opencv'

print_info()

Print video information.

extract_frame(position)

Extract a specific frame from the video.

Parameters:

Name Type Description Default
position int

Frame number to extract

required

Returns:

Type Description
VideoFrame | None

VideoFrame object or None if extraction fails

extract_frames(target_fps=None, max_frames=-1, filter_duplicates=False, diff_thresh=10)

Extract frames from the video.

Parameters:

Name Type Description Default
target_fps int

Target sampling rate (frames per second)

None
max_frames int

Maximum number of frames to extract (-1 = no limit)

-1
filter_duplicates bool

Whether to filter duplicate frames

False
diff_thresh int

Threshold for duplicate detection (higher = more strict)

10

Returns:

Type Description
list[VideoFrame]

List of VideoFrame objects

frame2text4llm.framer.base.VideoFrame dataclass

A dataclass representing a video frame with associated metadata.

time_formatted property

Return the time in MM:SS:mmm format (minutes:seconds:milliseconds)

to_dict()

Convert the frame to a dictionary format.

from_dict(data) classmethod

Create a VideoFrame from a dictionary.

frame2text4llm.framer.subtitle.SubtitleRegionDetector

A class for detecting the subtitle region in a video.

__init__(video_reader)

Initialize the subtitle region detector.

Parameters:

Name Type Description Default
video_reader VideoReader

VideoReader object for the video

required

detect_region()

Detect the subtitle region in the video.

Returns:

Type Description
Tuple[int, int, int, int]

Tuple (y1, y2, x1, x2) representing the subtitle region

visualize_region(region, frames, frame_index=0)

Visualize the detected subtitle region on a specific frame from a list.

Parameters:

Name Type Description Default
region Tuple[int, int, int, int]

Tuple (y1, y2, x1, x2) representing the subtitle region.

required
frames List[ndarray]

List of frames as numpy arrays (BGR format).

required
frame_index int

Index of the frame in the list to visualize.

0

OCR

frame2text4llm.ocr.manager.OCRManager

Manages different OCR tools for text extraction from images. Supports multiple OCR engines: Tesseract, PaddleOCR, OpenAI, and Mistral.

__init__(video_reader, region=None)

Initialize OCR Manager with available tools.

region: (y1, y2, x1, x2) of the subtitles. Left out, it is detected on the first frame processed and kept for the whole video: detecting it reads three frames from the file.

get_tool_instance(tool_name, api_key=None, model_name=None)

Get or create an instance of the requested OCR tool.

Parameters:

Name Type Description Default
tool_name str

Name of the OCR tool.

required
api_key str | None

API key for cloud-based OCR services.

None
model_name str | None

Model name for VLM tools (e.g., "Florence-2-base", "InternVL2-1B").

None

Returns:

Type Description
Any

Instance of the OCR tool.

process(image, tool='tesseract', lang='eng', api_key=None, detect_subtitle_reggion=True, model_name=None)

Process an image using the selected OCR tool.

Parameters:

Name Type Description Default
image ndarray

NumPy array containing the image.

required
tool str

OCR tool to use ('tesseract', 'paddleocr', 'openai', 'mistral', 'vlm').

'tesseract'
lang str

Language code (for tools that support it).

'eng'
api_key str | None

API key for cloud-based OCR services.

None
detect_subtitle_reggion bool

Whether to detect subtitle regions in the image and crop it before ocr.

True
model_name str | None

Model name for VLM tools (e.g., "Florence-2-base", "InternVL2-1B").

None

Returns:

Type Description
str

Extracted text from the image.

batch_process(images, tool='tesseract', lang='', api_key=None)

Process multiple images using the selected OCR tool.

Parameters:

Name Type Description Default
images list[ndarray]

List of NumPy arrays containing images.

required
tool str

OCR tool to use.

'tesseract'
lang str

Language code.

''
api_key str | None

API key for cloud-based OCR services.

None

Returns:

Type Description
list[str]

List of extracted texts.

frame2text4llm.ocr.batch.OCRBatchProcessor

A class to handle batch processing of OCR tasks with parallelization and progress tracking.

__init__(ocr_manager, default_tool='tesseract', default_lang='en', default_cores=4)

Initialize the OCR batch processor.

Parameters:

Name Type Description Default
ocr_manager

The OCR manager instance that handles actual text extraction

required
default_tool str

Default OCR engine to use

'tesseract'
default_lang str

Default language code

'en'
default_cores int

Default number of concurrent threads

4

process_batch(frames, tool=None, lang=None, api_key=None, model_name=None, n_cores=None, preserve_order=True, show_progress=True, custom_processor=None, description='🔠 OCR Processing', merge_segments=False, sim_thresh=0.8, overlay_frames=None)

Run OCR on a batch of video frames using parallel processing.

Parameters:

Name Type Description Default
frames list[Any]

List of Frame objects (must contain .image)

required
tool str | None

OCR tool to use (defaults to instance default)

None
lang str | None

Language code (defaults to instance default)

None
api_key str | None

API key for tools like OpenAI or Mistral

None
model_name str | None

Model name for VLM tools (e.g., "Florence-2-base", "InternVL2-1B")

None
n_cores int | None

Number of threads to use (defaults to instance default)

None
preserve_order bool

Whether to preserve original frame order in results

True
show_progress bool

Whether to display a progress bar

True
custom_processor Callable | None

Optional custom function to process images

None
description str

Description for the progress bar

'🔠 OCR Processing'
merge_segments bool

Whether to merge similar consecutive segments

False
sim_thresh float

Similarity threshold for merging (0.0-1.0)

0.8
overlay_frames int | None

Take out the words read on this many frames in a row or more (logos, banners), before merging; 60 is 30 s at 2 frames a second. None keeps them.

None

Returns:

Type Description
list[dict[str, Any]]

List of dicts with frame metadata and extracted text

frame2text4llm.ocr.utils.text_utils._merge_segments(ocr_results, sim_thresh=0.8, min_shared=0.5, max_gap=1)

Merge consecutive frames that show the same text into one segment.

The OCR reads one subtitle differently from frame to frame: a line skipped, an accent, a word lost. Two readings are the same subtitle when they look alike (sim_thresh) or when at least min_shared of the shorter one's words are in the other; and up to max_gap frames read as empty inside a subtitle do not end it. The segment keeps the longest reading, and best_time is the frame it was read on, the one to read again with a better OCR.

A segment ends at the first frame that shows other text, or none: that is when the subtitle left the screen, so a subtitle seen on a single frame still lasts one frame interval. The frames must be in time order and not deduplicated, or a segment runs on to the next kept frame.

frame2text4llm.ocr.utils.filters.strip_overlays(results, frames=60, holes=2)

Take out of the frames' text the words read on at least frames frames in a row, up to holes frames missed: 30 s at 2 frames a second. No subtitle stays that long; a logo, a banner or a credit line does. Left in, they make _merge_segments join every subtitle shown with them into one segment.

Words are compared without case or punctuation, and two neighbours also count joined, so that the readings clideo com, clideocom and clideo.com of one logo are one word. Changes results in place, in time order as OCRBatchProcessor returns them, and returns the words taken from each frame, by its time_formatted.

frame2text4llm.ocr.utils.filters.mark_non_subtitles(segments, duration, title='')

Give each merged segment a drop: "" for a subtitle, else what it is.

  • "credits": a credits keyword, two words in capitals, a first name and a NAME, or the video's title. Two credits in a row after 85 % of the video's duration (seconds) drop everything after them too.
  • "ticker": three segments or more in a row, each the one before moved, as a scrolling text is read.

Changes segments in place and returns them. Segments are _merge_segments's, in time order.

frame2text4llm.ocr.utils.filters.windows(segments, gap=2.0, low=5.0, high=30.0)

The subtitles in windows to cut audio on: a subtitle alone is often too short for speech recognition.

Consecutive subtitles less than gap s apart form a run, and each run is packed into windows of at least low s; a short tail joins the window before it. A segment with a drop (mark_non_subtitles) ends a run, since its sound has no text. A window shorter than low or longer than high s is left out.

Each window is a segment, so that AudioGenerator.extract_segments cuts it: its subtitles' text end to end, the first one's start_time, the last one's end_time, and the subtitles themselves in subtitles.

Language

frame2text4llm.language.LanguageID

MMS-LID (facebook/mms-lid-256, 256 languages) on audio files of 16 kHz mono, such as the windows AudioGenerator cuts. languages maps the ISO 639-3 codes whose probabilities are kept to column names.

hear(paths, batch=8)

For each file, the language heard most (by languages' name when it is one of them, else its code) and the probability of each of languages. Files are sorted by length, so that a batch pads little.

frame2text4llm.language.written_in(text)

"french", "english" or "other" (another language, names, a title), by counting function words; "other" when they tie.

Copies

frame2text4llm.copies.TextIndex

The subtitles of the videos read so far, to find which one a new line comes from.

add(key, texts)

A video's subtitles, under key.

closest(line, candidates=10)

The score out of 100 of the best alignment of line inside a video's text, and that video's key; (0, "") when no video shares a 5-gram with it. Only the candidates videos sharing most 5-grams are aligned.

Engines

frame2text4llm.ocr.tools.easyocr.EasyOCR

EasyOCR implementation for text extraction from images.

process_image(image, lang='en')

Extract text from image using EasyOCR.

Parameters:

Name Type Description Default
image ndarray

NumPy array containing the image

required
lang str

Language code

'en'

Returns:

Type Description
str

Extracted text from the image

frame2text4llm.ocr.tools.paddleocr.PaddleOCR

PaddleOCR implementation.

__init__(cache_dir=None)

Initialize PaddleOCR and create cache directory if needed.

process_image(image, lang='en')

Extract text from image using PaddleOCR.

Parameters:

Name Type Description Default
image ndarray

NumPy array containing the image

required
lang str

Language code (for PaddleOCR, use 'en', 'ch', etc.)

'en'

Returns:

Type Description
str

Extracted text from the image

frame2text4llm.ocr.tools.vlmocr.VLMOCR

VLM-based OCR implementation (using lightweight models)

__init__(model_name='Qwen/Qwen3-VL-2B-Instruct', prompt=PROMPT)

Initialize VLM OCR with specified model.

Parameters:

Name Type Description Default
model_name str

Florence-2 ('microsoft/Florence-2-base-ft', 'microsoft/Florence-2-base'), 'OpenGVLab/InternVL2-1B', or any image-text-to-text model of transformers, such as 'Qwen/Qwen3-VL-2B-Instruct', which reads subtitles best (docs: Choosing an OCR engine).

'Qwen/Qwen3-VL-2B-Instruct'
prompt str

What an image-text-to-text model is asked; it answers "-" for no subtitle.

PROMPT

process_images(images, lang='en', batch_size=16)

process_image on many images: an image-text-to-text model reads batch_size at a time, which keeps the GPU busy; Florence-2 and InternVL2 still read one by one.

process_image(image, lang='en')

Extract text from image using VLM models.

Parameters:

Name Type Description Default
image ndarray

NumPy array containing the image

required
lang str

Language code (not used by VLM models)

'en'

Returns:

Type Description
str

Extracted text from the image

frame2text4llm.ocr.tools.mistralocr.MistralOCR

OCR implementation using Mistral's OCR API.

__init__(api_key=None)

Initialize Mistral OCR engine.

Parameters:

Name Type Description Default
api_key str | None

Optional API key. If not provided, tries to use MISTRAL_API_KEY env var.

None

process_image(base64_img)

Extract text from image using Mistral OCR API.

Parameters:

Name Type Description Default
base64_img str

Base64-encoded image string.

required
lang

Optional language hint (not directly used).

required

Returns:

Type Description
str

Extracted text from the image.

frame2text4llm.ocr.tools.openaiocr.OpenAIOCR

OCR implementation using OpenAI's Vision API.

__init__(api_key=None)

Initialize OpenAI OCR engine.

Parameters:

Name Type Description Default
api_key str | None

Optional API key. If not provided, tries to use OPENAI_API_KEY env var.

None

process_image(base64_img)

Extract text from image using OpenAI's Vision model.

Parameters:

Name Type Description Default
base64_img str

Base64-encoded image string.

required

Returns:

Type Description
str

Extracted text from the image.

Download and export

frame2text4llm.consumer.youtube_consumer.YoutubeConsumer

Class to consume playlists and videos from YouTube.

frame2text4llm.export.audio_generator.AudioGenerator

Extracts audio from a video and segments it according to timestamps. Segments are saved as .wav files in a temporary directory.

extract_segments(segments, sample_rate=16000)

For each segment (with start_time, end_time), extract audio and save as wav. Returns list of audio file paths.

frame2text4llm.export.exporter.SRTExporter

Bases: BaseExporter

Export OCR results to SRT format.

export(data, output_path)

Export OCR results to SRT format.

Parameters:

Name Type Description Default
data list[dict[str, Any]]

List of dictionaries containing OCR results

required
output_path str

Path to save the SRT file

required

frame2text4llm.export.exporter.JSONExporter

Bases: BaseExporter

Export OCR results to JSON format.

export(data, output_path)

Export OCR results to JSON format.

Parameters:

Name Type Description Default
data list[dict[str, Any]]

List of dictionaries containing OCR results

required
output_path str

Path to save the JSON file

required

frame2text4llm.export.huggingface_cooker.HuggingfaceCooker

A class to prepare and upload datasets to Huggingface Hub.

__init__(username, token=None, repository_id=None)

Initialize the Huggingface Cooker.

Parameters:

Name Type Description Default
username str

Huggingface username

required
token str | None

Huggingface API token (optional if not pushing to hub)

None
repository_id str | None

Repository ID to push to (optional)

None

cook_dataset(ocr_results, video_path, output_dir, push_to_hub=False)

Cook a Huggingface dataset from OCR results and video file.

Parameters:

Name Type Description Default
ocr_results list[dict[str, Any]]

List of OCR results with timestamps and text

required
video_path str

Path to the video file

required
output_dir str

Directory to save the prepared dataset

required
push_to_hub bool

Whether to push the dataset to Huggingface Hub

False

Returns:

Type Description
Dataset

The created Huggingface Dataset