API reference¶
Generated from the docstrings.
Frames¶
frame2text4llm.framer.video.VideoReader
¶
__init__(video_path, engine='opencv')
¶
Initialize the video reader.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
video_path
|
str
|
Path to the video file |
required |
engine
|
str
|
Video processing engine ('opencv' or 'ffmpeg') |
'opencv'
|
print_info()
¶
Print video information.
extract_frame(position)
¶
Extract a specific frame from the video.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
position
|
int
|
Frame number to extract |
required |
Returns:
| Type | Description |
|---|---|
VideoFrame | None
|
VideoFrame object or None if extraction fails |
extract_frames(target_fps=None, max_frames=-1, filter_duplicates=False, diff_thresh=10)
¶
Extract frames from the video.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
target_fps
|
int
|
Target sampling rate (frames per second) |
None
|
max_frames
|
int
|
Maximum number of frames to extract (-1 = no limit) |
-1
|
filter_duplicates
|
bool
|
Whether to filter duplicate frames |
False
|
diff_thresh
|
int
|
Threshold for duplicate detection (higher = more strict) |
10
|
Returns:
| Type | Description |
|---|---|
list[VideoFrame]
|
List of VideoFrame objects |
frame2text4llm.framer.base.VideoFrame
dataclass
¶
frame2text4llm.framer.subtitle.SubtitleRegionDetector
¶
A class for detecting the subtitle region in a video.
__init__(video_reader)
¶
Initialize the subtitle region detector.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
video_reader
|
VideoReader
|
VideoReader object for the video |
required |
detect_region()
¶
Detect the subtitle region in the video.
Returns:
| Type | Description |
|---|---|
Tuple[int, int, int, int]
|
Tuple (y1, y2, x1, x2) representing the subtitle region |
visualize_region(region, frames, frame_index=0)
¶
Visualize the detected subtitle region on a specific frame from a list.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
region
|
Tuple[int, int, int, int]
|
Tuple (y1, y2, x1, x2) representing the subtitle region. |
required |
frames
|
List[ndarray]
|
List of frames as numpy arrays (BGR format). |
required |
frame_index
|
int
|
Index of the frame in the list to visualize. |
0
|
OCR¶
frame2text4llm.ocr.manager.OCRManager
¶
Manages different OCR tools for text extraction from images. Supports multiple OCR engines: Tesseract, PaddleOCR, OpenAI, and Mistral.
__init__(video_reader, region=None)
¶
Initialize OCR Manager with available tools.
region: (y1, y2, x1, x2) of the subtitles. Left out, it is detected on the first frame processed and kept for the whole video: detecting it reads three frames from the file.
get_tool_instance(tool_name, api_key=None, model_name=None)
¶
Get or create an instance of the requested OCR tool.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
tool_name
|
str
|
Name of the OCR tool. |
required |
api_key
|
str | None
|
API key for cloud-based OCR services. |
None
|
model_name
|
str | None
|
Model name for VLM tools (e.g., "Florence-2-base", "InternVL2-1B"). |
None
|
Returns:
| Type | Description |
|---|---|
Any
|
Instance of the OCR tool. |
process(image, tool='tesseract', lang='eng', api_key=None, detect_subtitle_reggion=True, model_name=None)
¶
Process an image using the selected OCR tool.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image
|
ndarray
|
NumPy array containing the image. |
required |
tool
|
str
|
OCR tool to use ('tesseract', 'paddleocr', 'openai', 'mistral', 'vlm'). |
'tesseract'
|
lang
|
str
|
Language code (for tools that support it). |
'eng'
|
api_key
|
str | None
|
API key for cloud-based OCR services. |
None
|
detect_subtitle_reggion
|
bool
|
Whether to detect subtitle regions in the image and crop it before ocr. |
True
|
model_name
|
str | None
|
Model name for VLM tools (e.g., "Florence-2-base", "InternVL2-1B"). |
None
|
Returns:
| Type | Description |
|---|---|
str
|
Extracted text from the image. |
batch_process(images, tool='tesseract', lang='', api_key=None)
¶
Process multiple images using the selected OCR tool.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
images
|
list[ndarray]
|
List of NumPy arrays containing images. |
required |
tool
|
str
|
OCR tool to use. |
'tesseract'
|
lang
|
str
|
Language code. |
''
|
api_key
|
str | None
|
API key for cloud-based OCR services. |
None
|
Returns:
| Type | Description |
|---|---|
list[str]
|
List of extracted texts. |
frame2text4llm.ocr.batch.OCRBatchProcessor
¶
A class to handle batch processing of OCR tasks with parallelization and progress tracking.
__init__(ocr_manager, default_tool='tesseract', default_lang='en', default_cores=4)
¶
Initialize the OCR batch processor.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ocr_manager
|
The OCR manager instance that handles actual text extraction |
required | |
default_tool
|
str
|
Default OCR engine to use |
'tesseract'
|
default_lang
|
str
|
Default language code |
'en'
|
default_cores
|
int
|
Default number of concurrent threads |
4
|
process_batch(frames, tool=None, lang=None, api_key=None, model_name=None, n_cores=None, preserve_order=True, show_progress=True, custom_processor=None, description='🔠OCR Processing', merge_segments=False, sim_thresh=0.8, overlay_frames=None)
¶
Run OCR on a batch of video frames using parallel processing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frames
|
list[Any]
|
List of Frame objects (must contain |
required |
tool
|
str | None
|
OCR tool to use (defaults to instance default) |
None
|
lang
|
str | None
|
Language code (defaults to instance default) |
None
|
api_key
|
str | None
|
API key for tools like OpenAI or Mistral |
None
|
model_name
|
str | None
|
Model name for VLM tools (e.g., "Florence-2-base", "InternVL2-1B") |
None
|
n_cores
|
int | None
|
Number of threads to use (defaults to instance default) |
None
|
preserve_order
|
bool
|
Whether to preserve original frame order in results |
True
|
show_progress
|
bool
|
Whether to display a progress bar |
True
|
custom_processor
|
Callable | None
|
Optional custom function to process images |
None
|
description
|
str
|
Description for the progress bar |
'🔠OCR Processing'
|
merge_segments
|
bool
|
Whether to merge similar consecutive segments |
False
|
sim_thresh
|
float
|
Similarity threshold for merging (0.0-1.0) |
0.8
|
overlay_frames
|
int | None
|
Take out the words read on this many frames in a row or more (logos, banners), before merging; 60 is 30 s at 2 frames a second. None keeps them. |
None
|
Returns:
| Type | Description |
|---|---|
list[dict[str, Any]]
|
List of dicts with frame metadata and extracted text |
frame2text4llm.ocr.utils.text_utils._merge_segments(ocr_results, sim_thresh=0.8, min_shared=0.5, max_gap=1)
¶
Merge consecutive frames that show the same text into one segment.
The OCR reads one subtitle differently from frame to frame: a line skipped, an accent, a word
lost. Two readings are the same subtitle when they look alike (sim_thresh) or when at least
min_shared of the shorter one's words are in the other; and up to max_gap frames read as
empty inside a subtitle do not end it. The segment keeps the longest reading, and best_time
is the frame it was read on, the one to read again with a better OCR.
A segment ends at the first frame that shows other text, or none: that is when the subtitle left the screen, so a subtitle seen on a single frame still lasts one frame interval. The frames must be in time order and not deduplicated, or a segment runs on to the next kept frame.
frame2text4llm.ocr.utils.filters.strip_overlays(results, frames=60, holes=2)
¶
Take out of the frames' text the words read on at least frames frames in a row, up to holes frames
missed: 30 s at 2 frames a second. No subtitle stays that long; a logo, a banner or a credit line does.
Left in, they make _merge_segments join every subtitle shown with them into one segment.
Words are compared without case or punctuation, and two neighbours also count joined, so that the readings
clideo com, clideocom and clideo.com of one logo are one word. Changes results in place, in time
order as OCRBatchProcessor returns them, and returns the words taken from each frame, by its
time_formatted.
frame2text4llm.ocr.utils.filters.mark_non_subtitles(segments, duration, title='')
¶
Give each merged segment a drop: "" for a subtitle, else what it is.
- "credits": a credits keyword, two words in capitals, a first name and a NAME, or the video's
title. Two credits in a row after 85 % of the video'sduration(seconds) drop everything after them too. - "ticker": three segments or more in a row, each the one before moved, as a scrolling text is read.
Changes segments in place and returns them. Segments are _merge_segments's, in time order.
frame2text4llm.ocr.utils.filters.windows(segments, gap=2.0, low=5.0, high=30.0)
¶
The subtitles in windows to cut audio on: a subtitle alone is often too short for speech recognition.
Consecutive subtitles less than gap s apart form a run, and each run is packed into windows of at least
low s; a short tail joins the window before it. A segment with a drop (mark_non_subtitles) ends a
run, since its sound has no text. A window shorter than low or longer than high s is left out.
Each window is a segment, so that AudioGenerator.extract_segments cuts it: its subtitles' text end to
end, the first one's start_time, the last one's end_time, and the subtitles themselves in
subtitles.
Language¶
frame2text4llm.language.LanguageID
¶
MMS-LID (facebook/mms-lid-256, 256 languages) on audio files of 16 kHz mono, such as the windows
AudioGenerator cuts. languages maps the ISO 639-3 codes whose probabilities are kept to column names.
hear(paths, batch=8)
¶
For each file, the language heard most (by languages' name when it is one of them, else its
code) and the probability of each of languages. Files are sorted by length, so that a batch pads
little.
frame2text4llm.language.written_in(text)
¶
"french", "english" or "other" (another language, names, a title), by counting function words; "other" when they tie.
Copies¶
frame2text4llm.copies.TextIndex
¶
The subtitles of the videos read so far, to find which one a new line comes from.
Engines¶
frame2text4llm.ocr.tools.easyocr.EasyOCR
¶
EasyOCR implementation for text extraction from images.
process_image(image, lang='en')
¶
Extract text from image using EasyOCR.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image
|
ndarray
|
NumPy array containing the image |
required |
lang
|
str
|
Language code |
'en'
|
Returns:
| Type | Description |
|---|---|
str
|
Extracted text from the image |
frame2text4llm.ocr.tools.paddleocr.PaddleOCR
¶
PaddleOCR implementation.
__init__(cache_dir=None)
¶
Initialize PaddleOCR and create cache directory if needed.
process_image(image, lang='en')
¶
Extract text from image using PaddleOCR.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image
|
ndarray
|
NumPy array containing the image |
required |
lang
|
str
|
Language code (for PaddleOCR, use 'en', 'ch', etc.) |
'en'
|
Returns:
| Type | Description |
|---|---|
str
|
Extracted text from the image |
frame2text4llm.ocr.tools.vlmocr.VLMOCR
¶
VLM-based OCR implementation (using lightweight models)
__init__(model_name='Qwen/Qwen3-VL-2B-Instruct', prompt=PROMPT)
¶
Initialize VLM OCR with specified model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name
|
str
|
Florence-2 ('microsoft/Florence-2-base-ft', 'microsoft/Florence-2-base'), 'OpenGVLab/InternVL2-1B', or any image-text-to-text model of transformers, such as 'Qwen/Qwen3-VL-2B-Instruct', which reads subtitles best (docs: Choosing an OCR engine). |
'Qwen/Qwen3-VL-2B-Instruct'
|
prompt
|
str
|
What an image-text-to-text model is asked; it answers "-" for no subtitle. |
PROMPT
|
process_images(images, lang='en', batch_size=16)
¶
process_image on many images: an image-text-to-text model reads batch_size at a time,
which keeps the GPU busy; Florence-2 and InternVL2 still read one by one.
process_image(image, lang='en')
¶
Extract text from image using VLM models.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image
|
ndarray
|
NumPy array containing the image |
required |
lang
|
str
|
Language code (not used by VLM models) |
'en'
|
Returns:
| Type | Description |
|---|---|
str
|
Extracted text from the image |
frame2text4llm.ocr.tools.mistralocr.MistralOCR
¶
OCR implementation using Mistral's OCR API.
__init__(api_key=None)
¶
Initialize Mistral OCR engine.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
api_key
|
str | None
|
Optional API key. If not provided, tries to use MISTRAL_API_KEY env var. |
None
|
process_image(base64_img)
¶
Extract text from image using Mistral OCR API.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base64_img
|
str
|
Base64-encoded image string. |
required |
lang
|
Optional language hint (not directly used). |
required |
Returns:
| Type | Description |
|---|---|
str
|
Extracted text from the image. |
frame2text4llm.ocr.tools.openaiocr.OpenAIOCR
¶
OCR implementation using OpenAI's Vision API.
__init__(api_key=None)
¶
Initialize OpenAI OCR engine.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
api_key
|
str | None
|
Optional API key. If not provided, tries to use OPENAI_API_KEY env var. |
None
|
process_image(base64_img)
¶
Extract text from image using OpenAI's Vision model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base64_img
|
str
|
Base64-encoded image string. |
required |
Returns:
| Type | Description |
|---|---|
str
|
Extracted text from the image. |
Download and export¶
frame2text4llm.consumer.youtube_consumer.YoutubeConsumer
¶
Class to consume playlists and videos from YouTube.
frame2text4llm.export.audio_generator.AudioGenerator
¶
Extracts audio from a video and segments it according to timestamps. Segments are saved as .wav files in a temporary directory.
extract_segments(segments, sample_rate=16000)
¶
For each segment (with start_time, end_time), extract audio and save as wav. Returns list of audio file paths.
frame2text4llm.export.exporter.SRTExporter
¶
Bases: BaseExporter
Export OCR results to SRT format.
export(data, output_path)
¶
Export OCR results to SRT format.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
list[dict[str, Any]]
|
List of dictionaries containing OCR results |
required |
output_path
|
str
|
Path to save the SRT file |
required |
frame2text4llm.export.exporter.JSONExporter
¶
Bases: BaseExporter
Export OCR results to JSON format.
export(data, output_path)
¶
Export OCR results to JSON format.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
list[dict[str, Any]]
|
List of dictionaries containing OCR results |
required |
output_path
|
str
|
Path to save the JSON file |
required |
frame2text4llm.export.huggingface_cooker.HuggingfaceCooker
¶
A class to prepare and upload datasets to Huggingface Hub.
__init__(username, token=None, repository_id=None)
¶
Initialize the Huggingface Cooker.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
username
|
str
|
Huggingface username |
required |
token
|
str | None
|
Huggingface API token (optional if not pushing to hub) |
None
|
repository_id
|
str | None
|
Repository ID to push to (optional) |
None
|
cook_dataset(ocr_results, video_path, output_dir, push_to_hub=False)
¶
Cook a Huggingface dataset from OCR results and video file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ocr_results
|
list[dict[str, Any]]
|
List of OCR results with timestamps and text |
required |
video_path
|
str
|
Path to the video file |
required |
output_dir
|
str
|
Directory to save the prepared dataset |
required |
push_to_hub
|
bool
|
Whether to push the dataset to Huggingface Hub |
False
|
Returns:
| Type | Description |
|---|---|
Dataset
|
The created Huggingface Dataset |