WebPageData

class WebPageData(full_text=None, languages=None, metadata=None, page=None, request=None, snippets=<factory>, statistics=None, structured=None, url_tree=None)

Bases: object

Extracted and processed data for one web page.

Attributes

full_text
languages
metadata
page
request
snippets
statistics
structured
url_tree

Methods

read_json(path)	Read web-page data from a JSON file.
to_dict()	Convert the web-page data to a dictionary.
write_json([path])	Serialize the web-page data and optionally write it to disk.

Parameters:

• full_text (str | None)

• languages (Dict[str, Any] | None)

• metadata (Dict[str, Any] | None)

• page (Dict[str, Any] | None)

• request (Dict[str, Any] | None)

• snippets (SnippetSet)

• statistics (Dict[str, Any] | None)

• structured (List[Any] | None)

• url_tree (Dict[str, Any] | None)

write_json(path=None)

Serialize the web-page data and optionally write it to disk.

Parameters:

• path (Path | str | None) – Optional destination file path.

• self (WebPageData)

Returns:

JSON representation of the web-page data.

Return type:

str

to_dict()

Convert the web-page data to a dictionary.

Returns:

Web-page payload.

Parameters:

self (WebPageData)

Return type:

Dict[str, Any]

classmethod read_json(path)

Read web-page data from a JSON file.

Parameters:

path (Path | str) – Source file path.

Returns:

Parsed web-page data.

Return type:

WebPageData