Centralize URL fetching with host validation and default timeouts

refactor: centralize URL fetching with host validation and default timeouts (#4388)
Centralize URL fetching with host validation and default timeouts445c95-refactor-centralize-url-fetching-with-host-validation-cve-2026-71428refactorcentralize url fetching host validation default

Committed by william-u10d

commit message
refactor: centralize URL fetching with host validation and default timeouts (#4388)

## Summary
 
Introduces `unstructured/safe_http.py` as a single, shared entry point
for
outbound URL fetches, and routes the `url=` code paths in `partition`,
`partition_html`, and `partition_md` through it. Previously each of
these
called `requests.get` directly with inconsistent (and in some cases
absent)
timeout and validation behavior; this centralizes that logic in one
place so
it stays consistent and testable.
 
## What changed
 
- **New `unstructured/safe_http.py`** — a `safe_get()` helper that all
three
  partitioners now use instead of calling `requests.get` directly.
- The helper applies, consistently:
  - an `http`/`https` scheme allowlist
  - a hostname denylist with IDNA normalization
  - address validation performed at TCP connect time, so the address
    validated is the address actually connected to
  - manual redirect following with per-hop re-validation, dropping
    credential material (`Authorization`/`Cookie`/proxy auth, plus
    `auth=`/`cookies=`) on cross-origin hops via requests' own
    `should_strip_auth`
  - default `(connect, read)` timeouts, and refusal of proxied requests
- an opt-out via `allow_private=` / the `UNSTRUCTURED_ALLOW_PRIVATE_URL`
    environment variable for controlled local usage
 
## Behavior changes (why this is a minor release)
 
Fetches that resolve to non-routable, loopback, or link-local addresses
are
now rejected by default. Outbound fetches also now carry default
timeouts
where some previously had none. Callers that legitimately need to reach
a
private/internal host can opt out with
`UNSTRUCTURED_ALLOW_PRIVATE_URL=1` (or
`allow_private=True`).
 
## Tests
 
Adds `test_unstructured/test_safe_http.py` covering IP/hostname
classification, connect-time validation, redirect re-validation,
cross-origin
credential stripping, and the opt-out.
 
## Version
 
Bumps to `0.24.0` (see CHANGELOG) — minor, reflecting the behavior
changes
above.

Editor assessment

This commit patches CVE-2026-71428. The subject describes centralizing URL fetching with host validation and default timeouts, and although it is typed refactor the body makes clear this is a security consolidation, so the taxonomy choice is defensible. The body explains that partition, partition_html, and partition_md each called requests.get with inconsistent or absent validation, then lists the scheme allowlist, IDNA-normalized host denylist, connect-time address checks, per-hop redirect revalidation with credential stripping, timeouts, and the private-URL opt-out. It also documents the behaviour change and new tests. Markdown headings and a minor version bump are noisy, but the reasoning is strong.

Browse categories

More refactor examples · Back to search