--- ## What is mdget? `mdget` ("markdown get" or "midget") is a simple command‑line utility that fetches a webpage and converts its HTML content to clean Markdown. It’s designed for quick, scriptable extraction of content from the web into a readable, portable format. ## How do I use mdget? mdget [OPTIONS]... [URL] By default, `mdget` fetches the given URL and outputs the converted Markdown to standard output. ### Options -o, --output FILE Write output to FILE instead of stdout -t, --timeout SECONDS Request timeout in seconds (default: 30) -u, --user-agent AGENT Custom User‑Agent header (default: "mdget/0.1.0") -h, --help Show this help message and exit -v, --version Show version information and exit ### Examples # Basic usage: fetch a webpage and print Markdown to stdout mdget https://example.com # Save the output to a file mdget --output page.md https://example.com # Use a custom timeout and User‑Agent mdget --timeout 10 --user-agent "MyBot/1.0" example.com # Automatically add https:// prefix to bare domains mdget github.com > github.md # Show version information mdget --version # Show help message mdget --help ## How does `mdget` work? `mdget` performs the following steps for each request: 1. Parse command‑line arguments 2. Validate and normalise the URL; if the provided URL does not start with `http://` or `https://`, `https://` is prepended. 3. Fetch the HTML using the [`req` HTTP client library](https://req.cool/), with configurable timeout and an optional User‑Agent. 4. Convert HTML to Markdown using the `html‑to‑markdown` library, which preserves semantic structure and handles relative URLs correctly. 5. Output the result to `STDOUT` or `-o/--output` if specified. The program exits with a non‑zero status if the HTTP request fails, the server returns an error, or the HTML‑to‑Markdown conversion fails for whatever reason. ## Dependencies `mdget` is written in Go and uses the following libraries: * [`req`](https://github.com/imroc/req) – a modern, feature‑rich HTTP client * [`html‑to‑markdown`](https://github.com/JohannesKaufmann/html-to-markdown) – a robust HTML‑to‑Markdown converter * [`go‑arg`](https://github.com/alexflint/go-arg) – struct‑based argument parsing with automatic help and version flags ## How do I build mdget? `mdget` can be built with a standard Go toolchain: go build ./cmd/mdget.go ## Licence This project is licensed under [European Union Public Licence 1.2](https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12).